This study provides a solid validity inferential network to guide the development, interpretation, and use of machine learning-based next-generation science assessments (NGSAs). Given that machine learning (ML) has been broadly implemented in the automatic scoring of constructed responses, essays, simulations, educational games, and interdisciplinary assessments to advance the evidence collection and inference of student science learning, we contend that additional validity issues arise for science assessments due to the involvement of ML. These emerging validity issues may not be addressed by prior validity frameworks developed for either non-science or non-ML assessments. We thus examine the changes brought in by ML to science assessments and identify seven critical validity issues of ML-based NGSAs: potential risk of misrepresenting the construct of interest, potential confounders due to that more variables may involve, nonalignment between interpretation and use of scores and designed learning goals, nonalignment between interpretation and use of scores and actual learning quality, nonalignment between machine scores and rubrics, limited generalizable ability of machine algorithmic models, and limited extrapolating ability of machine algorithmic models. Based on the seven validity issues identified, we propose a validity inferential network to address the cognitive, instructional, and inferential validity of ML-based NGSAs. To demonstrate the utility of this network, we present an exemplar of ML-based next-generation science assessments that was developed using a seven-step ML framework. We articulate how we used the validity inferential network to ensure accountable assessment design, as well as valid interpretation and use of machine scores.

As cutting-edge technologies, such as machine learning (ML), are increasingly involved in science assessments, it is essential to conceptualize how assessment practices are innovated by technologies. To partially meet this need, this article focuses on ML-based science assessments and elaborates on how ML innovates assessment practices in science education. The article starts with an articulation of the practice nature of assessment both of learning and for learning, identifying four essential assessment practices: identifying learning goals, eliciting performance, interpreting observations, and decision-making and action-taking. I then extend a three-dimensional framework for innovative assessments, including construct, functionality, and automaticity, and based on which to conceptualize innovative assessments in three levels: substitute, transform, and redefine. Using the framework, I elaborate on how the 10 articles included in this special issue, Applying Machine Learning in Science Assessment: Opportunity and Challenge, advanced our knowledge of the innovations that ML brought to science assessment practices. I contend that the 10 articles exemplify a great deal of effort to transform the four components of assessment practices: ML allows assessments to target complex, diverse, and structural constructs, and thus better approaching the three-dimensional science learning goals of the Next Generation Science Standards (NGSS Lead States, 2013); ML extends the approaches used to eliciting performance and collecting evidence; ML provides a means to better interpreting observations and using evidence; ML supports immediate and complex decision-making and action-taking. I conclude this article by pushing the field to consider the underlying educational theories that are needed for innovative assessment practices and the necessities of establishing a romance between assessment practices and the relevant educational theories, which I contend are the prominent challenges to forward innovative and ML-based assessment practices in science education.

Culturally relevant practices are valuable assets for ethnically-racially diverse schools, but few studies examine whether such practices promote students’ engagement in school longitudinally and whether ethnicity-race moderates the effects of such practices on students’ engagement. To address this gap, the present study examined whether schools that acknowledge and promote positive messages about youth’s ethnicity-race (i.e., school cultural socialization practices) promoted multiple dimensions of students’ school engagement and whether these links differed between African American and European American students. Data were collected in four waves during a two-year period from 403 fifth graders (55.1% males; 63% African American, 37% European American). The results revealed that African American youth who perceived more school cultural socialization reported greater behavioral and affective engagement (but not cognitive engagement) six months later. European Americans’ perceived school cultural socialization was unrelated to their levels of engagement in later months. Across groups, neither type of engagement predicted subsequent school cultural socialization, supporting the direction of effects in the results. Implications are discussed regarding how educators can leverage cultural socialization to promote school engagement among African American youth. © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature.

As stereotype threat was initially examined in experimental settings, the effects of such threats have often been tested by temporarily manipulating social identity threats. This study expands the literature by examining 9th-grade adolescents' naturalistic stereotype threat, using data from the National Study of Learning Mindsets in the United States (n = 6040, age: 13-17, M-age = 14.31, 6.9% Black boys, 6.5% Black girls, 13.1% Latinos, 12.3% Latinas, 31.5% White boys, 29.7% White girls). The results indicate that Black and Latinx students experience higher levels of stereotype threat in high school mathematics classrooms than do their White peers. When students perceive that their teachers have created fixed mindset climates, they experience greater stereotype threat. Stereotype threat, in turn, negatively predicts Black and Latino boys and White girls' later achievement via anxiety. These findings highlight the importance of creating mathematics classrooms that cultivate a growth mindset and minimize social identity threat.

Though adolescents' science identity beliefs predict positive STEM outcomes, researchers have yet to examine developmental differences within racial/ethnic groups despite theoretical arguments for such studies. The current study examined science identity trajectories for Black (14%), Latinx (22%), Asian (4%), and White (52%) students (N = 21,170; 50% girls) from 9(th) grade to three years post-high school and the variability within each racial/ethnic group based on gender and college generational status. Contrary to the literature, students' science identities increased over time, and the increases were larger for potential first- versus continuing-generation White students. Potential continuing-generation boys had stronger 9(th) grade science identities than potential first-generation girls in all groups except Asians. The findings suggest who might benefit from additional supports within each racial/ethnic group.

This study integrates theories of achievement motivation and emotion to investigate daily academic behavior in an undergraduate online course. Using cluster analysis and hierarchical logistic regression, we analyze profiles of task values and anticipated emotions to understand expectations and completion of academic tasks over the duration of a week. Students’ task specific interest, opportunity cost, and anticipated satisfaction and regret varied across tasks and were predictive of both their expectations of task completion and actual task completion reported the following day. The results shed light on the important role of achievement motivation as situated and dynamic, highlighting the interplay between task priorities, task values, and anticipated emotions in academic task engagement. © 2021, The Author(s).

Prior research suggests that students endorsing a science identity are more likely to participate in optional science experiences and choose STEM careers. Science identity is a topical identity, which refers to an identity related to a topic rather than a social or cultural group. However, studies of topical identities typically examine them in isolation. The current study identified typically occurring combinations of topical identities as identity complexes to determine whether science identity would tend to occur within STEM-only complexes or together within larger topical identity complexes. Over 1200 urban public-school students in 6th, 7th, and 9th grades from two different regions in the USA completed surveys asking about their topical identities, choice preferences, and optional science experiences. Latent class analyses revealed that students often endorse science identities in the context of other unrelated identities like athletic and artistic identities. Further, the frequency (overall and relative to each other) of the two high-science identity complexes varied substantially by gender, ethnicity, and grade. These patterns were not simple reflects of the commonly observed overall rates of science identity by demographics. Further, students in topical complexes with high science identity still had high participation in optional science experiences despite having many topical identities that could compete for time. © 2019, Springer Nature B.V.

How can we evaluate the performance of a disambiguation method implemented on big bibliographic data? This study suggests that the open researcher profile system, ORCID, can be used as an authority source to label name instances at scale. This study demonstrates the potential by evaluating the disambiguation performances of Author-ity2009 (which algorithmically disambiguates author names in MEDLINE) using 3 million name instances that are automatically labeled through linkage to 5 million ORCID researcher profiles. Results show that although ORCID-linked labeled data do not effectively represent the population of name instances in Author-ity2009, they do effectively capture the 'high precision over high recall' performances of Author-ity2009. In addition, ORCID-linked labeled data can provide nuanced details about the Author-ity2009's performance when name instances are evaluated within and across ethnicity categories. As ORCID continues to be expanded to include more researchers, labeled data via ORCID-linkage can be improved in representing the population of a whole disambiguated data and updated on a regular basis. This can benefit author name disambiguation researchers and practitioners who need large-scale labeled data but lack resources for manual labeling or access to other authority sources for linkage-based labeling. The ORCID-linked labeled data for Author-ity2009 are publicly available for validation and reuse.

This study utilizes interviews from 22 young female engineers from diverse racial/ethnic backgrounds as they first entered the White and male-dominated engineering labor force with the goal of examining: (1) how these women endorsed a gender-blind frame that characterizes their workplaces as fundamentally meritocratic, and alternatively, (2) how they named gender as relevant or salient to experiences and interactions at work. Drawing on the insights of intersectional scholars to answer the previous questions, the study calls attention to how the invocation of these frames differed for women of color compared to their majority White female peers. Results revealed that most respondents strongly endorsed the idea that engineering workplaces are meritocratic and that their gender is not relevant. However, there is also evidence of racial divergence in the themes expressed. For example, some White women expressed a narrative contradictory to meritocracy, discussing their workplaces as like family, while in contrast, women of color often expressed uncomfortable experiences of standing out. Overall, the results suggest that female engineers' tendency to disavow, either explicitly or implicitly, that discrimination and bias occurs in their workplaces, likely contributes to continued gender and racial inequality; subsequently, programs and interventions to facilitate awareness of inequality are critically needed.

This longitudinal study explores three research questions. First, what is the prevalence of math and science gender stereotypes among high school students, their parents, and teachers? Second, are parents' and teachers' gender stereotypes related to adolescents' stereotypes? And third, are adolescents' gender stereotypes associated with their math and science identity and outcomes? We used a nationally representative U.S. sample (N = 22,190, 50% girls, 53% White, 22% Latinx, 13% Black) of adolescents surveyed at 9th and 11th grade, their parents, and teachers. Adolescents' transcripts were also collected at the end of high school. Adolescent gender stereotypes became significantly more traditional from 9 to 11th grade. Parents were three times more likely to believe that males are better at math/science (compared to believing females are better), and we found significant positive relations between parents' and adolescents' stereotypes. Finally, adolescents' math/science gender stereotypes were significantly related to their math/science identity, which in turn was related to their STEM outcomes over the course of high school. Our findings give insight to the development of academic gender stereotypes in adolescence, their potential precursors, and their relations to academic outcomes.