Publications
Multi-modal approaches have increasingly shown promise in exploring the human side of engineering via the assessment of authentic responses to learning or working environments. This study explores the utility of non-invasive physiological wrist sensors in measuring the reactive and regulatory responses of a group of 161 engineering students taking an authentic engineering practice exam. The practice exam was categorized into Conceptual problems (e.g., rote memorization) and Analytical problems (e.g., requiring application of learned concepts through equations and free-body diagrams). Responses were measured through electrodermal activity and indicators of performance. Findings identified that the type of practice exam problem, even if designed to be within the moderate range of difficulty, influenced how students reacted to and regulated their performance to the problem (as seen by stronger positive correlations in the Analytical problems) and that these may occur via multi-componential processes.
One of the central goals of doctoral programs is to develop independent researchers and scholars who will lead the next generation of knowledge production. Despite extant evidence of inequalities in doctoral education, few studies have closely examined the experiences of first-generation college students who pursue a Ph.D. We examine how first-generation and continuing-generation doctoral students conceptualize the role of the faculty advisor/principal investigator (PI) in supporting their development as researchers. Our analysis of interviews from 111 first-year Ph.D. students in the biological sciences indicates that first-generation and continuing-generation students had similar overarching conceptions of PIs and the role of PIs in their development. However, the two groups ascribed different meanings to the same concepts. First-generation students expected more direct, skill-based guidance and assistance with learning to do research the right way. Conversely, continuing-generation students expected independence and support for their specific needs. We rely on Bourdieu's conceptualization of habitus to explain these differences and conclude by offering implications for advancing equity in doctoral education and supporting first-generation students, particularly regarding the alignment of student-advisor expectations.
This paper presents an analysis of how school mathematics credentials, particularly calculus, coupled with elite college degrees can allow for presumed proficiencies, resulting in acceptance to a selective alternative route program (SARP) for secondary mathematics. However, such credentials do not guarantee deep mathematics knowledge nor effective teaching, particularly in low-income schools serving a majority of Black and Latinx students. We present a portrait of two NYC Teaching Fellows secondary mathematics teachers, their mathematics preparation and credentials, and how their supervisors viewed them. We further present findings from two years of classroom observations and show that these SARP mathematics teachers resorted to the direct, procedural teaching style they had known themselves, held superficial mathematical understandings, regularly made math errors, and often struggled to coherently answer students' mathematical questions.
Interactions around unexpected, incorrect, or dis-preferred responses can be powerful sites of learning for both teachers and students. The information that teachers uncover through probing student thinking can then guide their pedagogical response. We report on a study of prospective teachers' skills and capabilities around a particular problem of practice: eliciting student thinking when a student has an incorrect answer. In this case, if the student's thinking is sufficiently probed, the student is able to recognize the mistake and revise their work. Focusing on prospective teachers at the beginning of a teacher preparation program, we illustrate how knowledge of this kind of eliciting skill can be gathered through the use of a live teaching simulation. Our findings reveal that these prospective teachers were more fluent with eliciting the student's process than the student's conceptual understanding. Further, they focused more on eliciting the revised method and/or solution than asking about why the mistake was made. We consider the findings in terms of the skills brought by prospective teachers that could be built upon in teacher education, skills that need to be learned, and skills that need to be unlearned.
A correction to this paper has been published: https://doi.org/10.1007/s10857-021-09491-7. © 2021, Springer Nature B.V.
We systematically compared two coding approaches to generate training datasets for machine learning (ML): (i) a holistic approach based on learning progression levels and (ii) a dichotomous, analytic approach of multiple concepts in student reasoning, deconstructed from holistic rubrics. We evaluated four constructed response assessment items for undergraduate physiology, each targeting five levels of a developing flux learning progression in an ion context. Human-coded datasets were used to train two ML models: (i) an 8-classification algorithm ensemble implemented in the Constructed Response Classifier (CRC), and (ii) a single classification algorithm implemented in LightSide Researcher's Workbench. Human coding agreement on approximately 700 student responses per item was high for both approaches with Cohen's kappas ranging from 0.75 to 0.87 on holistic scoring and from 0.78 to 0.89 on analytic composite scoring. ML model performance varied across items and rubric type. For two items, training sets from both coding approaches produced similarly accurate ML models, with differences in Cohen's kappa between machine and human scores of 0.002 and 0.041. For the other items, ML models trained with analytic coded responses and used for a composite score, achieved better performance as compared to using holistic scores for training, with increases in Cohen's kappa of 0.043 and 0.117. These items used a more complex scenario involving movement of two ions. It may be that analytic coding is beneficial to unpacking this additional complexity.
Machine learning (ML) has been increasingly employed in science assessment to facilitate automatic scoring efforts, although with varying degrees of success (i.e., magnitudes of machine-human score agreements [MHAs]). Little work has empirically examined the factors that impact MHA disparities in this growing field, thus constraining the improvement of machine scoring capacity and its wide applications in science education. We performed a meta-analysis of 110 studies of MHAs in order to identify the factors most strongly contributing to scoring success (i.e., high Cohen's kappa [kappa]). We empirically examined six factors proposed as contributors to MHA magnitudes: algorithm, subject domain, assessment format, construct, school level, and machine supervision type. Our analyses of 110 MHAs revealed substantial heterogeneity in kappa(mean=.64; range = .09-.97, taking weights into consideration). Using three-level random-effects modeling, MHA score heterogeneity was explained by the variability both within publications (i.e., the assessment task level: 82.6%) and between publications (i.e., the individual study level: 16.7%). Our results also suggest that all six factors have significant moderator effects on scoring success magnitudes. Among these, algorithm and subject domain had significantly larger effects than the other factors, suggesting that technical features and assessment external features might be primary targets for improving MHAs and ML-based science assessments.
This study provides a solid validity inferential network to guide the development, interpretation, and use of machine learning-based next-generation science assessments (NGSAs). Given that machine learning (ML) has been broadly implemented in the automatic scoring of constructed responses, essays, simulations, educational games, and interdisciplinary assessments to advance the evidence collection and inference of student science learning, we contend that additional validity issues arise for science assessments due to the involvement of ML. These emerging validity issues may not be addressed by prior validity frameworks developed for either non-science or non-ML assessments. We thus examine the changes brought in by ML to science assessments and identify seven critical validity issues of ML-based NGSAs: potential risk of misrepresenting the construct of interest, potential confounders due to that more variables may involve, nonalignment between interpretation and use of scores and designed learning goals, nonalignment between interpretation and use of scores and actual learning quality, nonalignment between machine scores and rubrics, limited generalizable ability of machine algorithmic models, and limited extrapolating ability of machine algorithmic models. Based on the seven validity issues identified, we propose a validity inferential network to address the cognitive, instructional, and inferential validity of ML-based NGSAs. To demonstrate the utility of this network, we present an exemplar of ML-based next-generation science assessments that was developed using a seven-step ML framework. We articulate how we used the validity inferential network to ensure accountable assessment design, as well as valid interpretation and use of machine scores.
As cutting-edge technologies, such as machine learning (ML), are increasingly involved in science assessments, it is essential to conceptualize how assessment practices are innovated by technologies. To partially meet this need, this article focuses on ML-based science assessments and elaborates on how ML innovates assessment practices in science education. The article starts with an articulation of the practice nature of assessment both of learning and for learning, identifying four essential assessment practices: identifying learning goals, eliciting performance, interpreting observations, and decision-making and action-taking. I then extend a three-dimensional framework for innovative assessments, including construct, functionality, and automaticity, and based on which to conceptualize innovative assessments in three levels: substitute, transform, and redefine. Using the framework, I elaborate on how the 10 articles included in this special issue, Applying Machine Learning in Science Assessment: Opportunity and Challenge, advanced our knowledge of the innovations that ML brought to science assessment practices. I contend that the 10 articles exemplify a great deal of effort to transform the four components of assessment practices: ML allows assessments to target complex, diverse, and structural constructs, and thus better approaching the three-dimensional science learning goals of the Next Generation Science Standards (NGSS Lead States, 2013); ML extends the approaches used to eliciting performance and collecting evidence; ML provides a means to better interpreting observations and using evidence; ML supports immediate and complex decision-making and action-taking. I conclude this article by pushing the field to consider the underlying educational theories that are needed for innovative assessment practices and the necessities of establishing a romance between assessment practices and the relevant educational theories, which I contend are the prominent challenges to forward innovative and ML-based assessment practices in science education.
Culturally relevant practices are valuable assets for ethnically-racially diverse schools, but few studies examine whether such practices promote students’ engagement in school longitudinally and whether ethnicity-race moderates the effects of such practices on students’ engagement. To address this gap, the present study examined whether schools that acknowledge and promote positive messages about youth’s ethnicity-race (i.e., school cultural socialization practices) promoted multiple dimensions of students’ school engagement and whether these links differed between African American and European American students. Data were collected in four waves during a two-year period from 403 fifth graders (55.1% males; 63% African American, 37% European American). The results revealed that African American youth who perceived more school cultural socialization reported greater behavioral and affective engagement (but not cognitive engagement) six months later. European Americans’ perceived school cultural socialization was unrelated to their levels of engagement in later months. Across groups, neither type of engagement predicted subsequent school cultural socialization, supporting the direction of effects in the results. Implications are discussed regarding how educators can leverage cultural socialization to promote school engagement among African American youth. © 2021, The Author(s), under exclusive licence to Springer Science+Business Media, LLC part of Springer Nature.


