Publications
Self-efficacy has emerged as one of the most important noncognitive variables explaining academic behavior. It has been shown to influence students' academic and career decisions as well as their academic performance. Multiple studies have reported differences in self-efficacy between men and women in science, technology, engineering, and mathematics classes. A student's personality, characterized by the five-factor model, is also related to academic performance; some personality facets are substantially different for men and women. This work examines the relations among the five-factor model of personality (agreeableness, conscientiousness, extraversion, neuroticism, and openness), self-efficacy toward physics and mathematics, and course outcomes in university physics and mathematics classes. Women reported significantly higher neuroticism in all classes, a medium to large effect size, and significantly higher conscientiousness in Calculus 1 and Physics 1, small effects. Men reported higher self-efficacy in two-semester Calculus 1, one-semester Calculus 1, Physics 1, and Physics 2, small effects. Conscientiousness and neuroticism had competing mediational effects on the relation of gender to self-efficacy. The path through neuroticism accounted for 25%-47% of the total effect of gender on self-efficacy (increasing self-efficacy for men) and the path through conscientiousness accounted for 12%-23% of the total effect (increasing self-efficacy for women). Self-efficacy mediated the relation of conscientiousness to course grade in all classes, accounting for 30%-45% of the total effect.
This study examined the mediation and moderation of membership in a demographic group underrepresented in physics classes on course outcomes measured by course grades and Force and Motion Conceptual Evaluation (FMCE) post-test scores. The study used a large dataset (N=4490) of course grades, SAT and ACT mathematics scores (ACTM), and matched FMCE pretest and post-test scores to investigate differences by gender, underrepresented ethnic or racial minority (UERM) status, and status as a first-generation college student (FGCS). For UERM and FGCS students, ACTM and pretest scores significantly mediated the relation of membership in the demographic group and both course grade and post-test score. Differences between minority and majority members of these groups were largely removed by controlling for ACTM and pretest scores. The overwhelming majority of the effect acted through ACTM for course grade (60% and 45%, respectively), while more of the effect acted through pretest score for the post-test (36% and 48%, respectively). As such, for these groups prior preparation measures predict physics outcomes (course grades or post-test scores) differently. The mediational relations for gender were dramatically different. No mediation was detected for the relation of gender to course grade because no significant difference in course grade existed. Sixty percent of the effect of gender on post-test score was not explained by either ACTM or pretest score; pretest score accounted for 38% of the effect. As such, the majority of the difference in post-test scores between men and women was not explained by either ACTM or pretest scores. Significant moderation was also detected showing that the relation of these variables was not consistent for members of all demographic groups. © 2021 authors. Published by the American Physical Society.
The Conceptual Survey of Electricity and Magnetism (CSEM) is a widely used multiple-choice instrument measuring a student's conceptual understanding of electricity and magnetism. This study applied modified module analysis (MMA) and modified module analysis-partial (MMA-P), network analytic methods that identify groups of correlated responses, to CSEM data from two institutions (N = 2538 and 3595). In module analysis, groups of correlated responses are called communities. As in previous applications of MMA and MMA-P to mechanics conceptual inventories, a number of communities related to physics concepts and some communities related to the structure of blocked items in the inventory were identified. An item block is a set of items all referring to each other or to a common stem. Many blocked communities involved responses where the response to the later item would be correct if the response to the earlier item was correct. This suggests a modified scoring rubric for the CSEM is needed to account for these connections between items. A modified scoring rubric is proposed; however, the modified overall average scores changed by less than 1%. The communities of incorrect responses to the CSEM related to physical concepts had varied explanations. These explanations ranged from seemingly straightforward errors (the electric field pointing to higher potential or reversing the right-hand rule), to misconceptions about Newton's 2nd and 3rd laws carried over from mechanics, to naive reasoning conflating general topics in electricity and magnetism. The identification of incorrect communities allowed the computation of misconception scores showing how prevalent the misconceptions were in the classes studied.
According to Bandura's social cognitive theory, a student's self-efficacy influences his or her academic and career decisions, and his or her performance outcomes; as such, a student's self-efficacy changes with time in response to the student's experiences. Self-efficacy may also vary by academic domain. Differences in STEM self-efficacy have often been reported between men and women. The purpose of this study is to explore the evolution of domain-specific STEM self-efficacy in students in gateway physics and mathematics courses and how academic feedback influences the evolution of these differences with time. Further, this study explored whether gender differences in self-efficacy are consistent across STEM domains and how these differences change in response to academic feedback. Self-efficacy in multiple academic domains (current mathematics/science class, other STEM classes, and intended profession) was assessed at multiple time points with subscales adapted from the Motivated Strategies for Learning Questionnaire. Linear mixed effects modeling was used to understand how academic feedback provided by test scores influenced changes in self-efficacy. Students in all classes expressed different levels of self-efficacy toward different domains with the lowest self-efficacy toward their current class and the highest toward their intended profession. Only the current math/science class self-efficacy of men and women differed significantly, with women expressing lower self-efficacy. The differences in current class self-efficacy were evident very early in the class before substantive class feedback was received. The evolution of self-efficacy within the class and between classes was the same for men and women.
Machine learning algorithms have recently been used to predict students' performance in an introductory physics class. The prediction model classified students as those likely to receive an A or B or students likely to receive a grade of C, D, F or withdraw from the class. Early prediction could better allow the direction of educational interventions and the allocation of educational resources. However, the performance metrics used in that study become unreliable when used to classify whether a student would receive an A, B, or C (the ABC outcome) or if they would receive a D, F or withdraw (W) from the class (the DFW outcome) because the outcome is substantially unbalanced with between 10% to 20% of the students receiving a D, F, or W. This work presents techniques to adjust the prediction models and alternate model performance metrics more appropriate for unbalanced outcome variables. These techniques were applied to three samples drawn from introductory mechanics classes at two institutions (N = 7184, 1683, and 926). Applying the same methods as the earlier study produced a classifier that was very inaccurate, classifying only 16% of the DFW cases correctly; tuning the model increased the DFW classification accuracy to 43%. Using a combination of institutional and in-class data improved DFW accuracy to 53% by the second week of class. As in the prior study, demographic variables such as gender, underrepresented minority status, fast-generation college student status, and low socioeconomic status were not important variables in the final prediction models.
Investigating student learning and understanding of conceptual physics is a primary research area within physics education research. Multiple quantitative methods have been employed to analyze commonly used mechanics conceptual inventories: the Force Concept Inventory (FCI) and the Force and Motion Conceptual Evaluation (FMCE). Recently, researchers have applied network analytic techniques to explore the structure of the incorrect responses to the FCI identifying communities of incorrect responses which could be mapped on to common misconceptions. In this study, the method used to analyze the FCI, modified module analysis was applied to a large sample of FMCE pretest and post-test responses (N-pre = 3956, N-post = 3719). The communities of incorrect responses identified were consistent with the item groups described in previous works. As in the work with the FCI, the network was simplified by only retaining nodes selected by a substantial number of students. Retaining as nodes only those incorrect answer choices selected by at least 20% of the students produced communities associated with only four misconceptions. The incorrect response communities identified for men and women were substantially different, as was the change in these communities from pretest to post-test. The 20% threshold was far more restrictive than the 4% threshold applied to the FCI in the prior work that generated similar structures. Retaining nodes selected by 5% or 10% of students generated a large number of complex communities. The communities identified at the 10% threshold were generally associated with common misconceptions producing a far richer set of incorrect communities than the FCI; this may indicate that the FMCE is a superior instrument for characterizing the breadth of student misconceptions about Newtonian mechanics.
This study examines the correlation of physics conceptual inventory pretest scores with post-instruction achievement measures (post-test scores, test averages, and course grades). The correlation for demographic groups in the minority in the physics classes studied (women, underrepresented racial/enthic students, first generation college students, and rural students) were compared with their majority peers. Three conceptual inventories were examined: the Force and Motion Conceptual Evaluation (FMCE) (N = 2450), the Force Concept Inventory (FCI) (N = 2373) and the CSEM (N1 = 1796, N2 = 2537). While many of the correlations were similar, for some of the demographic groups, the correlations were substantially different. There was little consistency in the differences measured. In most cases where the correlations differed, the correlation for the group in the minority was the smaller. As such, pretest scores may not predict course performance for some minority demographic groups as accurately as they predict outcomes for majority students. The pattern of correlation differences did not appear to be related to the size of the pretest score. If pretest scores are used for instructional decisions that have academic consequences, instructors should be aware of these potential inaccuracies and ensure the pretest used is equally valid for all students.
This work applied exploratory factor analysis (EFA) and graded Item Response Theory (IRT) to a large sample ( N = 4522) of Colorado Learning Attitudes about Science Survey (CLASS) post-test scores. EFA failed to reproduce the factor structure suggested by the authors of the CLASS and strongly supported the alternate 3-factor model suggested by Douglas et al. Graded IRT allowed an examination of the progression from non-expert-like to expert-like beliefs. This progression was generally uniform with a linear relation between the difficulty of each step in the progression. Some items within the factors identified by Douglas et al. had difficulty and discrimination parameters substantially different from other items in the factor suggesting the subscale is not unidimensional. The expert-like latent ability trait estimated by IRT correlated more strongly with measures of physics performance than measures of general academic performance indicating that expert-like beliefs are not a general property of high performing students.
The Force and Motion Conceptual Evaluation is commonly used to measure the conceptual understanding of Newtonian mechanics. Several studies have reported a substantial difference in pretest scores between men and women. This study examines the contribution of several prior preparation factors to explain the variance in pretest score and whether these factors explain gender differences in the pretest score. The study examined a large sample (N = 1060) of students taking introductory calculus-based mechanics at the university level. Women outperformed men on most prior preparation and college achievement measures. No significant differences between men and women were found in high school physics taking patterns. Linear regression analysis showed only 23% of the variance in FMCE pretest score could be explained using a linear combination of prior preparation variables. Controlling for these variables failed to explain the gender difference in pretest scores; conversely, the gender difference increased controlling for prior preparation.
Over the last decade, the "gender gap" in physics conceptual inventory scores has been extensively studied by the physics education research community. Researchers have identified many factors that influence the overall differences in post-test scores between men and women. More recently, it has been shown that the Force Concept Inventory (FCI) contains eight items that are substantially unfair; six are unfair to women, two are unfair to men. The Force and Motion Conceptual Evaluation (FMCE) and the Conceptual Survey of Electricity and Magnetism (CSEM), however, contain fewer unfair items. In this work, results from prior studies are used to further explore the gender gap in five large samples of conceptual inventory data: the FCI (N 1 3663), the FMCE (N 2 2551, N 3 3719), and the CSEM (N 4 1767, N 5 2439). The gender gap in these samples is partitioned into four components: the gender gap resulting from the student's academic performance, the gender gap resulting from prior preparation in physics, the gender gap resulting from instrumental fairness, and the gender gap of students with equal academic performance and physics preparation on the fair instrument. For all samples, very little of the gender gap was explained by differences in academic performance between men and women, measured by ACT or SAT math percentile scores or physics test average. The percentage of the gender gap resulting from instrumental fairness varied across samples from 30% in the FCI to 2% to 6% in the CSEM. A substantial part of the gender gap in four of the five samples (30%-40%) was explained by differences in prior physics preparation, measured by pretest scores on the conceptual inventories. Further correcting for conceptual physics prior preparation using the post-test score in the previous class reduced gender differences substantially.


