Research on Automated Formative Feedback of Problem-Solving Strategy Writing in Introductory Physics using Natural Language Processing
Effective Years: 2023-2026
Problem solving is an essential 21st century STEM workforce skill, but in most undergraduate introductory STEM courses students use ineffective problem-solving strategies. The pedagogical challenge is that these courses typically have very large enrollments which prohibits instructors' providing of real-time feedback to students on their unproductive strategies, given how labor intensive providing such individualized feedback is. The investigators propose to use natural language processing (NLP) to develop an automated system that will provide personalized formative feedback to students in undergraduate physics courses about their problem-solving strategies. There are two research questions: (1) How accurately can an NLP classifier score such essays and how well can it generalize across types of problems?, and (2) How well does providing students with such feedback improve their problem-solving strategies and actual ability to solve problems?
This project leverages existing research in STEM education that has shown that the use of strategy writing, and real-time formative assessment can improve students' problem-solving skills. Starting with a vast corpus of existing data, the investigators will use state-of-the-art supervised and unsupervised learning to train a machine learning algorithm (MLA) to classify strategies articulated by students. Their study will span three phases with increasing generalization, first generalizing over vocabulary used by students to describe their strategies for solving specific problems, then generalizing over sets of isomorphic problems using the same strategies, and finally generalizing over classes of transfer problems that have the same underlying principles but different surface features. Their goal is to determine the accuracy with which an MLA can be trained to provide feedback on short essays written by students describing their strategies for solving a problem. They will also investigate the extent to which real-time formative feedback to students, based on their own strategy essays, can help them to refine their strategies iteratively in order to solve problems with increasing generality. In developing and training their machine learning algorithm, the researchers will implement proven strategies to address issues of fairness in artificial intelligence (AI). The ultimate goal of the project is to transform the development of expert-like problem-solving strategies in STEM undergraduates and thereby potentially to increase their retention in STEM majors.
This project is supported by NSF's EDU Core Research (ECR) program. The ECR program emphasizes fundamental STEM education research that generates foundational knowledge in the field. Investments are made in critical areas that are essential, broad, and enduring: STEM learning and STEM learning environments, broadening participation in STEM, and STEM workforce development.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.