Automated feedback as students answer open-ended math questions has significant potential in improving learning outcomes at large scale. A key part of automated feedback systems is an error classification component, which identifies student errors and enables appropriate, predefined feedback to be deployed. Most existing approaches to error classification use a rule-based method, which has limited capacity to generalize. Existing data-driven methods avoid these limitations but specifically require mathematical expressions in student responses to be parsed into syntax trees. This requirement is itself a limitation, since student responses are not always syntactically valid and cannot be converted into trees. In this work, we introduce a flexible method for error classification using pre-trained large language models. We demonstrate that our method can outperform existing methods in algebra error classification, and is able to classify a larger set of student responses. Additionally, we analyze common classification errors made by our method and discuss limitations of automated error classification. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Dialogue Acts (DAs) can be used to explain what expert tutors do and what students know during the tutoring process. Most empirical studies adopt the random sampling method to obtain sentence samples for manual annotation of DAs, which are then used to train DA classifiers. However, these studies have paid little attention to sample informativeness, which can reflect the information quantity of the selected samples and inform the extent to which a classifier can learn patterns. Notably, the informativeness level may vary among the samples and the classifier might only need a small amount of low informative samples to learn the patterns. Random sampling may overlook sample informativeness, which consumes human labelling costs and contributes less to training the classifiers. As an alternative, researchers suggest employing statistical sampling methods of Active Learning (AL) to identify the informative samples for training the classifiers. However, the use of AL methods in educational DA classification tasks is under-explored. In this paper, we examine the informativeness of annotated sentence samples. Then, the study investigates how the AL methods can select informative samples to support DA classifiers in the AL sampling process. The results reveal that most annotated sentences present low informativeness in the training dataset and the patterns of these sentences can be easily captured by the DA classifier. We also demonstrate how AL methods can reduce the cost of manual annotation in the AL sampling process. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Humans adopt various problem-solving strategies depending on their mastery level, problem type, and complexity. Many of these problem-solving strategies have been integrated within intelligent problem-solvers to solve structured and complex problems efficiently. One such strategy is the means-ends analysis which involves comparing the goal and the givens of a problem and iteratively setting up subgoal(s) at each step until the subgoal(s) are straightforward to derive from the givens. However, little is known about the impact of explicitly teaching novices such a strategy for structured problem-solving with tutors. In this study, we teach novices a subgoal-directed problem-solving strategy inspired by means-ends analysis using a problem-based training intervention within an intelligent logic-proof tutor. As we analyzed students’ performance and problem-solving approaches after training, we observed that the students who learned the strategy used it more when solving new problems, constructed optimal logic proofs, and outperformed those who did not learn the strategy. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Developing natural language processing (NLP) models to automatically score students’ written responses to science problems is critical for science education. However, collecting sufficient student responses and labeling them for training or fine-tuning NLP models is time and cost-consuming. Recent studies suggest that large-scale pre-trained language models (PLMs) can be adapted to downstream tasks without fine-tuning by using prompts. However, no research has employed such a prompt approach in science education. As students’ written responses are presented with natural language, aligning the scoring procedure as the next sentence prediction task using prompts can skip the costly fine-tuning stage. In this study, we developed a zero-shot approach to automatically score student responses via Matching exemplars as Next Sentence Prediction (MeNSP). This approach employs no training samples. We first apply MeNSP in scoring three assessment tasks of scientific argumentation and found machine-human scoring agreements, Cohen’s Kappa ranges from 0.30 to 0.57, and F1 score ranges from 0.54 to 0.81. To improve scoring performance, we extend our research to the few-shots setting, either randomly selecting labeled student responses at each grading level or manually constructing responses to fine-tune the models. We find that one task’s performance is improved with more samples, Cohen’s Kappa from 0.30 to 0.38, and F1 score from 0.54 to 0.59; for the two other tasks, scoring performance is not improved. We also find that randomly selected few-shots perform better than the human expert-crafted approach. This study suggests that MeNSP can yield referable automatic scoring for student-written responses while significantly reducing the cost of model training. This method can benefit low-stakes classroom assessment practices in science education. Future research should further explore the applicability of the MeNSP in different types of assessment tasks in science education and further improve the model performance. Our code is available at https://github.com/JacksonWuxs/MeNSP. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

To assess student knowledge, educators face a tradeoff between open-ended versus fixed-response questions. Open-ended questions are easier to formulate, and provide greater insight into student learning, but are burdensome. Machine learning methods that could reduce the assessment burden also have a cost, given that large datasets of reliably assessed examples (labeled data) are required for training and testing. We address the human costs of assessment and data labeling using selective prediction, where the output of a machine learned model is used when the model makes a confident decision, but otherwise the model defers to a human decision-maker. The goal is to defer less often while maintaining human assessment quality on the total output. We refer to the deferral criteria as a deferral policy, and we show it is possible to learn when to defer. We first trained an autograder on a combination of historical data and a small amount of newly labeled data, achieving moderate performance. We then used the autograder output as input to a logistic regression to learn when to defer. The learned logistic regression equation constitutes a deferral policy. Tests of the selective prediction method on a held out test set showed that human-level assessment quality can be achieved with a major reduction of human effort. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

When students use e-learning systems such as learning management systems and e-book systems, the operation logs are stored and analyzed to understand student learning behaviors. For implementing some applications, such as dashboard systems and at-risk student detection, the operation logs are mainly transformed into features designed by researchers. Such hand-crafted features, like the number of operations, are easily interpretable. However, the power of the hand-craft features may be limited for the recent large-scale educational dataset. In machine learning research, data-driven features are demonstrated to be a better representation than hand-crafted features. However, there are few discussions in the educational data due to a need for many operation logs. In this study, we collect reading logs of an e-book system. We propose a representation learning method for the reading logs based on contrastive learning. Our proposed method transforms time-series reading logs into reading behavior feature vectors directly without hand-crafted features. In our experiments, we demonstrate that the power of our feature representation is better than a traditional count-based hand-crafted feature representation in the at-risk student detection task. In addition, we investigate the characteristics of the feature space learned by our proposed method. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

The use of predictive models in education promises individual support and personalization for students. To develop trustworthy models, we need to understand what factors and causes contribute to a prediction. Thus, it is necessary to develop models that are not only accurate but also explainable. Moreover, we need to conduct holistic model evaluations that also quantify explainability or other metrics next to established performance metrics. This paper explores the use of Explainable Boosting Machines (EBMs) for the task of academic risk prediction. EBMs are an extension of Generative Additive Models and promise a state-of-the-art performance on tabular datasets while being inherently interpretable. We demonstrate the benefits of using EBMs in the context of academic risk prediction trained on online learning behavior data and show the explainability of the model. Our study shows that EBMs are equally accurate as other state-of-the-art approaches while being competitive on relevant metrics for trustworthy academic risk prediction such as earliness, stability, fairness, and faithfulness of explanations. The results encourage the broader use of EBMs for other Artificial Intelligence in education tasks. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

We consider the task of automatically generating math word problems (MWPs) of various difficulties that meet the needs of teachers in teaching and testing students in corresponding educational stages. Existing methods fail to produce high-quality problems while allowing the teacher control over the problem difficulty level. In this work, we introduce a controllable MWP generation pipeline that samples from an energy language model with various expert model components for realizing the target attributes. We control the difficulty of the resulting MWPs from mathematical and linguistic aspects by imposing constraints on equations, vocabulary, and topics. We also use other control attributes including fluency and distance to the conditioning sequence to manage language quality and creativity. Experiments and evaluation results demonstrate our approach improves upon the baselines in generating solvable, well-formed, and diverse MWPs of controlled difficulty levels. Lastly, we solicit feedback from various math educators who approve the effectiveness of our system for their MWP design processes. They suggest our outputs align with the expectations of problem designers showing a possibility of using such problem generators in real-life educational scenarios. Our code and data are available on request. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Proof Blocks is a software tool that allows students to practice writing mathematical proofs by dragging and dropping lines instead of writing proofs from scratch. Proof Blocks offers the capability of assigning partial credit and providing solution quality feedback to students. This is done by computing the edit distance from a student’s submission to some predefined set of solutions. In this work, we propose an algorithm for the edit distance problem that significantly outperforms the baseline procedure of exhaustively enumerating over the entire search space. Our algorithm relies on a reduction to the minimum vertex cover problem. We benchmark our algorithm on thousands of student submissions from multiple courses, showing that the baseline algorithm is intractable, and that our proposed algorithm is critical to enable classroom deployment. Our new algorithm has also been used for problems in many other domains where the solution space can be modeled as a DAG, including but not limited to Parsons Problems for writing code, helping students understand packet ordering in networking protocols, and helping students sketch solution steps for physics problems. Integrated into multiple learning management systems, the algorithm serves thousands of students each year. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Students use learning analytics systems to make day-to-day learning decisions, but may not understand their potential flaws. This work delves into student understanding of an example learning analytics algorithm, Bayesian Knowledge Tracing (BKT), using Cognitive Task Analysis (CTA) to identify knowledge components (KCs) comprising expert student understanding. We built an interactive explanation to target these KCs and performed a controlled experiment examining how varying the transparency of limitations of BKT impacts understanding and trust. Our results show that, counterintuitively, providing some information on the algorithm’s limitations is not always better than providing no information. The success of the methods from our BKT study suggests avenues for the use of CTA in systematically building evidence-based explanations to increase end user understanding of other complex AI algorithms in learning analytics as well as other domains. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.