We examined to what extent subgroups of students identified with learning disabilities (LDs; N = 630) in the Early Childhood Longitudinal Study, Kindergarten Class of 1998 to 1999 (ECLS-K): 1998 national longitudinal study displayed heterogeneity in longitudinal profiles of reading and mathematics achievement from first to eighth grades. Multivariate growth mixture modeling yielded four classes of combined reading and mathematics trajectories for students with LD. The largest class of students with LD (Class 2, 54.3%) showed mean T-scores for both achievement domains that averaged about 1 SD below the mean, with modest decline over time. Almost a quarter of the sample (Class 1, 22.3%) displayed mean T-scores in both achievement areas near the peer-normed average; these students were mostly White, from higher socioeconomic status (SES) backgrounds, and had experienced earlier classification as LD as well as shorter duration of LD service. Classifying heterogeneity in longitudinal trajectories of both achievement areas shows promise to better understand the educational needs of students classified LD.

Background and Context. With increasing efforts to bring computing education opportunities into elementary schools, there is a growing need for assessments, with arguments for validity, to support research evaluation at these grade levels. After successfully piloting a 10-question computational thinking assessment (Assessment of Computing for Elementary Students - ACES) for 4th graders in Spring 2020, we used our analyses of item difficulty and discrimination to iterate on the assessment. Objectives. To increase the number of potential items for ACES, we created isomorphic versions of existing questions. The nature of the changes varied from incidental changes that we did not believe would impact student performance to more radical changes that seemed likely to influence question difficulty. We sought to understand the impact of these changes on student performance. Method. Using these isomorphic questions, we created two versions of our assessment and piloted them in Spring 2021 with 235 upper-elementary (4th grade) students. We analyzed the reliability of the assessments using Cronbach's alpha. We used Chi-squared tests to analyze questions that were identical across the two assessments to form a baseline of comparison and then ran Chi-Squared and Kruskal-Wallis H tests to analyze the differences between the isomorphic copies of the questions. Findings. Both assessment versions demonstrated good reliability, with identical Cronbach's alphas of 0.868. We found statistically similar performance on the identical questions between our two groups of students, allowing us to compare their performance on the isomorphic questions. Students performed differently on the isomorphic questions, indicating the changes to the questions had a differential impact on student performance. Implications. This paper builds on existing work by presenting methods for creating isomorphic questions. We provide valuable lessons learned, both on those methods and on the impact of specific types of changes on student performance.

With increased focus on historically excluded populations, there have been recent calls for HCI research methods to more adequately acknowledge and address the historical context of racism, sexism, gendered racism, epistemic violence, classism, and so on. In this article, we utilize Black feminist epistemologies to serve as critical frameworks for understanding the historical context that reveals the interconnected systems of power that mutually influence one another to create unequal outcomes or social inequalities for different populations. Leveraging Black feminist thought (BFT) and intersectionality as critical social theories of design praxis, we introduce intersectional analysis of power—a method that enables HCI researchers, designers, and practitioners to identify and situate saturated sites of violence in a historical context and to transform the ways in which they engage with populations that have been historically oppressed. Engaging in self-reflection as researchers, we apply an intersectional analysis of power to co-design technologies with community street outreach workers who address violence in their predominantly Black communities. We: (1) identify the saturated site of violence; (2) identify the intersecting systems of power and who holds power (past and present); (3) describe the “conceptual glue” that binds these intersecting systems together and the assumption(s) that those who hold power are employing to guide their interactions; (4) examine the ways in which Black people are subjugated, surveilled, and/or expected to assimilate to “normative” ways of being and behaving; and (5) identify acts of resistance. This article contributes an alternative to traditional HCI and design methods that falsely perpetuate a lens of neutrality and colorblindness that centers on whiteness, innovation, and capitalism and ignores the history of State-sanctioned violence and structural oppression.

To assess student knowledge, educators face a tradeoff between open-ended versus fixed-response questions. Open-ended questions are easier to formulate, and provide greater insight into student learning, but are burdensome. Machine learning methods that could reduce the assessment burden also have a cost, given that large datasets of reliably assessed examples (labeled data) are required for training and testing. We address the human costs of assessment and data labeling using selective prediction, where the output of a machine learned model is used when the model makes a confident decision, but otherwise the model defers to a human decision-maker. The goal is to defer less often while maintaining human assessment quality on the total output. We refer to the deferral criteria as a deferral policy, and we show it is possible to learn when to defer. We first trained an autograder on a combination of historical data and a small amount of newly labeled data, achieving moderate performance. We then used the autograder output as input to a logistic regression to learn when to defer. The learned logistic regression equation constitutes a deferral policy. Tests of the selective prediction method on a held out test set showed that human-level assessment quality can be achieved with a major reduction of human effort. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Automated feedback as students answer open-ended math questions has significant potential in improving learning outcomes at large scale. A key part of automated feedback systems is an error classification component, which identifies student errors and enables appropriate, predefined feedback to be deployed. Most existing approaches to error classification use a rule-based method, which has limited capacity to generalize. Existing data-driven methods avoid these limitations but specifically require mathematical expressions in student responses to be parsed into syntax trees. This requirement is itself a limitation, since student responses are not always syntactically valid and cannot be converted into trees. In this work, we introduce a flexible method for error classification using pre-trained large language models. We demonstrate that our method can outperform existing methods in algebra error classification, and is able to classify a larger set of student responses. Additionally, we analyze common classification errors made by our method and discuss limitations of automated error classification. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Automatic Short Answer Grading aims to automatically grade short answers authored by students. Recent work has shown that this task can be effectively reformulated as a Natural Language Inference problem. State-of-the-art is defined by the use of large pretrained language models fine-tuned in the domain dataset. But how to quantify the effectiveness of the models in small data regimes still remains an open issue. In this work we present a set of experiments to analyse the impact of different annotation strategies when not enough training examples for fine-tuning the model are available. We find that when annotating few examples, it is preferable to have more question variability than more answers per question. With this annotation strategy, our model outperforms state-of-the-art systems utilizing only 10% of the full-training set. Finally, experiments show that the use of out-of-domain annotated question-answer examples can be harmful when fine-tuning the models. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Humans adopt various problem-solving strategies depending on their mastery level, problem type, and complexity. Many of these problem-solving strategies have been integrated within intelligent problem-solvers to solve structured and complex problems efficiently. One such strategy is the means-ends analysis which involves comparing the goal and the givens of a problem and iteratively setting up subgoal(s) at each step until the subgoal(s) are straightforward to derive from the givens. However, little is known about the impact of explicitly teaching novices such a strategy for structured problem-solving with tutors. In this study, we teach novices a subgoal-directed problem-solving strategy inspired by means-ends analysis using a problem-based training intervention within an intelligent logic-proof tutor. As we analyzed students’ performance and problem-solving approaches after training, we observed that the students who learned the strategy used it more when solving new problems, constructed optimal logic proofs, and outperformed those who did not learn the strategy. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Developing natural language processing (NLP) models to automatically score students’ written responses to science problems is critical for science education. However, collecting sufficient student responses and labeling them for training or fine-tuning NLP models is time and cost-consuming. Recent studies suggest that large-scale pre-trained language models (PLMs) can be adapted to downstream tasks without fine-tuning by using prompts. However, no research has employed such a prompt approach in science education. As students’ written responses are presented with natural language, aligning the scoring procedure as the next sentence prediction task using prompts can skip the costly fine-tuning stage. In this study, we developed a zero-shot approach to automatically score student responses via Matching exemplars as Next Sentence Prediction (MeNSP). This approach employs no training samples. We first apply MeNSP in scoring three assessment tasks of scientific argumentation and found machine-human scoring agreements, Cohen’s Kappa ranges from 0.30 to 0.57, and F1 score ranges from 0.54 to 0.81. To improve scoring performance, we extend our research to the few-shots setting, either randomly selecting labeled student responses at each grading level or manually constructing responses to fine-tune the models. We find that one task’s performance is improved with more samples, Cohen’s Kappa from 0.30 to 0.38, and F1 score from 0.54 to 0.59; for the two other tasks, scoring performance is not improved. We also find that randomly selected few-shots perform better than the human expert-crafted approach. This study suggests that MeNSP can yield referable automatic scoring for student-written responses while significantly reducing the cost of model training. This method can benefit low-stakes classroom assessment practices in science education. Future research should further explore the applicability of the MeNSP in different types of assessment tasks in science education and further improve the model performance. Our code is available at https://github.com/JacksonWuxs/MeNSP. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

The use of predictive models in education promises individual support and personalization for students. To develop trustworthy models, we need to understand what factors and causes contribute to a prediction. Thus, it is necessary to develop models that are not only accurate but also explainable. Moreover, we need to conduct holistic model evaluations that also quantify explainability or other metrics next to established performance metrics. This paper explores the use of Explainable Boosting Machines (EBMs) for the task of academic risk prediction. EBMs are an extension of Generative Additive Models and promise a state-of-the-art performance on tabular datasets while being inherently interpretable. We demonstrate the benefits of using EBMs in the context of academic risk prediction trained on online learning behavior data and show the explainability of the model. Our study shows that EBMs are equally accurate as other state-of-the-art approaches while being competitive on relevant metrics for trustworthy academic risk prediction such as earliness, stability, fairness, and faithfulness of explanations. The results encourage the broader use of EBMs for other Artificial Intelligence in education tasks. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.

We consider the task of automatically generating math word problems (MWPs) of various difficulties that meet the needs of teachers in teaching and testing students in corresponding educational stages. Existing methods fail to produce high-quality problems while allowing the teacher control over the problem difficulty level. In this work, we introduce a controllable MWP generation pipeline that samples from an energy language model with various expert model components for realizing the target attributes. We control the difficulty of the resulting MWPs from mathematical and linguistic aspects by imposing constraints on equations, vocabulary, and topics. We also use other control attributes including fluency and distance to the conditioning sequence to manage language quality and creativity. Experiments and evaluation results demonstrate our approach improves upon the baselines in generating solvable, well-formed, and diverse MWPs of controlled difficulty levels. Lastly, we solicit feedback from various math educators who approve the effectiveness of our system for their MWP design processes. They suggest our outputs align with the expectations of problem designers showing a possibility of using such problem generators in real-life educational scenarios. Our code and data are available on request. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.