Publications
We explore the use of small language models (SLMs) for automatic question generation as a complement to the prevalent use of their large counterparts in learning analytics research. We present a novel question generation pipeline that leverages both the text generation and the probabilistic reasoning abilities of SLMs to generate high-quality questions. Adopting a “generate-then-validate” strategy, our pipeline first performs expansive generation to create an abundance of candidate questions and refine them through selective validation based on novel probabilistic reasoning. We conducted two evaluation studies, one with seven human experts and the other with a large language model (LLM), to assess the quality of the generated questions. Most judges (humans or LLMs) agreed that the generated questions had clear answers and generally aligned well with the intended learning objectives. Our findings suggest that an SLM can effectively generate high-quality questions when guided by a well-designed pipeline that leverages its strengths.
Persistence after failure is critical for learning—but when students make mistakes in intelligent tutoring systems, they often choose not to try again. How can digital platforms encourage students to persist at these moments? We conducted a randomized controlled trial in an intelligent tutoring system for math and science, involving 164,532 students (Grades 8-12) who completed 17 million practice problems. We tested two scalable interventions: a brief persuasive prompt encouraging students to try again, and a visual default nudge that highlighted the retry option. Both interventions increased persistence after failure, and when combined, their effects were additive—suggesting they operate through distinct psychological mechanisms. The nudge had a much larger immediate effect, but the prompt showed proportionally greater spillover to untreated problems. These findings advance theories of persuasive design, demonstrating that implicit, interface-level nudges and explicit motivational prompts can be combined to avoid redundancy while amplifying impact.
Abstract The ability to generalize and apply knowledge to new situations is the hallmark of effective learning. There are multiple ways to improve learning towards generalization. Retrieval practice and worked examples have both been shown to improve learning, retention, and generalization. The literature on category learning also suggests that exposure to more varied examples, rather than the same example repeatedly, improves generalization. However, while retrieval practice is usually investigated with repeated items, worked examples are usually investigated using varied content. Across two experiments, we investigated how combining content variability with worked examples or retrieval practice promoted generalization when learners received initial instruction before practice (Experiment 1) and when they did not (Experiment 2). We found that when learners received initial instruction, retrieval practice led to better generalization than worked examples, regardless of whether the stimuli were repeated or varied. When initial instruction was not provided, worked examples resulted in better generalization than retrieval practice when items were repeated. However, retrieval practice produced comparable generalization to worked examples when items were varied. These findings suggest that variability supports generalization, particularly when learners must induce underlying rules, and that retrieval practice, traditionally associated with memory retention, can also facilitate inductive learning when applied to varied items. This study emphasizes the importance of adapting learning strategies to align with the instructional context and task demands.
Cognitive science of learning points to solutions for making use of existing study and instruction time more effectively and efficiently. However, solutions are not and cannot be one-size-fits-all. This paper outlines the danger of overreliance on specific strategies as one-size-fits-all recommendations and highlights instead the cognitive learning processes that facilitate meaningful and long-lasting learning. Three of the most commonly recommended strategies from cognitive science provide a starting point; understanding the underlying processes allows us to tailor these recommendations to implement at the right time, in the right way, for the right content, and for the right students. Recommendations regard teacher training, the funding and incentivizing of educational interventions, guidelines for the development of educational technologies, and policies that focus on using existing instructional time more wisely. © The Author(s) 2023.
What conditions are necessary for students to learn from practice and feedback, without the need for upfront lecture? Across two experiments (N = 597), we examined how practice with feedback supports memory, generalization, metacognition, and motivation. Participants were randomly assigned to one of three instructional formats: a traditional lecture, practice with correct-answer feedback, or practice with explanatory feedback. In both studies, the lecture condition introduced linear regression through definitions and a worked example, while the practice conditions used matched problem sets with feedback that either (a) provided only correct answers or (b) explained why answers were correct. In Study 1, practice questions were multiple-choice; in Study 2, both practice and posttest items were open-ended, and personalized explanatory feedback was generated in real time using GPT-4o. For memory, both types of feedback outperformed lecture, suggesting that attempting a response and receiving feedback—even without explanations—enhances encoding. For generalization, however, feedback needed to include explanations, and learners needed sufficient prior knowledge to benefit. Study 2 also showed that practice—regardless of feedback type—improved metacognitive calibration compared to lecture, helping learners more accurately assess their understanding. While lecture produced greater situational interest for less-confident learners in Study 1, this pattern reversed in Study 2, where personalized, explanatory feedback elicited higher interest for this group. Together, these findings clarify when and for whom practice with feedback can replace lecture-based instruction, and they highlight the potential of generative A.I. to scale personalized, explanatory feedback.
Many experiments have demonstrated that more practice testing and less studying of examples produces better learning whereas other experiments show the opposite, that more example study and less practice produces better learning. We present empirical and computational evidence that resolves and explains this apparent inconsistency. We show how a practice testing instructional treatment supports memory learning processes needed for verbatim fact content whereas an example-integrated instructional treatment supports inductive learning processes needed for general skill content. In an experiment comparing both instructional treatments on both types of content, we observe a cross-over interaction such that participants learn verbatim facts better from pure practice testing but learn general skills better from example-integrated practice. We use a computational learning architecture, AL, to create an executable theory that explains and predicts these results. Simulated students developed in AL interactively learn from the materials provided in the same four conditions as human learners and reproduce the same cross-over interaction. We further demonstrate that the benefit of integrated examples for general skill learning are a result of, and thus explained by, AL's inductive learning mechanisms whereas the benefit of practice for verbatim fact learning result from AL's memory learning mechanisms.
Spaced retrieval practice has been repeatedly demonstrated to improve learning, but its implementation is often constrained by real-world time limitations. This study investigated whether, under fixed study durations, learners should prioritize spacing or repetition. Across two experiments (total N = 1589), participants practiced Indonesian vocabulary under four conditions that varied in spacing and repetition. Item difficulty was also manipulated. Results showed that increasing repetitions at the cost of spacing enhanced immediate test performance, particularly for harder items. These findings suggest that spaced retrieval practice is effective only when learners have sufficient prior repetitions to retrieve information successfully. This study highlights the trade-offs between spacing and repetition under time constraints and offers practice insights for optimizing learning strategies.
Decades of research show that tests, beyond assessing student knowledge, are powerful tools for promoting learning. However, high-stakes tests can also cause stress and disengagement. To utilize tests to encourage and motivate students, we implemented a mastery-based testing system in a large-enrollment general chemistry course (N = 234). This system allowed students to take three versions of each unit test, studying digital course resources in between to increase their mastery of the content. Students chose to take advantage of the mastery testing system when they struggled with unit tests, averaging six total repeated attempts. This level of repeated testing was associated with a 60% increase in students' use of online study resources over the duration of the course, and a five-point overall increase in final exam scores (11 points for first-generation college students). Critically, student engagement with digital learning materials mediated these performance gains, suggesting that the benefits of mastery-based testing systems were not only due to students responding to the tests themselves. Instead, the findings suggest that mastery-based testing systems can enhance performance in introductory STEM courses by providing motivation and structure to support students' self-regulated learning, helping them invest more time in effective, distributed study strategies. • With mastery-based testing, students chose to repeat exams when they struggled • Repeating tests was linked to an increase in use of online study resources • Repeating tests was linked to higher final exam grades, especially for first-gen students • Mastery-based testing may support self-regulated learning in STEM courses
Cognitive science of learning points to solutions for making use of existing study and instruction time more effectively and efficiently. However, solutions are not and cannot be one-size-fits-all. This paper outlines the danger of overreliance on specific strategies as one-size-fits-all recommendations and highlights instead the cognitive learning processes that facilitate meaningful and long-lasting learning. Three of the most commonly recommended strategies from cognitive science provide a starting point; understanding the underlying processes allows us to tailor these recommendations to implement at the right time, in the right way, for the right content, and for the right students. Recommendations regard teacher training, the funding and incentivizing of educational interventions, guidelines for the development of educational technologies, and policies that focus on using existing instructional time more wisely. © The Author(s) 2023.
The development of intelligent tutoring systems and other educational technology necessitated the implementation and development of computational models of student learning. At the foundation of these models is the assumption that they accurately implement and track human cognitive processes. However, the extent to which this assumption is correct has not been fully tested. In the current paper, we use data from a large-scale longitudinal lab study to investigate the match between the processes instantiated in the models and human memory and learning processes. When fitted to the entire dataset retrospectively, the selected models of student learning (Bayesian Knowledge Tracing and Additive Factors Model) capture the qualitative trends of learning across sessions and relatively acceptable fit metrics. However, when the models are used to predict future behavior, as is often the goal in applied contexts, the picture changes considerably. We show that two of the most popular types of student learning models fail to account for basic cognitive principles—the spacing effect, and patterns of forgetting and learning across sessions. In fact, in some instances, having a poor model of human learning and memory may be worse than having no model of human learning and memory at all. At the same time, cognitive approaches can and should greatly benefit from computational modeling approaches; computational modeling offers opportunities for furthering both theory and practical impact.


