Publications
STUDY OBJECTIVES: This study aimed to compare YASA's automated sleep staging to manual staging in the context of a multi-night experimental sleep restriction protocol. METHODS: Seventy-five adults (58% female; 57% nonwhite) participated in up to seven nights of laboratory-based polysomnography measurements. The study involved one adaptation night, followed by three nights of normal sleep (9 h time in bed) and three nights of sleep restriction (5.5 h time in bed). Condition order was counterbalanced. Manual sleep staging was performed by a registered polysomnographic technician. Sleep data were exported and processed using Python 3.12, MNE 1.8.0, and YASA 0.6.5 for automated sleep staging. RESULTS: Across 483 valid sleep nights, there was 82.9% overall agreement between YASA scoring and manual scoring. Stage-specific agreement was 40.3% (N1), 85.1% (N2), 86.9% (N3), and 78.7% (REM). Agreement was higher during normal sleep nights than sleep restriction nights, particularly for wake, N1, and REM sleep classifications. CONCLUSIONS: YASA-based staging exhibited good overall agreement with manual scoring during normal sleep nights. However, caution is needed when interpreting N1 estimates, as well as in interpreting data during short sleep nights. CURRENT KNOWLEDGE/STUDY RATIONALE: Sleep stage scoring is typically conducted manually by at least one experienced technician, but this process is laborious and subject to biases. This study investigated whether a machine-learning-based open tool-YASA-could automatically stage polysomnography data with acceptable accuracy. STUDY IMPACT: Across 430,813 epochs, YASA showed acceptable accuracy in sleep staging, making it an efficient tool for the sleep community. However, caution is still needed for some applications, such as interpreting YASA's N1 estimates as well as data from short sleep nights.
This work-in-progress research paper explores Chat-Debugging, a novel use case for large language models as an assistant for hardware debugging tasks to improve students' debugging skills. Hardware debugging can be a time-consuming and stressful skill to develop, leading to frustration and other negative emotions. While past work has explored streamlining and automating software-based circuit debugging where digital circuits are dominant, Chat-Debugging aids in physical hardware debugging where circuits may be analog, digital, or mixed-signal. Qualitative data were collected from LLM chat logs and interviews with a fourth-year electrical engineering undergraduate student. Major themes were extracted using a constant comparative analysis. Chat-Debugging incorporates accurate hardware information, properly handles natural language descriptions of circuits, and improves debugging confidence. A successful Chat-Debugging session includes investigating multiple potential root causes proposed by the LLM, the patience and determination to eliminate root causes, and a student who leads the debugging process by assertively correcting the LLM's misunderstandings. This human-computer interaction can improve electrical and computer engineering students' confidence during debugging and improve their debugging skills.
Persistence after failure is critical for learning—but when students make mistakes in intelligent tutoring systems, they often choose not to try again. How can digital platforms encourage students to persist at these moments? We conducted a randomized controlled trial in an intelligent tutoring system for math and science, involving 164,532 students (Grades 8-12) who completed 17 million practice problems. We tested two scalable interventions: a brief persuasive prompt encouraging students to try again, and a visual default nudge that highlighted the retry option. Both interventions increased persistence after failure, and when combined, their effects were additive—suggesting they operate through distinct psychological mechanisms. The nudge had a much larger immediate effect, but the prompt showed proportionally greater spillover to untreated problems. These findings advance theories of persuasive design, demonstrating that implicit, interface-level nudges and explicit motivational prompts can be combined to avoid redundancy while amplifying impact.
People often seek out ways to watch others perform complex action sequences (e.g., sports). What makes some sequences more enjoyable to watch than others? We generated 24 video clips of gameplay from a Flappy Bird-style video game. Clips varied in difficulty (how often players succeeded on average) and in moment-to-moment uncertainty (how likely the player was to crash at any given step). Participants (N=864) rated each video on one of three dimensions: how much they enjoyed it, how difficult the level appeared, or how dangerous the player's trajectory appeared. We found that participants preferred videos where the player seemed to be completing more difficult obstacle courses, but dangerousness did not predict enjoyment ratings. These findings show how procedurally generated stimuli can isolate the factors that affect how enjoyable an action sequence is to watch.
Abstract General chemistry courses are prerequisites for many Science, Technology, Engineering, and Mathematics (STEM) majors. This reflects a, perhaps tacit, assumption that performance in introductory chemistry relates to one's suitability for a STEM career. Unfortunately, our field has little idea what is required to succeed in general chemistry courses across the United States. Therefore, we cannot evaluate if general chemistry grades are an appropriate proxy for one's potential in STEM. Here, we describe the intellectual work emphasized and rewarded on 235 exams administered in general chemistry courses at 57 colleges/universities across the USA. We parsed assessments into items, which might consist of multiple subparts, each with at least one embedded task. We focus on exams as they are ubiquitous in general chemistry courses and typically consequential for students' grades. First, we used the 3-Dimensional Learning Assessment Protocol to describe exam points allotted to items that prompt students to use core disciplinary ideas to make sense of phenomena (i.e., 3D Learning). Findings showed that prompting for 3D Learning on assessments was rare regardless of institution type, comprising less than 7% of exam points in our data set. Next, we developed and applied a protocol to describe tasks that did not have the potential to engage students in using core ideas in science practices (i.e., 0D tasks). This revealed that rote calculations were overemphasized comprising over 27% of all 0D tasks. Findings indicate that general chemistry courses gate access to STEM by student facility with rote calculations and drawing/labeling specialist pictures. We unpack why this is problematic and provide recommendations for building chemistry assessments that have the potential to be more meaningful and equitable.
Economic inequality is on the rise, and children will bear its burden. Yet children's perspectives are rarely considered. We urgently need interdisciplinary research to better understand how inequality gets into children's heads and under their skins, and to inform policies that center children's lived experiences. Economic inequality is on the rise, and children will bear its burden. Yet children's perspectives are rarely considered. We urgently need interdisciplinary research to better understand how inequality gets into children's heads and under their skins, and to inform policies that center children's lived experiences.
Measurement and assessment in Science, Technology, Engineering, and Mathematics (STEM) education is one central topic within STEM education scholarship. While there has been an increase in validation-related scholarship within STEM education, there are few guides for users to conduct validation work. Providing guidance for a broad readership, not just methodologists, offers potential for scholars from more backgrounds to engage in validation. To that end, the purpose of this paper is to build upon past scholarship and both articulate and situate validation as a methodology. Guideposts are provided to support readers as they engage in validation scholarship. A strategy is also provided to give readers support as they engage in validation scholarship. One key outcome from this paper is foundational work that scholars can leverage and extend, challenge, and generate new validation-related work, which in turn moves assessment practice and scholarship forward.
This study introduces InterDIFNet, a multilabel classification neural network for detecting intersectional differential item functioning (DIF) in educational and psychological assessments, with a focus on small sample sizes. Unlike traditional marginal DIF methods, which often fail to capture the effects of intersecting identities and require large samples, InterDIFNet models uniform and non-uniform DIF across multiple intersectional groups simultaneously and utilizes an optimized thresholding procedure to balance power and Type 1 error control. A Monte Carlo simulation compared InterDIFNet to the Truncated Lasso Penalty (TLP) test and other intersectional DIF methods across varying sample sizes, numbers of groups and DIF prevalence rates. Results show that when trained using TLP features, InterDIFNet consistently achieved higher power than TLP while maintaining comparable Type 1 error control, particularly in scenarios with three or more intersectional groups. An empirical application to real assessment data further demonstrated the method's practical utility. InterDIFNet provides a scalable, data-driven solution for identifying intersectional DIF across multiple small sample groups.
Research indicates that students learn effectively when they actively engage in planning, monitoring, and reflecting on their learning—a process known as self-regulated learning (SRL). This mixed-methods study examined the impact of a semester-long SRL intervention in an undergraduate physics course. In the first year, the instructor taught the course under standard conditions. The following year, after brief training sessions, the same instructor added digital prompts targeting various aspects of SRL to their course assignments. Results showed that students in this intervention group performed better on the final exam after controlling for preexisting self-efficacy, metacognitive awareness, and GPA. However, SRL outcomes did not increase in the intervention condition. Analysis of the prompts revealed that the instructor disproportionally focused on content- and management-related prompts and underutilized motivation, metacognition, and cognitive strategy prompts. Interview with the faculty indicated that the intervention was straightforward to implement, contributed to humanizing the course, and that additional training would have been desirable to help design prompts. Provided appropriate training and support structures are available, these findings show that integrating SRL strategies through digital prompts can be a practical and effective approach for supporting instructors in fostering students' SRL.
The present study examines the assistance dilemma of providing combinations of perceptual supports in math education by fitting a Bayesian knowledge tracing model (BKT) to an open-source, online order-of-operations intervention dataset. This online intervention dataset contained 688 U.S. middle schoolers' accuracy data who solved order-of-operations problems in pretest, intervention, post-test, and delayed post-test sessions. Combinations of spacing and color cues were embedded in intervention materials, alongside immediate correctness feedback and worked examples, creating nine between-subject cue conditions (congruent, incongruent, or neutral spacing and/or color). To examine the effect of perceptual cues on immediate performance and long term learning, we trained a BKT variant, conditioning guess, slip, learn, and forget parameters on perceptual cue conditions. The estimated cue-specific BKT parameters suggest two main findings. First, the presence of congruent perceptual cues (color and/or spacing, e.g., 2 - 8 + 3×11) increased guessing rate and reduced slipping rate, suggesting that visually highlighting high-order suboperations facilitated students' immediate problem solving. Second, students who underwent intervention with incongruent color cues (e.g., 2 - 8 + 3 × 11) showed reduced guessing rate but exhibited the highest learning rate and the lowest forgetting rate. These results imply that providing misleading visual highlights may hinder immediate performance but can be beneficial for long term learning. Furthermore, the impact of perceptual cues on immediate performance and long term learning indicated by BKT parameters approximately aligned with the patterns revealed by behavioral measures (i.e., intervention, post-test, and delayed post-test accuracy). Such consistencies demonstrate the feasibility of using the BKT framework to understand the assistance dilemma of perceptual supports. The code and data are available at: osf.io/q8kn6.


