Eye movements have long been used to reflect ongoing cognitive processing to develop explanatory and predictive models of mental states and processes. This relationship, deemed the eye-mind link, contains underlying assumptions of the mental processes occurring in the brain, which have rarely been explicitly investigated. We propose a multimodal approach to investigate alignment of eye movements and brain activations (eye-brain alignment) and how it might be predicted by unfolding cognitive processes. We applied this method to a dataset of 76 participants who read long, connected texts while their eye movements and the hemodynamic responses in their brains were tracked using functional near-infrared spectroscopy (fNIRS). We found that reliable eye-brain alignment signals varied based on the participants’ cognitive state during reading. Implications for multimodal modeling of cognitive processes are discussed. © 2024 Copyright held by the owner/author(s).

Engagement is critical to satisfaction and performance in a number of domains but is challenging to measure and sustain. Thus, there is considerable interest in developing affective computing technologies to automatically measure and enhance engagement, especially in the wild and at scale. This article provides an accessible introduction to affective computing research on engagement detection and enhancement using educational applications as an application domain. We begin with defining engagement as a multicomponential construct (i.e., a conceptual entity) situated within a context and bounded by time and review how the past six years of research has conceptualized it. Next, we examine traditional and affective computing methods for measuring engagement and discuss their relative strengths and limitations. Then, we move to a review of proactive and reactive approaches to enhancing engagement toward improving the learning experience and outcomes. We underscore key concerns in engagement measurement and enhancement, especially in digitally enhanced learning contexts, and conclude with several open questions and promising opportunities for future work. © 1963-2012 IEEE.

We examined associations between mind wandering - where attention shifts from the task at hand to task-unrelated thoughts - and learning outcomes. Our data consisted of 177 students who self-reported mind wandering while reading five long, connected texts on scientific research methods and completed learning assessments targeting multiple depths of processing (rote, inference, integration) at different timescales (during and after reading each text, after reading all texts, and after a week-long delay). We found that mind wandering negatively predicted measures of factual, text-based (explicit) information and global integration of information across multiple parts of the text, but not measures requiring a local inference on a single sentence. Further, mind wandering only predicted comprehension measures assessed during the reading session and not after a week-long delay. Our findings provide important nuances to the established negative link between mind wandering and learning outcomes, which has predominantly focused on rote comprehension assessed during the learning session itself. Implications for interventions to address mind wandering during learning are discussed. © 2023 Owner/Author.

Abstract Functional magnetic resonance imaging (fMRI) has provided unparalleled insights into the fundamental neural mechanisms governing human cognition, including complex processes such as reading. Here, we leverage the wealth of prior fMRI work to capture reading outside the MRI scanner using functional near infra-red spectroscopy (fNIRS). In a large sample of participants (n = 82) we observe significant prefrontal and temporal fNIRS activations during reading, which are largely reliable across participants, therefore providing a robust validation of prior fMRI work on reading-related language processing. These results lay the groundwork towards developing adaptive systems capable of assisting these higher-level processes, for example to support readers and language learners. This work also contributes to bridging the gap between laboratory findings and real-world applications in the realm of cognitive neuroscience.

In this work, we investigate how two factors, metacognitive skills and motivation, would impact student learning across domains. More specifically, our primary goal is to identify the critical, yet robust, interaction patterns of these two factors that would contribute to students' performance in learning logic first and then their performance on a subsequent new domain, probability. We are concerned with two types of metacognitive skills: strategy-awareness and time-awareness, that is, which problem-solving strategy to use and when to use it. Our data were collected from 495 participants across three consecutive semesters, and our results show that the only students who consistently outperform their peers across both domains are those who are not only highly motivated but also strategy-aware and time-aware.

Eye gaze patterns can reveal user attention, reading fluency, corrective responding, and other reading processes, suggesting they can be used to develop automated, real-time assessments of comprehension. However, past work has focused on modeling factual comprehension, whereas we ask whether gaze patterns reflect deeper levels of comprehension where inferencing and elaboration are key. We trained linear regression and random forest models to predict the quality of users' open-ended self-explanations (SEs) collected both during and after reading and scored on a continuous scale by human raters. Our models use theoretically grounded eye tracking features (number and duration of fixations, saccade distance, proportion of regressive and horizontal saccades, spatial dispersion of fixations, and reading time) captured from a remote, head-free eye tracker (Tobii TX300) as adult users read a long expository text (6500 words) in two studies (N = 106 and 131; 247 total). Our models: (1) demonstrated convergence with human-scored SEs (r = .322 and .354), by capturing both within-user and between-user differences in comprehension; (2) were distinct from alternate models of mind-wandering and shallow comprehension; (3) predicted multiple-choice posttests of inference-level comprehension (r = .288, .354) measured immediately after reading and after a week-long delay beyond the comparison models; and (4) generalized across new users and datasets. Such models could be embedded in digital reading interfaces to improve comprehension outcomes by delivering interventions based on users' level of comprehension.

What can eye movements reveal about reading, a complex skill ubiquitous in everyday life? Research suggests that gaze can reflect short-term comprehension for facts, but it is unknown whether it can measure long-term, deep comprehension. We tracked gaze while 147 participants read long, connected, informative texts and completed assessments of rote (factual) and inference comprehension (connecting ideas) while reading a text, after reading a text, after reading five texts, and after a seven-day delay. Gaze-based student-independent computational models predicted both immediate and long-term rote and inference comprehension with moderate accuracies. Surprisingly, the models were most accurate for comprehension assessed after reading all texts and predicted comprehension even after a week-long delay. This shows that eye movements can provide a lens into the cognitive processes underlying reading comprehension, including inference formation, and the consolidation of information into long-term memory, which has implications for intelligent student interfaces that can automatically detect and repair comprehension in real-time. © 2022 Copyright is held by the author(s).

The ability to identify whether a user is zoning out (mind wandering) from video has many HCI (e.g., distance learning, high-stakes vigilance tasks). However, it remains unknown how well humans can perform this task, how they compare to automatic computerized approaches, and how a fusion of the two might improve accuracy. We analyzed videos of users' faces and upper bodies recorded 10s prior to self-reported mind wandering (i.e., ground truth) while they engaged in a computerized reading task. We found that a state-of-the-art machine learning model had comparable accuracy to aggregated judgments of nine untrained human observers (area under receiver operating characteristic curve [AUC] =.598 versus .589). A fusion of the two (AUC = .644) outperformed each, presumably because each focused on complementary cues. Furthermore, adding more humans beyond 3-4 observers yielded diminishing returns. We discuss implications of human-computer fusion as a means to improve accuracy in complex tasks.

Eye movements provide a window into cognitive processes, but much of the research harnessing this data has been confined to the laboratory. We address whether eye gaze can be passively, reliably, and privately recorded in real-world environments across extended timeframes using commercial-off-the-shelf (COTS) sensors. We recorded eye gaze data from a COTS tracker embedded in participants (N=20) work environments at pseudorandom intervals across a two-week period. We found that valid samples were recorded approximately 30% of the time despite calibrating the eye tracker only once and without placing any other restrictions on participants. The number of valid samples decreased over days with the degree of decrease dependent on contextual variables (i.e., frequency of video conferencing) and individual difference attributes (e.g., sleep quality and multitasking ability). Participants reported that sensors did not change or impact their work. Our findings suggest the potential for the collection of eye-gaze in authentic environments. © 2022 ACM.

Eye tracking has been a research tool for decades, providing insights into interactions, usability, and, more recently, gaze-enabled interfaces. Recent work has utilized consumer-grade and webcam-based eye tracking, but is limited by the need to repeatedly calibrate the tracker, which becomes cumbersome for use outside the lab. To address this limitation, we developed an unsupervised algorithm that maps gaze vectors from a webcam to fxation features used for user modeling, bypassing the need for screen-based gaze coordinates, which require a calibration process. We evaluated our approach using three datasets (N=377) encompassing diferent UIs (computerized reading, an Intelligent Tutoring System), environments (laboratory or the classroom), and a traditional gaze tracker used for comparison. Our research shows that webcam-based gaze features correlate moderately with eye-tracker-based features and can model user engagement and comprehension as accurately as the latter. We discuss applications for research and gaze-enabled user interfaces for long-term use in the wild.