Gaze following aims to predict where a person is looking in a scene, by predicting the target location, or indicating that the target is located outside the image. Recent works detect the gaze target by training a heatmap regression task with a pixel-wise mean-square error (MSE) loss, while formulating the in/out prediction task as a binary classification task. This training formulation puts a strict, pixel-level constraint in higher resolution on the single annotation available in training, and does not consider annotation variance and the correlation between the two subtasks. To address these issues, we introduce the patch distribution prediction (PDP) method. We replace the in/out prediction branch in previous models with the PDP branch, by predicting a patch-level gaze distribution that also considers the outside cases. Experiments show that our model regularizes the MSE loss by predicting better heatmap distributions on images with larger annotation variances, meanwhile bridging the gap between the target prediction and in/out prediction subtasks, showing a significant improvement in performance on both subtasks on public gaze following datasets.

Learning using Computer-Assisted Instruction (CAI) demands a high level of attention given the tendency to be distracted and mind-wander. How does the online STEM instructor know when learners are having attentional problems and the extent to which these problems affect learning? In the present study, the visual attentional and cognitive state of physics graduate students were probed while they went through a multimedia instructional module to refresh their knowledge of Newton's II Law. Data from an eye tracker, webcam, egocentric glasses, screen recording, and mouse and keyboard events were integrated to record learners' attention overt attention to the learning environment (+/-) and thinking about learning content (+/-) to analyze students' attention spans during learning from this module. On average, learners were found to be on-task and on-screen for a vast majority of time, with evidence of mind wandering. The learning module improved the participants efficiency with which they answered the questions correctly on a post-test relative to the pre-test. Further, there is a positive albeit statistically non-significant correlation between the improvement from pre- to post-test efficiency and the time spent on-screen and on-task during the module.

Predicting human gaze is important in Human-Computer Interaction (HCI). However, to practically serve HCI applications, gaze prediction models must be scalable, fast, and accurate in their spatial and temporal gaze predictions. Recent scanpath prediction models focus on goaldirected attention (search). Such models are limited in their application due to a common approach relying on trained target detectors for all possible objects, and the availability of human gaze data for their training (both not scalable). In response, we pose a new task called ZeroGaze, a new variant of zero-shot learning where gaze is predicted for never-before-searched objects, and we develop a novel model, Gazeformer; to solve the ZeroGaze problem. In contrast to existing methods using object detector modules, Gazeformer encodes the target using a natural language model, thus leveraging semantic similarities in scanpath prediction. We use a transformer-based encoder-decoder architecture because transformers are particularly useful for generating contextual representations. Gazeformer surpasses other models by a large margin (19%-70%) on the ZeroGaze setting. It also outperforms existing target-detection models on standard gaze prediction for both target-present and target-absent search tasks. In addition to its improved performance, Gazeformer is more than five times faster than the state-of-the-art target-present visual search model. Code can be found at https://github.com/cvlab-stonybrook/Gazeformer/

Research has shown that students' responses to conceptual questions correlate with their eye movements.However, to what extent is it possible to predict whether a particular learner might answer a question correctly by monitoring their eye movements in real time?To answer this question, we used spatialtemporal eye-movement data from about 400 participants, as well as their responses to four conceptual physics questions with diagrams.Half of these data were used as a training set for a machine learning algorithm (MLA) that would predict the correctness of students' responses to these questions.The other half of the data were used as a test set to determine the performance of the MLA in terms of the accuracy of the prediction.We will discuss the results of our study with specific attention to the prediction accuracy of the MLA under different conditions.