Publications
and Context. The increasing integration of large language models (LLMs) in computing education presents an emerging challenge in understanding how students use LLMs and craft prompts to solve computational tasks. Prior research has used both qualitative and quantitative methods to analyze prompting behavior, but these approaches lack scalability or fail to effectively capture the semantic evolution of prompts.
Gender parity in academia remains elusive, even in fields, such as biological sciences, that award approximately half of their PhDs to women. Based on a sample of graduate students in biological sciences followed over 5 years during their PhD programs, we find that women publish in journals with lower impact factors and are less likely to publish in top journals. Notably, gender inequality in journal impact factors holds only for total publications but not for first-authored publications, which implies that differential opportunities for collaborations leading to high-impact publications may be one factor hindering gender equity.
Despite increased attempts to express equality in speech, biases often leak out through subtle linguistic cues. For example, the subject–complement statement (SCS, “Girls are as good as boys at math”) is used to advocate for equality but often reinforces gender stereotypes (boys are the standard against which girls are judged). We ask whether stereotypes conveyed by SCS can be counteracted by gesture. Two preregistered studies with 8- to 11-y-old children ( N = 320 total) investigate whether an equal gesture—two palms placed at the same height—mitigates the gender stereotype induced by SCS. Children who saw the equal gesture along with SCS were more likely to express egalitarian beliefs than children who saw no gesture or an unequal gesture. Children can extract meaning from gesture when making stereotypical inferences, suggesting that the equal gesture may prove to be an innovative, and simple, intervention to counteract stereotypes introduced by subtle language.
Teachers' hand movements shape how children learn mathematics, but not all movements are equally effective. Gesture-based instruction on problems like 4+2+5=__+5—where a teacher forms a V-shaped hand under the 4 and 2 and points to the blank (“grouping strategy”)—promotes learning more than action-based instruction, where the teacher manipulates magnetic numbers to mimic the gestures. How do gestures facilitate learning? We tested 8- to 10-year-old children (N = 76) using fNIRS to measure neural activity as they watched gesture-based or action-based videotaped lessons. Gesture-based instruction elicited greater intersubject neural synchrony in the motor cortex and angular gyrus, a region implicated in arithmetic processing. Critically, synchrony in the right angular gyrus during gesture instruction predicted learning gains, whereas synchrony during action-based instruction did not. Our findings indicate that gestures foster shared representations in brain regions supporting arithmetic reasoning, and highlight fNIRS as a powerful tool for capturing how learning unfolds in children.
Training gaze following models requires a large number of images with gaze target coordinates annotated by human annotators, which is a laborious and inherently ambiguous process. We propose the first semi-supervised method for gaze following by introducing two novel priors to the task. We obtain the first prior using a large pretrained Visual Question Answering (VQA) model, where we compute Grad-CAM heatmaps by 'prompting' the VQA model with a gaze following question. These heatmaps can be noisy and not suited for use in training. The need to refine these noisy annotations leads us to incorporate a second prior. We utilize a diffusion model trained on limited human annotations and modify the reverse sampling process to refine the Grad-CAM heatmaps. By tuning the diffusion process we achieve a trade-off between the human annotation prior and the VQA heatmap prior, which retains the useful VQA prior information while exhibiting similar properties to the training data distribution. Our method outperforms simple pseudo-annotation generation baselines on the GazeFollow image dataset. More importantly, our pseudo-annotation strategy, applied to a widely used supervised gaze following model (VAT), reduces the annotation need by 50%. Our method also performs the best on the VideoAttentionTarget dataset. Code is available at https://github.com/cvlab- stonybrook/GCDR-Gaze.git.
For computer systems to effectively interact with humans using spoken language, they need to understand how the words being generated affect the users' moment-by-moment attention. Our study focuses on the incremental prediction of attention as a person is seeing an image and hearing a referring expression defining the object in the scene that should be fixated by gaze. To predict the gaze scanpaths in this incremental object referral task, we developed the Attention in Referral Transformer model or ART, which predicts the human fixations spurred by each word in a referring expression. ART uses a multimodal transformer encoder to jointly learn gaze behavior and its underlying grounding tasks, and an autoregressive transformer decoder to predict, for each word, a variable number of fixations based on fixation history. To train ART, we created RefCOCO-Gaze, a large-scale dataset of 19,738 human gaze scanpaths, corresponding to 2,094 unique image-expression pairs, from 220 participants performing our referral task. In our quantitative and qualitative analyses, ART not only outperforms existing methods in scanpath prediction, but also appears to capture several human attention patterns, such as waiting, scanning, and verification. Code and dataset are available at: https://github.com/cvlab-stonybrook/ART.
Scientific reasoning is a critical foundational skill learners need to practice and know for increased science learning outcomes. Game-based learning environments (GBLEs) provide learners a platform for developing and practicing scientific reasoning skills but little is known about how learners should engage in scientific reasoning during game-based learning. As such, this paper aimed to understand if and how learners engaged in effective scientific reasoning activities during learning with a GBLE. This paper used analytical techniques from Complex Systems Theory to quantify learners’ scientific reasoning actions during game-based learning. High-school students (N = 170) played Crystal Island, a microbiology GBLE requiring learners to engage in scientific reasoning to successfully identify an illness infecting residents of a virtual island. Categorical auto-Recurrence Quantification Analysis was run on participants’ log files as they deployed scientific reasoning actions. This analysis revealed several metrics of complexity, including recurrence rate which is the proportion of repetitive to novel scientific reasoning actions. Results found that as time progressed, recurrence rates decreased. Successful learners (i.e., those who solved the mystery) demonstrated less repetition in their scientific reasoning activities where their recurrence rates decreased at a slower rate over time than learners who were unsuccessful in solving the mystery. Findings provide implications for adaptively scaffolding learners’ emerging complexities in their scientific reasoning processes during game-based learning to increase learners’ GBLE success. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
In order to rectify the historical marginalization of Black workers in computing fields, many academic and industrial initiatives have been established to increase the diversity and inclusion in the tech industry. In this study, we explored how one particular post-secondary education program at one historically black institution known as the Google in Residence (GIR) program impacted students' interests and their pursuit of computing-related careers. Data on employment outcomes at this institution showed that the GIR participants who obtained internships and full-time employment at Google were predominantly international students, not African American students who make up the majority of the undergraduate student and about half of the GIR participants. To understand this phenomenon better, we interviewed 27 students who spanned all 4 years of study and included African American and international students. The interviews were structured to elicit a wide range of factors influencing career decisions. Our analysis describes the impact of a particular GIR introductory-level programming course and other GIR program impacts beyond the introductory-level course. We found that the GIR introductory course was engaging, inspired students to change their degree in a more computing-oriented direction and supported internship applications. The informational workshops, networking events, mock-interviews, and advising of the computer science club, which were supported by the GIR program, built students' confidence in interviewing and obtaining full-time employment within the tech industry. While the GIR course and program had a positive influence on students of all backgrounds, the reputation of Google was especially motivating for international students. However, the African American students in this study generally had career aspirations and goals that were less tied to specific companies or industries. Based on our findings, potential improvements could include broadening awareness and access of the GIR course to non-computer science majors, offering summer programs to increase students' computational confidence, and increasing the diversity of GIR and computer science instructors.
Generative AI systems, especially Large Language Models (LLMs) such as ChatGPT, have recently emerged as significant contributors to creative processes. While LLMs can produce creative content that might be as good as or even better than human-created content, their widespread use risks reducing creative diversity across groups of people. In the present research, we aimed to quantify this homogenizing effect of LLMs on creative diversity, not only at the individual level but also at the collective level. Across three preregistered studies, we analyzed 2,200 college admissions essays. Using a novel measure—the diversity growth rate—we showed that each additional human-written essay contributed more new ideas than did each additional GPT-4 essay. Notably, this difference became more pronounced as more essays were included in the analysis and persisted despite efforts to enhance AI-generated content through both prompt and parameter modifications. Overall, our findings suggest that, despite their potential to enhance individual creativity, the widespread use of LLMs could diminish the collective diversity of creative ideas.


