Publications
In this article, the Funding information section was missing and should have read ‘The study is funded by IES R305D200015 and NSF EDU-CORE #2300382’. The original article has been corrected. © The Author(s), under exclusive licence to The Psychometric Society 2024.
Metaphor is crucial in human cognition and creativity, facilitating abstract thinking, analogical reasoning, and idea generation. Typically, human raters manually score the originality of responses to creative thinking tasks - a laborious and error-prone process. Previous research sought to remedy these risks by scoring creativity tasks automatically using semantic distance and large language models (LLMs). Here, we extend research on automatic creativity scoring to metaphor generation - the ability to creatively describe episodes and concepts using nonliteral language. Metaphor is arguably more abstract and naturalistic than prior targets of automated creativity assessment. We collected 4,589 responses from 1,546 participants to various metaphor prompts and corresponding human creativity ratings. We fine-tuned two open-source LLMs (RoBERTa and GPT-2) - effectively teaching them to score metaphors like humans - before testing their ability to accurately assess the creativity of new metaphors. Results showed both models reliably predicted new human creativity ratings (RoBERTa r = .72, GPT-2 r = .70), significantly more strongly than semantic distance (r = .42). Importantly, the fine-tuned models generalized accurately to metaphor prompts they had not been trained on (RoBERTa r = .68, GPT-2 r = .63). We provide open access to the fine-tuned models, allowing researchers to assess metaphor creativity in a reproducible and timely manner.
Modern assessment demands, resulting from educational reform efforts, call for strengthening diagnostic testing capabilities to identify not only the understanding of expected learning goals but also related intermediate understandings that are steppingstones on pathways to learning goals. An accurate and nuanced way of interpreting assessment results will allow subsequent instructional actions to be targeted. An appropriate psychometric model is indispensable in this regard. In this study, we developed a new psychometric model, namely, the diagnostic facet status model (DFSM), which belongs to the general class of cognitive diagnostic models (CDM), but with two notable features: (1) it simultaneously models students’ target understanding (i.e., goal facet) and intermediate understanding (i.e., intermediate facet); and (2) it models every response option, rather than merely right or wrong responses, so that each incorrect response uniquely contributes to discovering students’ facet status. Given that some combination of goal and intermediate facets may be impossible due to facet hierarchical relationships, a regularized expectation–maximization algorithm (REM) was developed for model estimation. A log-penalty was imposed on the mixing proportions to encourage sparsity. As a result, those impermissible latent classes had estimated mixing proportions equal to 0. A heuristic algorithm was proposed to infer a facet map from the estimated permissible classes. A simulation study was conducted to evaluate the performance of REM to recover facet model parameters and to identify permissible latent classes. A real data analysis was provided to show the feasibility of the model. © The Author(s), under exclusive licence to The Psychometric Society 2024. corrected publication 2024.
A growing body of literature indicates strong associations between indentations of the cerebral cortex (i.e., sulci) and individual differences in cognitive performance. Interruptions, or gaps, of sulci (historically known as pli de passage) are particularly intriguing as previous work suggests that these interruptions have a causal effect on cognitive development. Here, we tested how the presence and morphology of sulcal interruptions in the left posterior occipitotemporal sulcus (pOTS) longitudinally impact the development of a culturally-acquired skill: reading. Forty-three children were successfully followed from age 5 in kindergarten, at the onset of literacy instruction, to ages 7 and 8 with assessments of cognitive, pre-literacy, and literacy skills, as well as MRI anatomical scans at ages 5 and 8. Crucially, we demonstrate that the presence of a left pOTS gap at 5 years is a specific and robust longitudinal predictor of better future reading skills in children, with large observed benefits on reading behavior ranging from letter knowledge to reading comprehension. The effect of left pOTS interruptions on reading acquisition accumulated through time, and was larger than the impact of benchmark cognitive and familial predictors of reading ability and disability. Finally, we show that increased local U-fiber white matter connectivity associated with such sulcal interruptions possibly underlie these behavioral benefits, by providing a computational advantage. To our knowledge, this is the first quantitative evidence supporting a potential integrative gray-white matter mechanism underlying the cognitive benefits of macro-anatomical differences in sulcal morphology related to longitudinal improvements in a culturally-acquired skill.
We present CPSCoach 2.0, an automated system that provides feedback, instructional scaffolding, and practice to help individuals improve three collaborative problem-solving (CPS) skills drawn from a theoretical CPS framework: construction of shared knowledge, negotiation/coordination, and maintaining team function. CPSCoach 2.0 was developed and tested in the context of computer-mediated collaboration (video conferencing) with an educational game. It automatically analyzes users' speech during a round of collaborative gameplay to provide personalized feedback and to select a target CPS skill for improvement. After multiple cycles of iterative testing and refinement, we tested CPSCoach 2.0 in a user study where 21 dyads (n = 42) completed four rounds of feedback and scaffolding embedded within five rounds of game-play in a single session. Using a quasi-experimental matching procedure, we found that the use of CPSCoach 2.0 was associated with improvement in CPS skill development compared to matched controls. Further, users found the automated feedback to be moderately accurate and had positive perceptions of the system, and these impressions were stronger for those who received higher scores overall. Results demonstrate the use of automated feedback and instructional scaffolds to support the development of CPS skills.
Causal reasoning is a fundamental cognitive ability that enables individuals to learn about the complex interactions in the world around them. However, the mechanisms that underpin causal reasoning are not well understood. For example, it remains unresolved whether children's causal inferences are best explained by Bayesian inference or associative learning. The two experiments and computational models reported here were designed to examine whether 5‐ and 6‐year‐olds will retrospectively reevaluate objects—that is, adjust their beliefs about the causal status of some objects presented at an earlier point in time based on the observed causal status of other objects presented at a later point in time—when asked to reason about 3 and 4 objects and under varying degrees of information processing demands. Additionally, the experiments and models were designed to determine whether children's retrospective reevaluations were best explained by associative learning, Bayesian inference, or some combination of both. The results indicated that participants retrospectively reevaluated causal inferences under minimal information‐processing demands (Experiment 1) but failed to do so under greater information processing demands (Experiment 2) and that their performance was better captured by an associative learning mechanism, with less support for descriptions that rely on Bayesian inference. Research Highlights
Five‐ and 6‐year‐old children engage in retrospective reevaluation under minimal information‐processing demands (Experiment 1). Five‐ and 6‐year‐old children do not engage in retrospective reevaluation under more extensive information‐processing demands (Experiment 2). Across both experiments, children's retrospective reevaluations were better explained by a simple associative learning model, with only minimal support for a simple Bayesian model. These data contribute to our understanding of the cognitive mechanisms by which children make causal judgements.
There is limited time for elementary teacher professional learning in science in order to meet the aspirational goals of current reform efforts. In this study, we investigated what and how teachers learn on-the-job to gain insight into modes of support less often included in teacher education design. Specifically, we studied elementary teachers' participation in a system-level organizational routine: curriculum materials adoption processes (CMAPs). Using a communities of practice framework, we explored teacher learning in a comparative case study of three U.S. school districts' CMAP routines, observing CMAP committee meetings and interviewing participants about their expeirences. We found that what teachers learned varied across each district's CMAP. We argue this variation can be traced to two CMAP features: (1) teachers' use of boundary objects and (2) their boundary spanning roles and structures. Results have implications for the design of educational systems' organizational routines to more intentionally serve a dual role as a professional learning opportunity.
The ability to associate different types of number representations referring to the same quantity (symbolic Arabic numerals, signed/spoken number words, and nonsymbolic quantities), is an important predictor of overall mathematical success. This foundational skill-mapping-has not been examined in deaf and hard-of-hearing (DHH) children. To address this gap, we studied 188 4 1/2 to 9-year-old DHH and hearing children and systematically examined the relationship between their language experiences and mapping skills. We asked whether the timing of children's language exposure (early vs. later), the modality of their language (signed vs. spoken), and their rote counting abilities related to mapping performance. We found that language modality did not significantly relate to mapping performance, but timing of language exposure and counting skills did. These findings suggest that early access to language, whether spoken or signed, supports the development of age-typical mapping skills and that knowledge of number words is critical for this development.
The present study tested the learning avoidance model by examining the degree to which learning avoidance in various afterschool settings mediated the negative association between math anxiety and math achievement. Participants consisted of 207 third to sixth graders. Using a path model, findings showed that students’ math anxiety was negatively associated with both standardized math achievement test scores and parent-reported math school grades. Additionally, higher math anxiety was associated with more negative homework behaviors and less frequent participation in math-related extracurricular activities. Finally, the association between math anxiety and math achievement was partially mediated by negative math homework behaviors and participation in math extracurricular activities. Effort in math exam preparation did not contribute to explaining the association between math anxiety and math achievement. Overall, these findings support the learning avoidance model and suggest that avoidance behaviors in everyday learning in the afterschool setting may contribute to explaining the undesired math achievement among highly math anxious students.
Teachers' growth mindset supportive language (GMSL)--rhetoric emphasizing that one's skills can be improved over time--has been shown to significantly reduce disparities in academic achievement and enhance students' learning outcomes. Although teachers espouse growth mindset principles, most find it difficult to adopt GMSL in their practice due the lack of effective coaching in this area. We explore whether large language models (LLMs) can provide automated, personalized coaching to support teachers' use of GMSL. We establish an effective coaching tool to reframe unsupportive utterances to GMSL by developing (i) a parallel dataset containing GMSL-trained teacher reframings of unsupportive statements with an accompanying annotation guide, (ii) a GMSL prompt framework to revise teachers' unsupportive language, and (iii) an evaluation framework grounded in psychological theory for evaluating GMSL with the help of students and teachers. We conduct a large-scale evaluation involving 174 teachers and 1,006 students, finding that both teachers and students perceive GMSL-trained teacher and model reframings as more effective in fostering a growth mindset and promoting challenge-seeking behavior, among other benefits. We also find that model-generated reframings outperform those from the GMSL-trained teachers. These results show promise for harnessing LLMs to provide automated GMSL feedback for teachers and, more broadly, LLMs' potentiality for supporting students' learning in the classroom. Our findings also demonstrate the benefit of large-scale human evaluations when applying LLMs in educational domains.


