We investigated the feasibility of using eye gaze to model collaborative problem solving (CPS) behaviors in 96 triads (N= 288) who used videoconferencing to remotely collaborate on an educational physics game. Trained human raters coded spoken utterances based on a theoretically grounded framework consisting of three CPS facets: constructing shared knowledge, negotiation/coordination, and maintaining team function. We then trained random forest classifiers to identify each of the CPS facets using eye gaze features pertaining to each individual (e.g., number of fixations) and/or shared across individuals (e.g., eye gaze distance between collaborators) in conjunction with information about the unfolding task context. We found that the individual gaze features outperformed the shared features, and together yielded between 6% to 8% improvements in classification accuracy above task-context baseline models, using a cross-validation scheme that generalized across teams. We discuss how our findings support CPS theories and the development of real-time intervention systems that provide actionable feedback to improve collaboration.

Whereas social visual attention has been examined in computer-mediated (e.g., shared screen) or video-mediated (e.g., FaceTime) interaction, it has yet to be studied in mixed-media interfaces that combine video of the conversant along with other UI elements. We analyzed eye gaze of 37 dyads (74 participants) who were tasked with negotiating the price of a new car (as a buyer and seller) using mixed-media video conferencing under competitive or cooperative negotiation instructions (experimental manipulation). We used multidimensional recurrence quantification analysis to extract spatio-temporal patterns corresponding to mutual gaze (individuals look at each other), joint attention (individuals focus on the same elements of the interface), and gaze aversion (an individual looks at their partner, who is looking elsewhere). Our results indicated that joint attention predicted the sum of points attained by the buyer and seller (i.e., the joint score). In contrast, gaze aversion was associated with faster time to complete the negotiation, but with a lower joint score. Unexpectedly, mutual gaze was highly infrequent and unrelated to the negotiation outcomes and none of the gaze patterns predicted subjective perceptions of the negotiation. There were also no effects of gender composition or negotiation condition on the gaze patterns or negotiation outcomes. Our results suggest that social visual attention may operate differently in mixed-media collaborative interfaces than in face-to-face interaction. As mixed-media collaborative interfaces gain prominence, our work can be leveraged to inform the design of gaze-sensitive user interfaces that support remote negotiations among other tasks.

Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from.53 to.83, with increases in model accuracy when language information was included (AUROCS from.72 to.86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of.86,.78, and.79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes. © 2021, The Author(s), under exclusive licence to Springer Nature B.V. part of Springer Nature.

Little quantitative research has explored which clinician skills and behaviors facilitate communication. Mutual understanding is especially challenging when patients have limited health literacy (HL). Two strategies hypothesized to improve communication include matching the complexity of language to patients' HL (universal tailoring); or always using simple language (universal precautions). Through computational linguistic analysis of 237,126 email exchanges between dyads of 1094 physicians and 4331 English-speaking patients, we assessed matching (concordance/discordance) between physicians' linguistic complexity and patients' HL, and classified physicians' communication strategies. Among low HL patients, discordance was associated with poor understanding (P = 0.046). Physicians' universal tailoring strategy was associated with better understanding for all patients (P = 0.01), while universal precautions was not. There was an interaction between concordance and communication strategy (P = 0.021): The combination of dyadic concordance and universal tailoring eliminated HL-related disparities. Physicians' ability to adapt communication to match their patients' HL promotes shared understanding and equity. The 'Precision Medicine' construct should be expanded to include the domain of 'Precision Communication.'

As remote learning technologies play an increasingly larger role in education, clear evidence of effectiveness is needed for widely used online learning technologies, such as Assessment and LEarning in Knowledge Spaces (ALEKS). By adapting to individual students' knowledge states and personalizing interactive practice and feedback, ALEKS may potentially support learning in mathematics, which is foundational for success in science, technology, engineering, and mathematics fields. Given the mixed findings about the effects of ALEKS in previous research, we conducted a meta-analysis to examine if ALEKS was as effective as traditional instruction in mathematics education. Our analysis included 56 independent effect sizes obtained from 9,238 students in K-12 schools and institutions of higher education participating in 33 research studies between 2000 and August 2020. Results indicated that learning performance with ALEKS was comparable to that with traditional instruction (Hedge's g = 0.05, 95% CI [−0.01, 0.20]) and ALEKS was especially effective when used to supplement traditional instruction (g = 0.43, 95% CI [0.02, 0.83]). These findings have important implications for using ALEKS in mathematics learning across different levels of education.

The low cost of mobile computing devices, coupled with cheap and elastic cloud-based data storage and computing, along with advances in the fields of artificial intelligence and machine learning, has launched a data revolution in the science of learning. How do we gain knowledge about how people learn from data on hundreds of thousands of students and their teachers collected across extended time frames? This chapter considers this question, starting with a word about the data itself. How to make sense of all these data? The chapter briefly considers some of the key research areas and computational techniques used. The key research areas and computational techniques include: formative assessment; supervised, semisupervised, and unsupervised learning; relationship discovery; sequence modeling; content analysis and natural language processing; and multisensory, multimodal modeling. The chapter finally turns to four case studies from authors' lab to illustrate a broad range of research goals, data, and methods. (PsycInfo Database Record (c) 2024 APA, all rights reserved)

Collaborative problem solving (CPS) is an essential skill in the 21st century. There is a need for an appropriate framework and operationalization of CPS to guide its assessment and support and across multiple domains. Accordingly, we synthesized prior research on CPS to construct a generalized CPS competency model (i.e., skills and abilities) consisting of the following core facets: constructing shared knowledge, negotiation/coordination, and maintaining team function. Each facet has two sub-facets, which in turn, have multiple verbal and nonverbal indicators. We validated our model in two empirical studies involving triadic CPS, but in very different contexts - middle-school students playing an educational game in a 3-h, face-to-face session vs. college students engaging in a visual programming task for 20 min via videoconferencing. We used principal component analysis to investigate whether the empirical data aligned with our theorized model. Correlational analyses provided evidence on the orthogonality of the facets and their independence to individual differences in prior knowledge, intelligence and personality and regression analyses indicated that the facets predicted both subjective and objective outcome measures controlling for several covariates. Thus we provide initial evidence for the convergent, discriminant, and predictive validity of our model by using two different CPS contexts and student populations. This shows promise towards generalizing across various human-human CPS interactive environments.

Collaborative problem solving (CPS) in virtual environments is an increasingly important context of 21st century learning. However, our understanding of this complex and dynamic phenomenon is still limited. Here, we examine unimodal primitives (activity on the screen, speech, and body movements), and their multimodal combinations during remote CPS. We analyze two datasets where 116 triads collaboratively engaged in a challenging visual programming task using video conferencing software. We investigate how UI-interactions, behavioral primitives, and multimodal patterns were associated with teams' subjective and objective performance outcomes. We found that idling with limited speech (i.e., silence or backchannel feedback only) and without movement was negatively correlated with task performance and with participants' subjective perceptions of the collaboration. However, being silent and focused during solution execution was positively correlated with task performance. Results illustrate that in some cases, multimodal patterns improved the predictions and improved explanatory power over the unimodal primitives. We discuss how the findings can inform the design of real-time interventions for remote CPS.

Modeling team phenomena from multiparty interactions inherently requires combining signals from multiple teammates, often by weighting strategies. Here, we explored the hypothesis that strategic weighting signals from individual teammates would outperform an equal weighting baseline. Accordingly, we explored role-, trait-, and behavior-based weighting of behavioral signals across team members. We analyzed data from 101 triads engaged in computer-mediated collaborative problem solving (CPS) in an educational physics game. We investigated the accuracy of machine-learned models trained on facial expressions, acoustic-prosodics, eye gaze, and task context information, computed one-minute prior to the end of a game level, at predicting success at solving that level. AUROCs for unimodal models that equally weighted features from the three teammates ranged from .54 to .67, whereas a combination of gaze, face, and task context features, achieved an AUROC of .73. The various multiparty weighting strategies did not outperform an equal-weighting baseline. However, our best nonverbal model (AUROC = .73) outperformed a language-based model (AUROC = .67), and there were some advantages to combining the two (AUROC = .75). Finally, models aimed at prospectively predicting performance on a minute-by-minute basis from the start of the level achieved a lower, but still above-chance, AUROC of .60. We discuss implications for multiparty modeling of team performance and other team constructs. © 2020 ACM.