The low cost of mobile computing devices, coupled with cheap and elastic cloud-based data storage and computing, along with advances in the fields of artificial intelligence and machine learning, has launched a data revolution in the science of learning. How do we gain knowledge about how people learn from data on hundreds of thousands of students and their teachers collected across extended time frames? This chapter considers this question, starting with a word about the data itself. How to make sense of all these data? The chapter briefly considers some of the key research areas and computational techniques used. The key research areas and computational techniques include: formative assessment; supervised, semisupervised, and unsupervised learning; relationship discovery; sequence modeling; content analysis and natural language processing; and multisensory, multimodal modeling. The chapter finally turns to four case studies from authors' lab to illustrate a broad range of research goals, data, and methods. (PsycInfo Database Record (c) 2024 APA, all rights reserved)

Collaborative problem solving (CPS) is an essential skill in the 21st century. There is a need for an appropriate framework and operationalization of CPS to guide its assessment and support and across multiple domains. Accordingly, we synthesized prior research on CPS to construct a generalized CPS competency model (i.e., skills and abilities) consisting of the following core facets: constructing shared knowledge, negotiation/coordination, and maintaining team function. Each facet has two sub-facets, which in turn, have multiple verbal and nonverbal indicators. We validated our model in two empirical studies involving triadic CPS, but in very different contexts - middle-school students playing an educational game in a 3-h, face-to-face session vs. college students engaging in a visual programming task for 20 min via videoconferencing. We used principal component analysis to investigate whether the empirical data aligned with our theorized model. Correlational analyses provided evidence on the orthogonality of the facets and their independence to individual differences in prior knowledge, intelligence and personality and regression analyses indicated that the facets predicted both subjective and objective outcome measures controlling for several covariates. Thus we provide initial evidence for the convergent, discriminant, and predictive validity of our model by using two different CPS contexts and student populations. This shows promise towards generalizing across various human-human CPS interactive environments.

Multiparty collaborative problem solving-an increasingly important context in the 21st century workforce-suffers from a degradation of social and behavioral signals when attempted remotely, resulting in suboptimal outcomes. We investigate teams' multidimensional patterns of visual attention during a collaborative problem-solving task with an eye for leveraging insights to improve collaborative interfaces. Fifty-seven novices (forming 19 triads) engaged in a challenging programming task (Minecraft Hour of Code) using videoconferencing software with screen sharing. To discover patterns of individual-level gaze-UI coupling (coordination of a teammate's attention with respect to changes in the user interface) and team-level gaze-UI regularity (dynamics of teams' collective attention in context with changes in the user interface), we applied cross- and multidimensional recurrence quantification analyses, respectively. Individuals' eye gaze was significantly coupled with the ongoing screen activity whereas teams displayed significant patterns of gaze regularity, suggesting repetitive patterns in teams' attention. These measures predicted expert-coded collaborative processes of constructing shared knowledge and negotiation and coordination (but not maintaining team function) and correlated with task score (r = .425). They also predicted individually assessed subjective perceptions of team performance and the collaboration process, but not individual's learning or team's task scores. We discuss implications of our findings for the design of intelligent collaborative interfaces.

We adopt a multimodal approach to investigating team interactions in the context of remote collaborative problem solving (CPS). Our goal is to understand multimodal patterns that emerge and their relation with collaborative outcomes. We measured speech rate, body movement, and galvanic skin response from 101 triads (303 participants) who used video conferencing software to collaboratively solve challenging levels in an educational physics game. We use multi-dimensional recurrence quantification analysis (MdRQA) to quantify patterns of team-level regularity, or repeated patterns of activity in these three modalities. We found that teams exhibit significant regularity above chance baselines. Regularity was unaffected by task factors. but had a quadratic relationship with session time in that it initially increased but then decreased as the session progressed. Importantly, teams that produce more varied behavioral patterns (irregularity) reported higher emotional valence and performed better on a subset of the problem solving tasks. Regularity did not predict arousal or subjective perceptions of the collaboration. We discuss implications of our findings for the design of systems that aim to improve collaborative outcomes by monitoring the ongoing collaboration and intervening accordingly.

Collaborative problem solving (CPS) is a crucial 21st century skill; however, current technologies fall short of effectively supporting CPS processes, especially for remote, computer-enabled interactions. In order to develop next-generation computer-supported collaborative systems that enhance CPS processes and outcomes by monitoring and responding to the unfolding collaboration, we investigate automated detection of three critical CPS process – construction of shared knowledge, negotiation/coordination, and maintaining team function – derived from a validated CPS framework. Our data consists of 32 triads who were tasked with collaboratively solving a challenging visual computer programming task for 20 minutes using commercial videoconferencing software. We used automatic speech recognition to generate transcripts of 11,163 utterances, which trained humans coded for evidence of the above three CPS processes using a set of behavioral indicators. We aimed to automate the trained human-raters’ codes in a team-independent fashion (current study) in order to provide automatic real-time or offline feedback (future work). We used Random Forest classifiers trained on the words themselves (bag of n-grams) or with word categories (e.g., emotions, thinking styles, social constructs) from the Linguistic Inquiry Word Count (LIWC) tool. Despite imperfect automatic speech recognition, the n-gram models achieved AUROC (area under the receiver operating characteristic curve) scores of .85, .77, and .77 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively; these reflect 70%, 54%, and 54% improvements over chance. The LIWC-category models achieved similar scores of .82, .74, and .73 (64%, 48%, and 46% improvement over chance). Further, the LIWC model-derived scores predicted CPS outcomes more similar to human codes, demonstrating predictive validity. We discuss embedding our models in collaborative interfaces for assessment and dynamic intervention aimed at improving CPS outcomes. Copyright © ACM

We hypothesize that effective collaboration is facilitated when individuals and environmental components form a synergy where they work together and regulate one another to produce stable patterns of behavior, or regularity, as well as adaptively reorganize to form new behaviors, or irregularity. We tested this hypothesis in a study with 32 triads who collaboratively solved a challenging visual computer programming task for 20 min following an introductory warm-up phase. Multidimensional recurrence quantification analysis was used to examine fine-grained (i.e., every 10 s) collective patterns of regularity across team members' speech rate, body movement, and team interaction with the shared user interface. We found that teams exhibited significant patterns of regularity as compared to shuffled baselines, but there were no systematic trends in regularity across time. We also found that periods of regularity were associated with a reduction in overall behavior. Notably, the production of irregular behavior predicted expert-coded metrics of collaborative activity, such as teams' ability to construct shared knowledge and effectively negotiate and coordinate execution of solutions, net of overall behavioral production and behavioral self-similarity. Our findings support the theory that groups can interact to form interpersonal synergies and indicate that information about system-level dynamics is a viable way to understand and predict effective collaborative processes.

We investigated how affective states influence expository text comprehension and whether text valence moderates the effects (i.e., mood congruency). In Experiment 1 participants were randomly assigned to a happy or sad affective state (elicited via films) before reading a positive or negative version of a scientific text on animal adaptations. Participants (n = 79) in the sad (film) group had higher scores on deep-reasoning (d = .312) but not surface-level questions on a subsequent multiple-choice comprehension assessment; there was also no evidence for mood congruence. Using a neutral version of the same text, in Experiment 2 participants (n = 52) in a fearful condition performed better on surface-level comprehension questions (d = .594) compared with a sad condition, but the groups were on par for deep-reasoning questions. Experiment 3 (n = 595) did not replicate the findings from Experiment 2 (no comprehension differences between the sad and fear groups) and there were no differences between the fear and happy groups. However, the sad group outperformed the happy group on deep-reasoning questions (d = .210), thereby replicating Experiment 1. The overall findings were confirmed after pooling the data from the three experiments to increase power.

Linguistic alignment (LA) is the tendency during a conversation to reuse each other's linguistic expressions, including lexical, conceptual, or syntactic structures. LA is often argued to be a crucial driver in reciprocal understanding and interpersonal rapport, though its precise dynamics and effects are still controversial. One barrier to more systematic investigation of these effects lies in the diversity in the methods employed to analyze LA, which makes it difficult to integrate and compare results of individual studies. To overcome this issue, we have developed ALIGN (Analyzing Linguistic Interactions with Generalizable techNiques), an open-source Python package to measure LA in conversation (https://pypi.python.org/pypi/align) along with in-depth open-source tutorials hosted on ALIGN's GitHub repository (https://github.com/nickduran/align-linguistic-alignment). Here, we first describe the challenges in the study of LA and outline how ALIGN can address them. We then demonstrate how our analytical protocol can be applied to theory-driven questions using a complex corpus of dialogue (the Devil's Advocate corpus; Duran & Fusaroli, 2017). We close by identifying further challenges and point to future developments of the field.

We model coordination and coregulation patterns in 33 triads engaged in collaboratively solving a challenging computer programming task for approximately 20 minutes. Our goal is to prospectively model speech rate (words/sec) - an important signal of turn taking and active participation - of one teammate (A or B or C) from time lagged nonverbal signals (speech rate and acoustic-prosodic features) of the other two (i.e., A + B -> C; A + C -> B; B + C -> A) and task-related context features. We trained feed -forward neural networks (FFNNs) and long sshort-term memory recurrent neural networks (LSTMs) using group level nested cross-validation. LSTMs outperformed FFNNs and a chance baseline and could predict speech rate up to 6s into the future. A multimodal combination of speech rate, acoustic prosodic, and task context features outperformed unimodal and bimodal signals. The extent to which the models could predict an individual's speech rate was positively related to that individual's scores on a subsequent posttest, suggesting a link between coordination/coregulation and collaborative learning outcomes. We discuss applications of the models for real-time systems that monitor the collaborative process and intervene to promote positive collaborative outcomes.

Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real world applications (e.g., as a tool in therapy).