Students in computerized learning environments often direct their own learning processes, which requires metacognitive awareness of what should be learned next. We investigated a novel method of measuring verbalized metacognition by applying natural language processing (NLP) to transcripts of interviews conducted in a classroom with 99 middle school students who were using a computerized learning environment. We iteratively adapted the NLP method for the linguistic characteristics of these interviews, then applied it to study three research questions regarding the relationships between verbalized metacognition and measures of 1) learning, 2) confusion, and 3) metacognitive problem-solving strategies. Verbalized metacognition was not directly related to learning, but was related to confusion and metacognitive problem-solving strategies. Results also suggested that interviews themselves may improve learning by encouraging metacognition. We discuss implications for designing computerized environments that support self-regulated learning through metacognition. © 2021 ACM.

Worldwide, national initiatives have led to many school districts implementing computing curricula at the primary level. At that age, students are learning the foundational skills of reading and math. It is important to understand how computing can influence the development of these skills. While some argue that learning computing sharpens problem-solving skills that are applicable to other subjects, evidence supporting this belief is thin. In a quasi-experimental study of fourth-grade (ages 9-10) students, we compared state reading and math test scores of students receiving computing instruction with students who did not. Our findings demonstrated that a more open-ended, less scaffolded form of computing instruction was linked to performance gains in math, but not in reading (F(2, 232)=11.08, p< .01, p2=.0625). When looking at students who face academic challenges that can impact reading and math, the same trend applied to students with economic disadvantages and students with limited English proficiency, but not for students with disabilities. These results suggest that moderately scaffolded computing instruction supports the development of skills applicable to math, a step towards better understanding the relationship between learning opportunities in computing and outcomes in other subjects. © 2021 ACM.

The demand for computational thinking (CT) problem solving abilities surge as every aspect of life becomes more dependent on complex digital technologies. Just as in math and language, a strong CT foundation needs to be established in early education in order for students to develop an instinctive CT perspective of the world. The urgent demand for CT instruction in elementary school quickly draws attention to the shortage of elementary school-level teachers qualified and interested in CT. Additionally, with a commitment to equity in the United States education system and knowledge of the high percentage of English language learning (ELL) students in schools, the obligation to create curricula that will provide access to CT knowledge, skills, and practices for elementary-level ELL students is loudly apparent. In response to these two needs, our team has adapted existing Scratch-based CT curriculum to support classroom teachers with minimal CT experience and to be more accessible to English language learners. The purpose of this paper is to share the framework that guided the curriculum adaptations, to describe the specific changes that were made, and to discuss discoveries made during the process. This journey may be helpful to anyone who is tasked with modifying a curriculum to meet the needs of novice content teachers and ELL students. © 2021 Owner/Author.

As more educators integrate their curricula with online learning, it is easier to crowdsource content from them. Crowdsourced tutoring has been proven to reliably increase students' next problem correctness. In this work, we confirmed the findings of a previous study in this area, with stronger confidence margins than previously, and revealed that only a portion of crowdsourced content creators had a reliable benefit to students. Furthermore, this work provides a method to rank content creators relative to each other, which was used to determine which content creators were most effective overall, and which content creators were most effective for specific groups of students. When exploring data from TeacherASSIST, a feature within the ASSISTments learning platform that crowdsources tutoring from teachers, we found that while overall this program provides a benefit to students, some teachers created more effective content than others. Despite this finding, we did not find evidence that the effectiveness of content reliably varied by student knowledge-level, suggesting that the content is unlikely suitable for personalizing instruction based on student knowledge alone. These findings are promising for the future of crowdsourced tutoring as they help provide a foundation for assessing the quality of crowdsourced content and investigating content for opportunities to personalize students' education. [This paper was published in: " L@S '21, June 22-25, 2021, Virtual Event, Germany," ACM, 2021.]

Whereas social visual attention has been examined in computer-mediated (e.g., shared screen) or video-mediated (e.g., FaceTime) interaction, it has yet to be studied in mixed-media interfaces that combine video of the conversant along with other UI elements. We analyzed eye gaze of 37 dyads (74 participants) who were tasked with negotiating the price of a new car (as a buyer and seller) using mixed-media video conferencing under competitive or cooperative negotiation instructions (experimental manipulation). We used multidimensional recurrence quantification analysis to extract spatio-temporal patterns corresponding to mutual gaze (individuals look at each other), joint attention (individuals focus on the same elements of the interface), and gaze aversion (an individual looks at their partner, who is looking elsewhere). Our results indicated that joint attention predicted the sum of points attained by the buyer and seller (i.e., the joint score). In contrast, gaze aversion was associated with faster time to complete the negotiation, but with a lower joint score. Unexpectedly, mutual gaze was highly infrequent and unrelated to the negotiation outcomes and none of the gaze patterns predicted subjective perceptions of the negotiation. There were also no effects of gender composition or negotiation condition on the gaze patterns or negotiation outcomes. Our results suggest that social visual attention may operate differently in mixed-media collaborative interfaces than in face-to-face interaction. As mixed-media collaborative interfaces gain prominence, our work can be leveraged to inform the design of gaze-sensitive user interfaces that support remote negotiations among other tasks.

College-educated women in the workforce are discovering a latent interest in and aptitude for computing motivated by the prevalence of computing as an integral part of jobs in many fields as well as continued headlines about the number of unfilled, highly paid computing jobs. One of these women's choices for retraining are the so-called coding boot camps that teach programming skills through intensive multi-week courses. This article reports on a qualitative research study focused on the Silicon Valley area of California. We used social cognitive career theory (SCCT) to investigate the larger context surrounding women entering computing professions through boot camp learning sites, including: the environment of a booming technology workforce, boot camps as learning settings, the characteristics of women who attend boot camps, how retraining at a coding boot camp influence women's computing self-efficacy and outcome expectations, and the performance attainments of women at boot camps. Interview data was collected from 14 women who had attended boot camps-first before graduating from the boot camp and again after six months in the workforce. To contextualize the computing ecosystem, we conducted single interviews with 6 boot camp organizers/curriculum developers, 7 industry hiring managers, and 14 university computer science faculty. To provide a contrast with women at boot camps, we interviewed 5 women who majored in computer science at the university and with 17 men who had attended boot camps. Structural data coding and analysis was done focused on the SCCT mechanisms of environment, person inputs, learning experiences, self-efficacy, outcome expectations, and performance attainments. Findings here demonstrate that training at a boot camp can be the catalyst for college-educated women to attain computing jobs and careers, although these entry-level jobs may be a compromise to the goal of a software development job and are unlikely to lead to a job at large, well-known, established technology companies.

With the global movement to incorporate computer science instruction into elementary education, learners are being introduced to computer science and computational thinking (CS/CT) ideas at increasingly younger ages. At these early ages, young learners are developing cognitive abilities foundational to their education. While other discipline-based education fields, such as math, science, and reading, have long studied the role of cognitive abilities, such as short-term working memory and long-term retrieval, in their respective fields, similar research in computer science education is relatively sparse. In this exploratory study, we examined the relationship between cognitive abilities and CS/CT performance of fourth-grade students (ages 9-10) who underwent either an introductory CT curriculum based on Use->Modify->Create or the same curriculum with additional scaffolding from the TIPP&SEE metacognitive learning strategy. Our analysis revealed performance on CT assessments to be weakly correlated with working memory and long-term retrieval, with correlations increasing as the CT concepts grew more complex. This suggests that scaffolding beyond TIPP&SEE may be needed with more complex CT concepts. We also found that when using TIPP&SEE, students scoring below average on cognitive ability tests performed as well as students in the control condition with average cognitive ability scores. These results indicate TIPP&SEE's potential in creating more equitable computing instruction. We hope that results from this initial exploration can help encourage further study into the role of cognitive abilities in CS/CT education for young learners.

The effectiveness of user interfaces are limited by the tendency for the human mind to wander. Intelligent user interfaces can combat this by detecting when mind wandering occurs and attempting to regain user attention through a variety of intervention strategies. However, collecting data to build mind wandering detection models can be expensive, especially considering the variety of media available and potential differences in mind wandering across them. We explored the possibility of using eye gaze to build cross-domain models of mind wandering where models trained on data from users in one domain are used for different users in another domain. We built supervised classification models using a dataset of 132 users whose mind wandering reports were collected in response to thought-probes while they completed tasks from seven different domains for six minutes each (five domains are investigated here: Illustrated Text, Narrative Film, Video Lecture, Naturalistic Scene, and Reading Text). We used global eye gaze features to build within- and cross- domain models using 5-fold user-independent cross validation. The best performing within-domain models yielded AUROCs ranging from .57 to .72, which were comparable for the cross-domain models (AUROCs of .56 to .68). Models built from coarse-grained locality features capturing the spatial distribution of gaze resulted in slightly better transfer on average (transfer ratios of .61 vs .54 for global models) due to improved performance in certain domains. Instance-based and feature-level domain adaptation did not result in any improvements in transfer. We found that seven gaze features likely contributed to transfer as they were among the top ten features for at least four domains. Our results indicate that gaze features are suitable for domain adaptation from similar domains, but more research is needed to improve domain adaptation between more dissimilar domains.

Despite the abundance of data generated from students' activities in virtual learning environments, the use of supervised machine learning in learning analytics is limited by the availability of labeled data, which can be difficult to collect for complex educational constructs. In a previous study, a subfield of machine learning called Active Learning (AL) was explored to improve the data labeling efficiency. AL trains a model and uses it, in parallel, to choose the next data sample to get labeled from a human expert. Due to the complexity of educational constructs and data, AL has suffered from the cold-start problem where the model does not have access to sufficient data yet to choose the best next sample to learn from. In this paper, we explore the use of past data to warm start the AL training process. We also critically examine the implications of differing contexts (urbanicity) in which the past data was collected. To this end, we use authentic affect labels collected through human observations in middle school mathematics classrooms to simulate the development of AL-based detectors of engaged concentration. We experiment with two AL methods (uncertainty sampling, L-MMSE) and random sampling for data selection. Our results suggest that using past data to warm start AL training could be effective for some methods based on the target population's urbanicity. We provide recommendations on the data selection method and the quantity of past data to use when warm starting AL training in the urban and suburban schools.

Reflection plays a critical role in learning by encouraging students to contemplate their knowledge and previous learning experiences to inform their future actions and higher-order thinking, such as reasoning and problem solving. Reflection is particularly important in inquiry-driven learning scenarios where students have the freedom to set goals and regulate their own learning. However, despite the importance of reflection in learning, there are significant theoretical, methodological, and analytical challenges posed by measuring, modeling, and supporting reflection. This paper presents results from a classroom study to investigate middle-school students' reflection during inquiry-driven learning with Crystal Island, a game-based learning environment for middle-school microbiology. To collect evidence of reflection during game-based learning, we used embedded reflection prompts to elicit written reflections during students' interactions with Crystal Island. Results from analysis of data from 105 students highlight relationships between features of students' reflections and learning outcomes related to both science content knowledge and problem solving. We consider implications for building adaptive support in game-based learning environments to foster deep reflection and enhance learning, and we identify key features in students' problem-solving actions and reflections that are predictive of reflection depth. These findings present a foundation for providing adaptive support for reflection during game-based learning.