Instructional activity recognition is an analytical tool for the observation of classroom education. One of the primary challenges in this domain is dealing with the intri- cate and heterogeneous interactions between teachers, students, and instructional objects. To address these complex dynamics, we present an innovative activity recognition pipeline designed explicitly for instructional videos, leveraging a multi-semantic attention mechanism. Our novel pipeline uses a transformer network that incorporates several types of instructional seman- tic attention, including teacher-to-students, students-to-students, teacher-to-object, and students-to-object relationships. This com- prehensive approach allows us to classify various interactive activity labels effectively. The effectiveness of our proposed algo- rithm is demonstrated through its evaluation on our annotated instructional activity dataset.

Many major reforms underway in American public school districts' elementary science programs are the result of the release of the Next Generation Science Standards (NGSS) in 2013. More than a decade later, many districts are continuing to put the vision of those standards into practice. A problem for districts pursuing these changes is the complexity of their environments; that is, the array of governmental, commercial, research, community, and other organizations and interests seeking to influence improvement in elementary science. The purpose of this three-paper dissertation is to support efforts to improve elementary science by conceptualizing the environments of elementary science reform; exploring the intersection between those environments and specific strategies for district-level improvement; and analyzing strategies for reconciling new approaches to improving elementary science with legacy approaches The first paper examines the ways district leaders engage those environments as they work to improve elementary science instruction in ways responsive the NGSS. Drawing on historic accounts of districts engaged in elementary science reform, this paper makes the claim that the organizational field supporting science education reform consists of specific categories of organizations that have maintained currency for decades and that subsets of these organizations appear to be more immediately salient to districts. The primary contribution of this paper is an organizational taxonomy that represents field-level organizations seeking to influence science education in elementary schools and that represents their relative salience for districts. The second paper explores how districts engage in system building to support NGSS aligned instruction in complex environments, the component parts of which are uncoordinated and variably aligned to these new standards. This paper draws on data collected as part of a comparative case study of system-building efforts in 13 districts including interviews, observations, and artifacts. Findings from this paper show that districts coordinate across a wide array of organizations in their environment, regardless of the specifics of their system building approaches. In doing so, organizations that are more proximal and/or more directly related to districts' day-to-day functionality are more salient and may filter the more distal parts of their environment. This paper makes the claim that districts engage with a wide array of organizations, regardless of the approach they take to improving elementary science. The third paper examines how districts are working to put the vision of the NGSS into practice in elementary schools where existing structures, norms, and routines may complicate efforts to respond to these challenges. As in the previous paper, this study draws on interviews, observations, and artifacts collected as part of a comparative case study of system-building efforts in 13 districts. Findings from this paper illustrate that there are at least three distinct approaches—integrating, retooling, and skirting—that districts use to manage relationships between new efforts to improve science instruction and pre-existing structures, norms, and routines. Through three cases, this paper illustrates how these approaches help districts leverage internal strengths to mitigate enduring challenges of instructional improvement. This dissertation provides theoretical and empirical perspectives on the complex environments in which elementary science reforms are enacted and the ways districts manage relationships with those environments. In addition to providing frameworks for future scholarship, these papers highlight the need for science reformers, science educators, and districts leaders to be mindful of the contexts in which that work is enacted.

Classroom videos are a common source of data for educational researchers studying classroom interactions as well as a resource for teacher education and professional development. Over the last several decades emerging technologies have been applied to classroom videos to record, transcribe, and analyze classroom interactions. With the rise of machine learning, we report on the development and validation of neural networks to classify instructional activities using video signals, without analyzing speech or audio features, from a large corpus of nearly 250 h of classroom videos from elementary mathematics and English language arts instruction. Results indicated that the neural networks performed fairly-well in detecting instructional activities, at diverse levels of complexity, as compared to human raters. For instance, one neural network achieved over 80% accuracy in detecting four common activity types: whole class activity, small group activity, individual activity, and transition. An issue that was not addressed in this study was whether the fine-grained and agnostic instructional activities detected by the neural networks could scale up to supply information about features of instructional quality. Future applications of these neural networks may enable more efficient cataloguing and analysis of classroom videos at scale and the generation of fine-grained data about the classroom environment to inform potential implications for teaching and learning. © 2024 The Authors

Instructional activity recognition is an analytical tool for the observation of classroom education. One of the primary challenges in this domain is dealing with the intri- cate and heterogeneous interactions between teachers, students, and instructional objects. To address these complex dynamics, we present an innovative activity recognition pipeline designed explicitly for instructional videos, leveraging a multi-semantic attention mechanism. Our novel pipeline uses a transformer network that incorporates several types of instructional seman- tic attention, including teacher-to-students, students-to-students, teacher-to-object, and students-to-object relationships. This com- prehensive approach allows us to classify various interactive activity labels effectively. The effectiveness of our proposed algo- rithm is demonstrated through its evaluation on our annotated instructional activity dataset.

This paper presents a novel spatiotemporal transformer network that introduces several original components to detect actions in untrimmed videos. First, the multi-feature selective semantic attention model calculates the correlations between spatial and motion features to model spatiotemporal interactions between different action semantics properly. Second, the motion-aware network encodes the locations of action semantics in video frames utilizing the motion-aware 2D positional encoding algorithm. Such a motion-aware mechanism memorizes the dynamic spatiotemporal variations in action frames that current methods cannot exploit. Third, the sequence-based temporal attention model captures the heterogeneous temporal dependencies in action frames. In contrast to standard temporal attention used in natural language processing, primarily aimed at finding similarities between linguistic words, the proposed sequence-based temporal attention is designed to determine both the differences and similarities between video frames that jointly define the meaning of actions. The proposed approach outperforms the state-of-the-art solutions on four spatiotemporal action datasets: AVA 2.2, AVA 2.1, UCF101-24, and EPIC-Kitchens.

Despite a tremendous increase in the use of video for conducting research in classrooms as well as preparing and evaluating teachers, there remain notable challenges to using classroom videos at scale, including time and financial costs. Recent advances in artificial intelligence could make the process of analyzing, scoring, and cataloguing videos more efficient. These advances include natural language processing, automated speech recognition, and deep neural networks. To train artificial intelligence to accurately classify activities in classroom videos, humans must first annotate a set of videos in a consistent way. This paper describes our investigation of the degree of inter-annotator reliability regarding identification of and duration of activities among annotators with and without experience analyzing classroom videos. Validity of human annotations is crucial for research involving temporal analysis within classroom video research. The study reported here represents an important step towards applying methods developed in other fields to validate temporal analytics within learning analytics research for classifying time- and event-based activities in classroom videos.

This paper proposes a novel age estimation algorithm, the Temporally-Aware Adaptive Graph Convolutional Network (TAA-GCN). Using a new representation based on graphs, the TAA-GCN utilizes skeletal, posture, clothing, and facial information to enrich the feature set associated with various ages. Such a novel graph representation has several advantages: First, reduced sensitivity to facial expression and other appearance variances; Second, robustness to partial occlusion and non-frontal-planar viewpoint, which is commonplace in real-world applications such as video surveillance. The TAA-GCN employs two novel components, (1) the Temporal Memory Module (TMM) to compute temporal dependencies in age; (2) Adaptive Graph Convolutional Layer (AGCL) to refine the graphs and accommodate the variance in appearance. The TAA-GCN outperforms the state-of-the-art methods on four public benchmarks, UTKFace, MORPHII, CACD, and FG-NET. Moreover, the TAA-GCN showed reliability in different camera viewpoints and reduced quality images. © 2022 Elsevier Ltd

This paper proposes a novel multi-modal transformer network for detecting actions in untrimmed videos. To enrich the action features, our transformer network utilizes a new multi-modal attention mechanism that computes the correlations between different spatial and motion modalities combinations. Exploring such correlations for actions has not been attempted previously. To use the motion and spatial modality more effectively, we suggest an algorithm that corrects the motion distortion caused by camera movement. Such motion distortion, common in untrimmed videos, severely reduces the expressive power of motion features such as optical flow fields. Our proposed algorithm outperforms the state-of-the-art methods on two public benchmarks, THUMOS14 and ActivityNet. We also conducted comparative experiments on our new instructional activity dataset, including a large set of challenging classroom videos captured from elementary schools. © 2023

Analyzing instructional videos via computer vision and machine learning holds promise for several tasks, such as assessing teacher performance and classroom climate, evaluating student engagement, and identifying racial bias in instruction. The traditional way of evaluating instructional videos depends on manual observation with human raters, which is time-consuming and requires a trained labor force. Therefore, this paper tests several deep network architectures in the automation of instruc- tional video analysis, where the networks are tailored to recognize classroom activity. Our experimental setup includes a set of 250 hours of primary and middle school videos that are annotated by expert human raters. We present several strategies to handle varying length of instructional activities, a major challenge in the detection of instructional activity. Based on the proposed strategies, we enhance and compare different deep networks for detecting instructional activity.

Video-based classroom observation tools can provide constructive feedback to K-6 teachers and assist their training and skill development. However, accurate classroom observation ratings necessitate observer training, requiring the investment of time, money and skilled labor. We propose a new AI-assisted pipeline to automate the classroom observation rating process utilizing a deep learning framework. Specifically, we combine three streams: action recognition, object detection, and age estimation networks to detect classroom instructional activities. We conducted the experiments on a novel labeled K-6 (elementary) classroom observation video dataset to detect human activities in the classroom environment. We labeled the data with the help of experienced annotators trained in classroom observation instruments. We also conducted multiple inter-rater reliability studies to ensure reliable labels. Our preliminary experimental results show promise for detecting multiple classroom instructional activity labels. © 2021 IEEE.