A Student-Teacher Multimodal Interaction Analysis System For Classroom Observation

Classroom observation is an effective way for teachers to improve professional development, and the analysis of student-teacher interactions is critical and significant to classroom observation. However, the traditional methods of the classroom observation are mainly based on manual coding by domain experts. Although several studies have been conducted to automate the coding and analyzing process, they are either based on audio information or video information collected from the classroom, which fails to jointly utilize multimodal information like domain experts. We thus propose a student-teacher multimodal interaction analysis system that conducts the analysis using both video and audio information and accordingly generates the informative reports based on the analysis results. A preliminary evaluation of the system validates the effectiveness of the built system and the analysis results could be further used for the evidence-based teaching behavior evaluation. The current limitations and possible optimization on the built system are discussed as well. We are planning to keep improving the system and deploy it to 1000 schools located at the rural areas in three years. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.