Publications
The purpose of this report is to present our process for establishing evidence of validity and reliability of an observation tool [PEMTP: Potentially Engaging Mathematics Teaching Practices] used to investigate mathematics teaching practices that have the potential to engage high school students. We first present our practices for establishing content-oriented evidence of validity for our observation tool. Next, we describe how we developed a framework to identify and specify potentially engaging mathematics teaching practices, including our efforts to pilot the tool to design descriptive levels for rubrics. We then report on a reliability investigation we conducted to examine the nature of coding disagreements. Our results indicate the value of reporting on validity processes for tools such as the PEMTP to illustrate the ways that the tool has been designed and revised, which can inform the use cases and potential future applications of the tool. These results also show how such validation efforts can lead to meaningful revisions – in the case of the PEMTP, the addition of two rubrics aligned with teaching mathematics for equity. Finally, the results of our reliability investigation indicate the value of using multiple raters and collaborative resolution meetings when investigating complex phenomena like classroom instruction.
We introduce the SRI speech-based collaborative learning corpus, a novel collection designed for the investigation and measurement of how students collaborate together in small groups. This is a multi-speaker corpus containing high-quality audio recordings of middle school students working in groups of three to solve mathematical problems. Each student was recorded via a head-mounted noise-cancelling microphone. Each group was also recorded via a stereo microphone placed nearby. A total of 80 sessions were collected with the participation of 134 students. The average duration of a session was 20 minutes. All students spoke English; for some students, English was a second language. Sessions have been annotated with time stamps to indicate which mathematical problem the students were solving and which student was speaking. Sessions have also been hand annotated with common indicators of collaboration for each speaker (e.g., inviting others to contribute, planning) and the overall collaboration quality for each problem. The corpus will be useful to education researchers interested in collaborative learning and to speech researchers interested in children's speech, speech analytics, and speech diarization. The corpus, both audio and annotation, will be made available to researchers.
Collaborative learning is a key skill for student success, but simultaneous monitoring of multiple small groups is untenable for teachers. This study investigates whether automatic audiobased monitoring of interactions can predict collaboration quality. Data consist of hand-labeled 30-second segments from audio recordings of students as they collaborated on solving math problems. Two types of features were explored: speech activity features, which were computed at the group level; and prosodic features (pitch, energy, durational, and voice quality patterns), which were computed at the speaker level. For both feature types, normalized and unnormalized versions were investigated; the latter facilitate real-time processing applications. Results using boosting classifiers, evaluated by F-measure and accuracy, reveal that (1) both speech activity and prosody features predict quality far beyond chance using majority-class approach; (2) speech activity features are the better predictors overall, but class performance using prosody shows potential synergies; and (3) it may not be necessary to session-normalize features by speaker. These novel results have impact for educational settings, where the approach could support teachers in the monitoring of group dynamics, diagnosis of issues, and development of pedagogical intervention plans. © 2016, International Speech Communications Association. All rights reserved.


