Involving Teachers In The Data-Driven Improvement Of Intelligent Tutors: A Prototyping Study

Several studies show that log data analysis can lead to effective redesign of intelligent tutoring systems (ITSs). However, teachers are seldom included in the data-driven redesign of ITS, despite their pedagogical content knowledge. Examining teachers’ possible contributions is valuable. To investigate what contributions teachers might make and whether (and how) data would be useful, we first built an interactive prototype tool for visualizing student log data, SolutionVis, based on needs identified in interviews with tutor authors. SolutionVis presents students’ problem-solving processes with an intelligent tutor, including meta-cognitive aspects (e.g., hint requests). We then conducted a within-subjects user study with eight teachers to compare teachers’ redesign suggestions obtained in three conditions: a baseline “no data” condition (where teachers examined just the tutor itself) and two “with data” conditions in which teachers worked with SolutionVis and with a list representation of student solutions, respectively. The results showed that teachers generated useful redesign ideas in all three conditions, that they viewed the availability of data (in both formats) as helpful and enabled them to generate a wider range of redesign suggestions, specifically with respect to hint design and feedback on gaming-the-system behaviors and struggle. The current work suggests potential benefits and ways of involving teachers in the data-driven improvement of ITSs. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.