Who And How: Using Sentence-Level NLP To Evaluate Idea Completeness

Real-time feedback is very important, yet challenging to provide for free-text learner contributions in Technology-Enhanced Learning. We study whether a generic NLP pipeline can identify completeness features of learner ideas during security training. We apply PoS Tagging and Dependency Parsing on contextualised short texts, collected within a dedicated learning environment and we compare the results to an expert-annotated ground truth. We scan these contributions for the absence of responsible stakeholder (who) or featured action (how). A total of 1174 contributions in two security domains were analysed. We report precision on who (PPV= 0.929 ) and on how (PPV= 0.691 ). We consider the first result to be sufficient to provide real-time formative feedback for the case of absent who. Our results suggest that for the purposes of providing feedback in free input problem-solving exercises, generic transformer pipelines without fine-tuning can achieve good performance on stakeholder identification. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.