Serendipitous Gains of Explaining a Classifier - Artificial versus Human Performance and Annotator Support in an Urgent Instructor-Intervention Model for MOOCs

Alrajhi, Laila; Pereira, Filipe Dwan; Cristea, Alexandra I.; Alamri, Ahmed

Abstract

Determining when instructor intervention is needed, based on learners’ comments and their urgency in massive open online course (MOOC) environments, is a known challenge. To solve this challenge, prior art used autonomous machine learning (ML) models. These models are described as having a "black-box" nature, and their output is incomprehensible to humans. This paper shows how to apply eXplainable Artificial Intelligence (XAI) techniques to interpret a MOOC intervention model for urgent comments detection. As comments were selected from the MOOC course and annotated using human experts, we additionally study the confidence between annotators (annotator agreement confidence), versus an estimate of the class score of making a decision via ML, to support intervention decision. Serendipitously, we show, for the first time, that XAI can be further used to support annotators creating high-quality, gold standard datasets for urgent intervention.

Citation

Alrajhi, L., Pereira, F. D., Cristea, A. I., & Alamri, A. (2023, September). Serendipitous Gains of Explaining a Classifier - Artificial versus Human Performance and Annotator Support in an Urgent Instructor-Intervention Model for MOOCs. Paper presented at HT '23: 34th ACM Conference on Hypertext and Social Media, Rome Italy

• Presentation Conference Type: Conference Paper (unpublished)
• Conference Name: HT '23: 34th ACM Conference on Hypertext and Social Media
• Start Date: Sep 4, 2023
• End Date: Sep 8, 2023
• Publication Date: Sep 4, 2023
• Deposit Date: Nov 7, 2023
• Publicly Available Date: Nov 7, 2023
• DOI: https://doi.org/10.1145/3603607.3613480
• Public URL: https://durham-repository.worktribe.com/output/1899258