Interpretable AI to Understand Early Effective and Ineffective Programming Behaviours from CS1 Learners

Pereira, Filipe Dwan; Oliveira, Elaine Harada Teixeira de; Oliveira, David Braga Fernandes de; Carvalho, Leandro Silva Galvão de; Cristea, Alexandra I.

Abstract

Building predictive models to estimate the learner performance in the beginning of CS1 courses is essential in education to allow early interventions. However, the educational literature notes the lack of studies on early learner behaviours that can be effective or ineffective, that is, programming behaviours that potentially lead to success or failure, respectively. Hence, beyond the prediction, it is crucial to explain what leads the predictive model to make the decisions (e.g., why a given student s is classified as `passed'), which would allow a better understanding of which early programming behaviours are to be encouraged and triggered. In this work in progress, we use a state-of-the-art unified approach to interpret black-box model predictions, which uses SHapley Additive exPlanations (SHAP) method. SHAP method can be used to explain linearly a complex model (e.g. DL or XGboost) in instance level. In our context of CS1 performance prediction, this method gets the predictive model and the features values for a given student as input and the possibility of explanation of which feature values are increasing or decreasing the learner chances of passing as output. That is, using SHAP we can identify early effective and ineffective behaviours in student-level granularity. More than that, using this local explanation as building blocks, we can also extract global data insight and give a summarisation of the model. A video explaining this work can be found at the following link (in Brazilian Portuguese): https://youtu.be/pd6Ma6uInHo

Citation

Pereira, F. D., Oliveira, E. H. T. D., Oliveira, D. B. F. D., Carvalho, L. S. G. D., & Cristea, A. I. (2021, April). Interpretable AI to Understand Early Effective and Ineffective Programming Behaviours from CS1 Learners. Presented at Anais Estendidos do I Simpósio Brasileiro de Educação em Computação (EDUCOMP Estendido 2021), Online

• Presentation Conference Type: Conference Paper (published)
• Conference Name: Anais Estendidos do I Simpósio Brasileiro de Educação em Computação (EDUCOMP Estendido 2021)
• Start Date: Apr 26, 2021
• End Date: Apr 30, 2021
• Online Publication Date: Apr 26, 2021
• Publication Date: 2021
• Deposit Date: Nov 5, 2021
• Publicly Available Date: Nov 5, 2021
• Pages: 16-17
• DOI: https://doi.org/10.5753/educomp_estendido.2021.14853
• Public URL: https://durham-repository.worktribe.com/output/1138200