Skip to main content

Research Repository

Advanced Search

Explaining Individual and Collective Programming Students’ Behavior by Interpreting a Black-Box Predictive Model

Pereira, Filipe Dwan; Fonseca, Samuel C.; Oliveira, Elaine H.T.; Cristea, Alexandra I.; Bellhauser, Henrik; Rodrigues, Luiz; Oliveira, David B.F.; Isotani, Seiji; Carvalho, Leandro S.G.

Explaining Individual and Collective Programming Students’ Behavior by Interpreting a Black-Box Predictive Model Thumbnail


Authors

Filipe Dwan Pereira

Samuel C. Fonseca

Elaine H.T. Oliveira

Henrik Bellhauser

Luiz Rodrigues

David B.F. Oliveira

Seiji Isotani

Leandro S.G. Carvalho



Abstract

Predicting student performance as early as possible and analysing to which extent initial student behaviour could lead to failure or success is critical in introductory programming (CS1) courses, for allowing prompt intervention in a move towards alleviating their high failure rate. However, in CS1 performance prediction, there is a serious lack of studies that interpret the predictive model’s decisions. In this sense, we designed a long-term study using very fine-grained log-data of 2056 students, collected from the first two weeks of CS1 courses. We extract features that measure how students deal with deadlines, how they fix errors, how much time they spend programming, and so forth. Subsequently, we construct a predictive model that achieved cutting-edge results with area under the curve (AUC) of.89, and an average accuracy of 81.3%. To allow an effective intervention and to facilitate human-AI collaboration towards prescriptive analytics, we, for the first time, to the best of our knowledge, go a step further than the prediction itself and leverage this field by proposing an approach to explaining our predictive model decisions individually and collectively using a game-theory based framework (SHAP), (Lundberg et al. , 2020) that allows interpreting our black-box non-linear model linearly. In other words, we explain the feature effects, clearly by visualising and analysing individual predictions, the overall importance of features, and identification of typical prediction paths. This method can be further applied to other emerging competitive models, as the CS1 prediction field progresses, ensuring transparency of the process for key stakeholders: administrators, teachers, and students.

Citation

Pereira, F. D., Fonseca, S. C., Oliveira, E. H., Cristea, A. I., Bellhauser, H., Rodrigues, L., …Carvalho, L. S. (2021). Explaining Individual and Collective Programming Students’ Behavior by Interpreting a Black-Box Predictive Model. IEEE Access, 9, 117097-117119. https://doi.org/10.1109/access.2021.3105956

Journal Article Type Article
Online Publication Date Aug 18, 2021
Publication Date 2021
Deposit Date Oct 28, 2021
Publicly Available Date Oct 28, 2021
Journal IEEE Access
Publisher Institute of Electrical and Electronics Engineers
Peer Reviewed Peer Reviewed
Volume 9
Pages 117097-117119
DOI https://doi.org/10.1109/access.2021.3105956

Files







You might also like



Downloadable Citations