Skip to main content

Research Repository

Advanced Search

Evaluation of a hybrid AI-human recommender for CS1 instructors in a real educational scenario

Dwan Pereira, Filipe; Oliveira, Elaine; Rodrigues, Luiz; Cabral, Luciano; Oliveira, David; Carvalho, Leandro; Gasevic, Dragan; Cristea, Alexandra; Dermeval, Diego; Ferreira Mello, Rafael

Authors

Filipe Dwan Pereira

Elaine Oliveira

Luiz Rodrigues

Luciano Cabral

David Oliveira

Leandro Carvalho

Dragan Gasevic

Diego Dermeval

Rafael Ferreira Mello



Abstract

Automatic code graders, also called Programming Online Judges (OJ), can support students and instructors in introduction to programming courses (CS1). Using OJs in CS1, instructors select problems to compose assignment lists, whereas students submit their code solutions and receive instantaneous feedback. Whilst this process reduces the instructors’ workload in evaluating students’ code, selecting problems to compose assignments is arduous. Recently, recommender systems have been proposed by the literature to support OJ users. Nonetheless, there is a lack of recommenders fitted for CS1 courses and the ones found in the literature have not been assessed by the target users in a real educational scenario. It is worth noting that hybrid human/AI systems are claimed to potentially surpass isolated human or AI. In this study, we adapted and evaluated a state-of-the-art hybrid human/AI recommender to support CS1 instructors in selecting problems to compose variations of CS1 assignments. We compared data-driven measures (e.g., time students take to solve problems, number of logical lines of code, and hit rate) extracted from student logs whilst solving programming problems from assignments created by instructors versus when solving assignments in collaboration with an adaptation of cutting-edge hybrid/AI method. As a result, employing a data analysis comparing experimental and control conditions using multi-level regressions, we observed that the recommender provided problems compatible with human-selected in all data-driven measures tested.

Citation

Dwan Pereira, F., Oliveira, E., Rodrigues, L., Cabral, L., Oliveira, D., Carvalho, L., …Ferreira Mello, R. (2023). Evaluation of a hybrid AI-human recommender for CS1 instructors in a real educational scenario. In Responsive and Sustainable Educational Futures: 18th European Conference on Technology Enhanced Learning, EC-TEL 2023, Aveiro, Portugal, September 4–8, 2023, Proceedings (308-323). https://doi.org/10.1007/978-3-031-42682-7_21

Presentation Conference Type Conference Paper (Published)
Conference Name Eighteenth European Conference on Technology Enhanced Learning: ECTEL 2023
Start Date Sep 4, 2023
End Date Sep 8, 2023
Acceptance Date May 27, 2023
Online Publication Date Aug 28, 2023
Publication Date 2023
Deposit Date Aug 16, 2023
Publicly Available Date Aug 29, 2024
Publisher Springer
Pages 308-323
Series Title Lecture Notes in Computer Science
Series Number 14200
Series ISSN 0302-9743
Book Title Responsive and Sustainable Educational Futures: 18th European Conference on Technology Enhanced Learning, EC-TEL 2023, Aveiro, Portugal, September 4–8, 2023, Proceedings
ISBN 9783031426810
DOI https://doi.org/10.1007/978-3-031-42682-7_21
Public URL https://durham-repository.worktribe.com/output/1718793
Publisher URL https://link.springer.com/conference/ectel