Evaluation of a hybrid AI-human recommender for CS1 instructors in a real educational scenario

Dwan Pereira, Filipe; Oliveira, Elaine; Rodrigues, Luiz; Cabral, Luciano; Oliveira, David; Carvalho, Leandro; Gasevic, Dragan; Cristea, Alexandra; Dermeval, Diego; Ferreira Mello, Rafael

Abstract

Automatic code graders, also called Programming Online Judges (OJ), can support students and instructors in introduction to programming courses (CS1). Using OJs in CS1, instructors select problems to compose assignment lists, whereas students submit their code solutions and receive instantaneous feedback. Whilst this process reduces the instructors’ workload in evaluating students’ code, selecting problems to compose assignments is arduous. Recently, recommender systems have been proposed by the literature to support OJ users. Nonetheless, there is a lack of recommenders fitted for CS1 courses and the ones found in the literature have not been assessed by the target users in a real educational scenario. It is worth noting that hybrid human/AI systems are claimed to potentially surpass isolated human or AI. In this study, we adapted and evaluated a state-of-the-art hybrid human/AI recommender to support CS1 instructors in selecting problems to compose variations of CS1 assignments. We compared data-driven measures (e.g., time students take to solve problems, number of logical lines of code, and hit rate) extracted from student logs whilst solving programming problems from assignments created by instructors versus when solving assignments in collaboration with an adaptation of cutting-edge hybrid/AI method. As a result, employing a data analysis comparing experimental and control conditions using multi-level regressions, we observed that the recommender provided problems compatible with human-selected in all data-driven measures tested.

Citation

Dwan Pereira, F., Oliveira, E., Rodrigues, L., Cabral, L., Oliveira, D., Carvalho, L., …Ferreira Mello, R. (2023). Evaluation of a hybrid AI-human recommender for CS1 instructors in a real educational scenario. In Responsive and Sustainable Educational Futures: 18th European Conference on Technology Enhanced Learning, EC-TEL 2023, Aveiro, Portugal, September 4–8, 2023, Proceedings (308-323). https://doi.org/10.1007/978-3-031-42682-7_21

• Conference Name: Eighteenth European Conference on Technology Enhanced Learning: ECTEL 2023
• Conference Location: Aveiro, Portugal
• Start Date: Sep 4, 2023
• End Date: Sep 8, 2023
• Acceptance Date: May 27, 2023
• Online Publication Date: Aug 28, 2023
• Publication Date: 2023
• Deposit Date: Aug 16, 2023
• Publicly Available Date: Aug 29, 2024
• Publisher: Springer
• Pages: 308-323
• Series Title: Lecture Notes in Computer Science
• Series Number: 14200
• Series ISSN: 0302-9743
• Book Title: Responsive and Sustainable Educational Futures: 18th European Conference on Technology Enhanced Learning, EC-TEL 2023, Aveiro, Portugal, September 4–8, 2023, Proceedings
• ISBN: 9783031426810
• DOI: https://doi.org/10.1007/978-3-031-42682-7_21
• Public URL: https://durham-repository.worktribe.com/output/1718793
• Publisher URL: https://link.springer.com/conference/ectel