The death of the short-form physics essay in the coming AI revolution

Yeadon, Will; Inyang, Oto-Obong; Mizouri, Arin; Peach, Alex; Testrow, Craig P

Abstract

The latest AI language modules can produce original, high quality full short-form (300-word) Physics essays within seconds. These technologies such as ChatGPT and davinci-003 are freely available to anyone with an internet connection. In this work, we present evidence of AI generated short-form essays achieving First-Class grades on an essay writing assessment from an accredited, current university Physics module. The assessment requires students answer five open-ended questions with a short, 300-word essay each. Fifty AI answers were generated to create ten submissions that were independently marked by five separate markers. The AI generated submissions achieved an average mark of 71 ± 2%, in strong agreement with the current module average of 71 ± 5%. A typical AI submission would therefore most-likely be awarded a First Class, the highest classification available at UK universities. Plagiarism detection software returned a plagiarism score between 2 ± 1% (Grammarly) and 7 ± 2% (TurnitIn). We argue that these results indicate that current natural language processing AI represent a significant threat to the fidelity of short-form essays as an assessment method in Physics courses.

Citation

Yeadon, W., Inyang, O., Mizouri, A., Peach, A., & Testrow, C. P. (2023). The death of the short-form physics essay in the coming AI revolution. Physics Education, 58(3), https://doi.org/10.1088/1361-6552/acc5cf

• Journal Article Type: Article
• Acceptance Date: Mar 20, 2023
• Online Publication Date: Apr 5, 2023
• Publication Date: 2023
• Deposit Date: Jul 3, 2023
• Publicly Available Date: Jul 3, 2023
• Journal: Physics Education
• Print ISSN: 0031-9120
• Electronic ISSN: 1361-6552
• Publisher: IOP Publishing
• Peer Reviewed: Peer Reviewed
• Volume: 58
• Issue: 3
• DOI: https://doi.org/10.1088/1361-6552/acc5cf