Document Open Access Logo

Authoring Programming Exercises for Automated Assessment Assisted by Generative AI

Authors Yannik Bauer , José Paulo Leal , Ricardo Queirós

PDF
Thumbnail PDF

File

OASIcs.ICPEC.2024.21.pdf
  • Filesize: 0.6 MB
  • 8 pages

Document Identifiers

Author Details

Yannik Bauer
  • DCC - FCUP, Porto, Portugal
José Paulo Leal
  • CRACS - INESC TEC, Porto, Portugal
  • DCC - FCUP, Porto, Portugal
Ricardo Queirós
  • CRACS - INESC TEC, Porto, Portugal
  • uniMAD - ESMAD, Polytechnic of Porto, Portugal

Funding

This work is co-funded by the Erasmus+ Programme of the European Union within the project FGPEPlusPlus, with Agreement Number 2023-1-PL01-KA220-HED-000164696.

Cite AsGet BibTex

Yannik Bauer, José Paulo Leal, and Ricardo Queirós. Authoring Programming Exercises for Automated Assessment Assisted by Generative AI. In 5th International Computer Programming Education Conference (ICPEC 2024). Open Access Series in Informatics (OASIcs), Volume 122, pp. 21:1-21:8, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024)
https://doi.org/10.4230/OASIcs.ICPEC.2024.21

Abstract

Generative AI presents both challenges and opportunities for educators. This paper explores its potential for automating the creation of programming exercises designed for automated assessment. Traditionally, creating these exercises is a time-intensive and error-prone task that involves developing exercise statements, solutions, and test cases. This ongoing research analyzes the capabilities of the OpenAI GPT API to automatically create these components. An experiment using the OpenAI GPT API to automatically create 120 programming exercises produced interesting results, such as the difficulties encountered in generating valid JSON formats and creating matching test cases for solution code. Learning from this experiment, an enhanced feature was developed to assist teachers in creating programming exercises and was integrated into Agni, a virtual learning environment (VLE). Despite the challenges in generating entirely correct programming exercises, this approach shows potential for reducing the time required to create exercises, thus significantly aiding teachers. The evaluation of this approach, comparing the efficiency and usefulness of using the OpenAI GPT API or authoring the exercises oneself, is in progress.

Subject Classification

ACM Subject Classification
  • Applied computing → Computer-assisted instruction
  • Computing methodologies → Artificial intelligence
  • Applied computing → Interactive learning environments
Keywords
  • ChatGPT
  • generative AI
  • programming exercises
  • automated assessment

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Eman A. Alasadi and Carlos R. Baiz. Generative ai in education and research: Opportunities, concerns, and solutions. Journal of Chemical Education, 100(8):2965-2971, 2023. URL: https://doi.org/10.1021/acs.jchemed.3c00323.
  2. Brett A Becker, Michelle Craig, Paul Denny, Hieke Keuning, Natalie Kiesler, Juho Leinonen, Andrew Luxton-Reilly, James Prather, and Keith Quille. Generative ai in introductory programming. Name of Journal, 2023. Google Scholar
  3. Brett A. Becker, Paul Denny, James Finnie-Ansley, Andrew Luxton-Reilly, James Prather, and Eddie Antonio Santos. Programming is hard - or at least it used to be: Educational opportunities and challenges of ai code generation. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2023, pages 500-506, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3545945.3569759.
  4. Erik Brynjolfsson, Danielle Li, and Lindsey Raymond. Generative ai at work, 2023. URL: https://arxiv.org/abs/2304.11771.
  5. Paul Denny, Hassan Khosravi, Arto Hellas, Juho Leinonen, and Sami Sarsa. Can we trust ai-generated educational content? comparative analysis of human and ai-generated learning resources, 2023. URL: https://arxiv.org/abs/2306.10509.
  6. Paul Denny, Sami Sarsa, Arto Hellas, and Juho Leinonen. Robosourcing educational resources - leveraging large language models for learnersourcing, 2022. URL: https://arxiv.org/abs/2211.04715.
  7. Enterprise DNA Experts. How to use chat gpt: A simple guide for beginners, September 2023. URL: https://blog.enterprisedna.co/how-to-use-chat-gpt/.
  8. Stefan Feuerriegel, Jochen Hartmann, Christian Janiesch, and Patrick Zschech. Generative ai. SSRN Electronic Journal, January 2023. URL: https://doi.org/10.2139/ssrn.4443189.
  9. Aaron Issac. Chatgpt functions malformed json, July 2023. URL: https://community.openai.com/t/chatgpt-functions-malformed-json/306509.
  10. Lukas Jürgensmeier and Bernd Skiera. Generative ai for scalable feedback to multimodal exercises in marketing analytics. Available at SSRN, 2024. Google Scholar
  11. Richard Lobb and Jenny Harlow. Coderunner: a tool for assessing computer programming skills. ACM Inroads, 7(1):47-51, February 2016. URL: https://doi.org/10.1145/2810041.
  12. Joseph Martinez. Return json from gpt, July 2023. URL: https://betterprogramming.pub/return-json-from-gpt-65d40bfc2ef6.
  13. Rytis Maskeliūnas, Robertas Damaševičius, Tomas Blažauskas, Jakub Swacha, Ricardo Queirós, and José Carlos Paiva. Fgpe+: The mobile fgpe environment and the pareto-optimized gamified programming exercise selection model—an empirical evaluation. Computers, 12(7):144, 2023. Google Scholar
  14. Rosario Michel-Villarreal, Eliseo Vilalta-Perdomo, David Ernesto Salinas-Navarro, Ricardo Thierry-Aguilera, and Flor Silvestre Gerardou. Challenges and opportunities of generative ai for higher education as explained by chatgpt. Education Sciences, 13(9), 2023. URL: https://doi.org/10.3390/educsci13090856.
  15. José Carlos Paiva, Ricardo Queirós, José Paulo Leal, and Jakub Swacha. Fgpe authorkit-a tool for authoring gamified programming educational content. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education, pages 564-564, 2020. Google Scholar
  16. José Carlos Paiva, Ricardo Queirós, José Paulo Leal, and Jakub Swacha. Yet another programming exercises interoperability language (short paper). In 9th Symposium on Languages, Applications and Technologies (SLATE 2020). Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020. Google Scholar
  17. Ivica Pesovski, Ricardo Santos, Roberto Henriques, and Vladimir Trajkovik. Generative ai for customizable learning experiences. Sustainability, 16(7), 2024. URL: https://doi.org/10.3390/su16073034.
  18. Prajish Prasad and Aamod Sane. A self-regulated learning framework using generative ai and its application in cs educational intervention design. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2024, pages 1070-1076, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3626252.3630828.
  19. Junaid Qadir. Engineering education in the era of chatgpt: Promise and pitfalls of generative ai for education. In 2023 IEEE Global Engineering Education Conference (EDUCON), pages 1-9, 2023. URL: https://doi.org/10.1109/EDUCON54358.2023.10125121.
  20. Ricardo Alexandre Peixoto de Queirós. Integration of a learning playground into a lms. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2, pages 626-626, 2022. Google Scholar
  21. Laurie Ruettimann. How to interact with chat gpt-4 effectively: Top tips for better questions, September 2023. URL: https://laurieruettimann.com/chat-gpt-4-ask-questions/.
  22. Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1, ICER '22, pages 27-43, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3501385.3543957.
  23. Jiahong Su and Weipeng Yang. Unlocking the power of chatgpt: A framework for applying generative ai in education. ECNU Review of Education, 6:1-12, April 2023. URL: https://doi.org/10.1177/20965311231168423.
  24. John Wrenn, Shriram Krishnamurthi, and Kathi Fisler. Who tests the testers? In Proceedings of the 2018 ACM Conference on International Computing Education Research, ICER '18, pages 51-59, New York, NY, USA, 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3230977.3230999.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact