Document Open Access Logo

Integrating Questions About Learners’ Code in an Automated Assessment System

Authors Afonso B. Caniço , André L. Santos

PDF HTML 5

Files

Thumbnail PDF
PDF
OASIcs.ICPEC.2025.5.pdf
  • Filesize: 0.75 MB
  • 14 pages
HTML5 Logo HTML (experimental)
OASIcs.ICPEC.2025.5.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Social and professional topics → Software engineering education
  • Social and professional topics → Student assessment
Keywords
  • programming education
  • student assessment
  • program comprehension
  • questions about learners’ code

Metrics

Abstract

Questions about Learners' Code (QLCs) assess programming students' program comprehension skills by providing personalised questions targeting the students' own program code. We conducted a preliminary, experimental implementation of integrating QLCs in the Automated Assessment System (AAS) used in an introductory programming course using Java. QLCs targeted some of the code assignments which students had to complete during the course. We collected 889 answers to QLCs, answered by 13 students over five course modules. We found that as the complexity of exercises increases, the success rate of the same type of QLC may not improve, and even exhibit a decline over time. We further analysed incorrect answers individually to relate them to possible misconceptions.

Cite As Get BibTex

Afonso B. Caniço and André L. Santos. Integrating Questions About Learners’ Code in an Automated Assessment System. In 6th International Computer Programming Education Conference (ICPEC 2025). Open Access Series in Informatics (OASIcs), Volume 133, pp. 5:1-5:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.ICPEC.2025.5

Author Details

Afonso B. Caniço
  • Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR, Portugal
André L. Santos
  • Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR, Portugal

Funding

This work was partially supported by Fundação para a Ciência e a Tecnologia, I.P. (FCT) [ISTAR Projects: UIDB/04466/2020 and UIDP/04466/2020].

References

  1. Joey Bevilacqua, Luca Chiodini, Igor Moreno Santos, and Matthias Hauswirth. Using notional machines to automatically assess students' comprehension of their own code. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 2, SIGCSE 2024, pages 1572-1573, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3626253.3635524.
  2. Teresa Busjahn and Carsten Schulte. The use of code reading in teaching programming. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research, Koli Calling '13, pages 3-11, New York, NY, USA, November 2013. Association for Computing Machinery. URL: https://doi.org/10.1145/2526968.2526969.
  3. Luca Chiodini, Igor Moreno Santos, Andrea Gallidabino, Anya Tafliovich, André L. Santos, and Matthias Hauswirth. A curated inventory of programming language misconceptions. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE '21, pages 380-386, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3430665.3456343.
  4. James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. The robots are coming: Exploring the implications of openai codex on introductory programming. In Proceedings of the 24th Australasian Computing Education Conference, ACE '22, pages 10-19, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3511861.3511863.
  5. Cazembe Kennedy and Eileen T. Kraemer. Qualitative observations of student reasoning: Coding in the wild. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE '19, pages 224-230, New York, NY, USA, 2019. Association for Computing Machinery. URL: https://doi.org/10.1145/3304221.3319751.
  6. Päivi Kinnunen and Beth Simon. My program is ok - am i? computing freshmen’s experiences of doing programming assignments. Computer Science Education, 22(1):1-28, 2012. URL: https://doi.org/10.1080/08993408.2012.655091.
  7. Teemu Lehtinen, Lassi Haaranen, and Juho Leinonen. Automated questionnaires about students’ javascript programs: Towards gauging novice programming processes. In Proceedings of the 25th Australasian Computing Education Conference, ACE '23, pages 49-58, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3576123.3576129.
  8. Teemu Lehtinen, Aleksi Lukkarinen, and Lassi Haaranen. Students struggle to explain their own program code. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE '21, pages 206-212, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3430665.3456322.
  9. Teemu Lehtinen, André L. Santos, and Juha Sorva. Let's ask students about their programs, automatically. In Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021, Proceedings/IEEE International Conference on Program Comprehension, pages 467-475, United States, May 2021. IEEE. International Conference on Program Comprehension, ICPC ; Conference date: 20-05-2021 Through 21-05-2021. URL: https://doi.org/10.1109/ICPC52881.2021.00054.
  10. Teemu Lehtinen, Otto Seppälä, and Ari Korhonen. Automated questions about learners' own code help to detect fragile prerequisite knowledge. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2023, pages 505-511, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3587102.3588787.
  11. Rachel S. Lim, Sophia Krause-Levy, Ismael Villegas Molina, and Leo Porter. Student expectations of tutors in computing courses. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2023, pages 437-443, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3545945.3569766.
  12. Raymond Lister, Colin Fidge, and Donna Teague. Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. SIGCSE Bull., 41(3):161-165, July 2009. URL: https://doi.org/10.1145/1595496.1562930.
  13. Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. Relationships between reading, tracing and writing skills in introductory programming. In Proceedings of the Fourth international Workshop on Computing Education Research, ICER '08, pages 101-112, New York, NY, USA, September 2008. Association for Computing Machinery. URL: https://doi.org/10.1145/1404520.1404531.
  14. Sandra Madison and James Gifford. Modular programming. Journal of Research on Technology in Education, 34(3):217-229, 2002. URL: https://doi.org/10.1080/15391523.2002.10782346.
  15. Marry L. McHugh. Interrater reliability: the kappa statistic. Biochemia Medica, pages 276-282, 2012. URL: https://doi.org/10.11613/bm.2012.031.
  16. James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randrianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. The widening gap: The benefits and harms of generative ai for novice programmers. In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1, ICER '24, pages 469-486, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3632620.3671116.
  17. Yizhou Qian and James Lehman. Students’ misconceptions and other difficulties in introductory programming: A literature review. ACM Trans. Comput. Educ., 18(1), October 2017. URL: https://doi.org/10.1145/3077618.
  18. André L. Santos, Tiago Soares, Nuno Garrido, and Teemu Lehtinen. Jask: Generation of questions about learners' code in java. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1, ITiCSE '22, pages 117-123, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3502718.3524761.
  19. Benjamin Xie, Dastyni Loksa, Greg L Nelson, Matthew J Davidson, Dongsheng Dong, Harrison Kwik, Alex Hui Tan, Leanne Hwa, Min Li, and Amy J Ko. A theory of instruction for introductory programming skills. Computer Science Education, pages 1-49, January 2019. Google Scholar
  20. Daniel Zingaro, Michelle Craig, Leo Porter, Brett A. Becker, Yingjun Cao, Phill Conrad, Diana Cukierman, Arto Hellas, Dastyni Loksa, and Neena Thota. Achievement goals in cs1: Replication and extension. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE '18, pages 687-692, New York, NY, USA, 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3159450.3159452.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact