Document Open Access Logo

Summative Assessment of Program Comprehension in CS1 with Questions About Learners' Code

Authors Afonso B. Caniço , André L. Santos

PDF

File

Thumbnail PDF
PDF
OASIcs.ICPEC.2026.1.pdf
  • Filesize: 0.86 MB
  • 15 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Social and professional topics → Software engineering education
  • Social and professional topics → Student assessment
Keywords
  • introductory programming
  • CS1
  • programming education
  • student assessment
  • program comprehension
  • questions about learners' code

Metrics

Abstract

Questions about Learners' Code (QLCs) assess to what extent learners understand their own code through personalized questions that target previously written code. We conducted an experiment involving 18 students of an Introductory Programming (CS1) course taught in Java, comparing the scores of their midterm test based on code-writing questions with the scores of a multiple-choice test based on QLCs. The questions targeted student code submitted throughout the semester in an automated assessment system. Regression analysis shows a strong association between the scores of both tests, suggesting comparable discriminatory power. Participants generally reacted positively and recognised the benefits of this kind of assessment. Our results suggest that QLC tests may serve as a valid complement for CS1 summative assessment targeting program comprehension.

Cite As Get BibTex

Afonso B. Caniço and André L. Santos. Summative Assessment of Program Comprehension in CS1 with Questions About Learners' Code. In 7th International Computer Programming Education Conference (ICPEC 2026). Open Access Series in Informatics (OASIcs), Volume 145, pp. 1:1-1:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/OASIcs.ICPEC.2026.1

Author Details

Afonso B. Caniço
  • ISTAR, Instituto Universitário de Lisboa (ISCTE-IUL), Portugal
André L. Santos
  • ISTAR, Instituto Universitário de Lisboa (ISCTE-IUL), Portugal

Funding

This research was partially supported by national funds through Fundação para a Ciência e a Tecnologia, I.P. (FCT), ISTAR-Iscte Pluriannual Projects UID/04466/2025 (https://doi.org/10.54499/UID/04466/2025), and an ISCTE-IUL scholarship.

Acknowledgements

We thank the anonymous participants for taking part in the study.

Supplementary Materials

References

  1. Afonso B. Caniço and André L. Santos. ambco-iscte/jask. Software, version 0.7.3. (visited on 2026-07-08). URL: https://github.com/ambco-iscte/jask
    full metadata available at: https://doi.org/10.4230/artifacts.27040
  2. Pablo Frank Bolton, Liberty Rose Lehr, Rahul Simha, and Michelle Lawson. The justification effect on two-tier multiple-choice exams. In 2024 ASEE Annual Conference & Exposition, Portland, Oregon, June 2024. ASEE Conferences. URL: https://doi.org/10.18260/1-2--48121.
  3. W.J. Buchanan and L. Saliou. Enhanced methods of coursework provision in computer networks. In ITRE 2004. 2nd International Conference Information Technology: Research and Education, pages 111-115, London, UK, 2004. IEEE. URL: https://doi.org/10.1109/ITRE.2004.1393657.
  4. Teresa Busjahn and Carsten Schulte. The use of code reading in teaching programming. In Proceedings of the 13th Koli Calling International Conference on Computing Education Research, Koli Calling '13, pages 3-11, New York, NY, USA, November 2013. Association for Computing Machinery. URL: https://doi.org/10.1145/2526968.2526969.
  5. Luca Chiodini, Igor Moreno Santos, Andrea Gallidabino, Anya Tafliovich, André L. Santos, and Matthias Hauswirth. A Curated Inventory of Programming Language Misconceptions. In Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE '21, pages 380-386, New York, NY, USA, June 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3430665.3456343.
  6. James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Proceedings of the 24th Australasian Computing Education Conference, ACE '22, pages 10-19, New York, NY, USA, February 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3511861.3511863.
  7. James Finnie-Ansley, Paul Denny, Andrew Luxton-Reilly, Eddie Antonio Santos, James Prather, and Brett A. Becker. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. In Proceedings of the 25th Australasian Computing Education Conference, ACE '23, pages 97-104, New York, NY, USA, January 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3576123.3576134.
  8. Genady Ya. Grabarnik and Serge Yaskolko. Automated partial credit for stem classes. In 2019 IEEE Frontiers in Education Conference (FIE), pages 1-5, Covington, KY, USA, 2019. IEEE. URL: https://doi.org/10.1109/FIE43999.2019.9028473.
  9. Manu Kapur. Examining productive failure, productive success, unproductive failure, and unproductive success in learning. Educational Psychologist, 51(2):289-299, 2016. URL: https://doi.org/10.1080/00461520.2016.1155457.
  10. Cazembe Kennedy and Eileen T. Kraemer. Qualitative observations of student reasoning: Coding in the wild. In Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE '19, pages 224-230, New York, NY, USA, July 2019. Association for Computing Machinery. URL: https://doi.org/10.1145/3304221.3319751.
  11. Päivi Kinnunen and Beth Simon. My program is ok - am i? computing freshmen’s experiences of doing programming assignments. Computer Science Education, 22(1):1-28, March 2012. URL: https://doi.org/10.1080/08993408.2012.655091.
  12. Teemu Lehtinen, Lassi Haaranen, and Juho Leinonen. Automated questionnaires about students’ javascript programs: Towards gauging novice programming processes. In Proceedings of the 25th Australasian Computing Education Conference, ACE '23, pages 49-58, New York, NY, USA, January 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3576123.3576129.
  13. Teemu Lehtinen, Aleksi Lukkarinen, and Lassi Haaranen. Students struggle to explain their own program code. In Carsten Schulte, Brett A. Becker, Monica Divitini, and Erik Barendsen, editors, Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE '21, pages 206-212, New York, NY, USA, June 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3430665.3456322.
  14. Teemu Lehtinen, André L. Santos, and Juha Sorva. Let's ask students about their programs, automatically. In Proceedings - 2021 IEEE/ACM 29th International Conference on Program Comprehension, ICPC 2021, Proceedings/IEEE International Conference on Program Comprehension, pages 467-475, United States, May 2021. IEEE. International Conference on Program Comprehension, ICPC ; Conference date: 20-05-2021 Through 21-05-2021. URL: https://doi.org/10.1109/ICPC52881.2021.00054.
  15. Teemu Lehtinen, Otto Seppälä, and Ari Korhonen. Automated questions about learners' own code help to detect fragile prerequisite knowledge. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1, ITiCSE 2023, pages 505-511, New York, NY, USA, June 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3587102.3588787.
  16. Rachel S. Lim, Sophia Krause-Levy, Ismael Villegas Molina, and Leo Porter. Student expectations of tutors in computing courses. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2023, pages 437-443, New York, NY, USA, March 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3545945.3569766.
  17. Raymond Lister, Colin Fidge, and Donna Teague. Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. SIGCSE Bull., 41(3):161-165, July 2009. URL: https://doi.org/10.1145/1595496.1562930.
  18. Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. Relationships between reading, tracing and writing skills in introductory programming. In Proceedings of the Fourth international Workshop on Computing Education Research, ICER '08, pages 101-112, New York, NY, USA, September 2008. Association for Computing Machinery. URL: https://doi.org/10.1145/1404520.1404531.
  19. Andrew Luxton-Reilly, Simon, Ibrahim Albluwi, Brett A. Becker, Michail Giannakos, Amruth N. Kumar, Linda Ott, James Paterson, Michael James Scott, Judy Sheard, and Claudia Szabo. Introductory programming: a systematic literature review. In Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE 2018 Companion, pages 55-106, New York, NY, USA, 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3293881.3295779.
  20. Sandra Madison and James Gifford. Modular Programming: Novice Misconceptions. Journal of Research on Technology in Education, 34(3):217-229, March 2002. URL: https://doi.org/10.1080/15391523.2002.10782346.
  21. Greg L. Nelson, Benjamin Xie, and Amy J. Ko. Comprehension first: Evaluating a novel pedagogy and tutoring system for program tracing in cs1. In Proceedings of the 2017 ACM Conference on International Computing Education Research, ICER'17, pages 2-11, New York, NY, USA, August 2017. Association for Computing Machinery. URL: https://doi.org/10.1145/3105726.3106178.
  22. Sara Nurollahian, Noelle Brown, Anna N. Rafferty, and Eliane Wiese. A Systematic Review of Approaches for Evaluating Students’ Function-Level Code Structure Knowledge with a Catalog of All Discussed Patterns. ACM Transactions on Computing Education, page 3787450, January 2026. URL: https://doi.org/10.1145/3787450.
  23. Sara Nurollahian, Anna N. Rafferty, Noelle Brown, and Eliane Wiese. Growth in knowledge of programming patterns: A comparison study of cs1 vs. cs2 students. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2024, pages 979-985, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3626252.3630865.
  24. José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. Automated assessment in computer science education: A state-of-the-art review. ACM Trans. Comput. Educ., 22(3), June 2022. URL: https://doi.org/10.1145/3513140.
  25. James Prather, Brent N Reeves, Juho Leinonen, Stephen MacNeil, Arisoa S Randrianasolo, Brett A. Becker, Bailey Kimmel, Jared Wright, and Ben Briggs. The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers. In Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1, volume 1 of ICER '24, pages 469-486, New York, NY, USA, August 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3632620.3671116.
  26. Yizhou Qian and James Lehman. Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review. ACM Trans. Comput. Educ., 18(1):1:1-1:24, October 2017. URL: https://doi.org/10.1145/3077618.
  27. Jean Salac and Diana Franklin. If they build it, will they understand it? exploring the relationship between student code and performance. In Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE '20, pages 473-479, New York, NY, USA, 2020. Association for Computing Machinery. URL: https://doi.org/10.1145/3341525.3387379.
  28. André L. Santos, Tiago Soares, Nuno Garrido, and Teemu Lehtinen. Jask: Generation of questions about learners' code in java. In Brett A. Becker, Keith Quille, Mikko-Jussi Laakso, Erik Barendsen, and Simon, editors, Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1, ITiCSE '22, pages 117-123, New York, NY, USA, July 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3502718.3524761.
  29. Jaromir Savelka, Arav Agarwal, Marshall An, Chris Bogart, and Majd Sakr. Thrilled by your progress! large language models (gpt-4) no longer struggle to pass assessments in higher education programming courses. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1, ICER '23, pages 78-92, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3568813.3600142.
  30. Valerie J. Shute. Focus on formative feedback. Review of Educational Research, 78(1):153-189, 2008. URL: https://doi.org/10.3102/0034654307313795.
  31. Anne Venables, Grace Tan, and Raymond Lister. A closer look at tracing, explaining and code writing skills in the novice programmer. In Proceedings of the fifth international workshop on Computing education research workshop, ICER '09, pages 117-128, New York, NY, USA, August 2009. Association for Computing Machinery. URL: https://doi.org/10.1145/1584322.1584336.
  32. Benjamin Xie, Dastyni Loksa, Greg L Nelson, Matthew J Davidson, Dongsheng Dong, Harrison Kwik, Alex Hui Tan, Leanne Hwa, Min Li, and Amy J Ko. A theory of instruction for introductory programming skills. Computer Science Education, 29(2-3):1-49, January 2019. Google Scholar
  33. Daniel Zingaro, Michelle Craig, Leo Porter, Brett A. Becker, Yingjun Cao, Phill Conrad, Diana Cukierman, Arto Hellas, Dastyni Loksa, and Neena Thota. Achievement goals in cs1: Replication and extension. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE '18, pages 687-692, New York, NY, USA, February 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3159450.3159452.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact