Document Open Access Logo

A Systematic Review of Formative Assessment to Support Students Learning Computer Programming

Authors Jagadeeswaran Thangaraj , Monica Ward , Fiona O’Riordan

PDF
Thumbnail PDF

File

OASIcs.ICPEC.2023.7.pdf
  • Filesize: 499 kB
  • 13 pages

Document Identifiers

Author Details

Jagadeeswaran Thangaraj
  • School of Computing, Dublin City University, Ireland
Monica Ward
  • School of Computing, Dublin City University, Ireland
Fiona O’Riordan
  • Teaching Enhancement Unit, Dublin City University, Ireland

Cite AsGet BibTex

Jagadeeswaran Thangaraj, Monica Ward, and Fiona O’Riordan. A Systematic Review of Formative Assessment to Support Students Learning Computer Programming. In 4th International Computer Programming Education Conference (ICPEC 2023). Open Access Series in Informatics (OASIcs), Volume 112, pp. 7:1-7:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2023)
https://doi.org/10.4230/OASIcs.ICPEC.2023.7

Abstract

Formative assessment aims to increase student understanding, instructor instruction, and learning by providing feedback on students' progress. The goal of this systematic review is to discover trends on formative assessment techniques used to support computer programming learners by synthesizing literature published between 2013 and 2023. 17 articles that were peer-reviewed and published in journals were examined from the initial search of 197 studies. According to the findings, all the studies were conducted at the higher education level and only a small number at the secondary school level. Overall, most studies found that motivation, scaffolding, and engagement were the three main goals of feedback, with less research finding that metacognitive goals were the intended outcomes. The two techniques for facilitating formative feedback that were used most frequently were compiler or testing based error messages and customised error messages. The importance of formative feedback is highlighted in the reviewed articles, supporting the contention that assessments used in programming courses should place a heavy emphasis on motivating students to increase their level of proficiency. This study also suggests a formative assessment that employs an adaptive strategy to evaluate the ability level of the novice students and motivate them to learn programming to acquire the necessary knowledge.

Subject Classification

ACM Subject Classification
  • Applied computing → Education
  • Social and professional topics → Computing education
  • Social and professional topics → Student assessment
Keywords
  • Automatic assessment
  • Computer programming
  • Formative assessment
  • Higher education
  • Novice programmer
  • Systematic review

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. Elizabeth Acosta-Gonzaga and Aldo Ramirez-Arellano. Scaffolding Matters? Investigating Its Role in Motivation, Engagement and Learning Achievements in Higher Education. Sustainability, 14(20), 2022. URL: https://doi.org/10.3390/su142013419.
  2. Francisca A. Adamopoulos. Learning Programming, Student Motivation, pages 1-10. Springer International Publishing, Cham, 2019. URL: https://doi.org/10.1007/978-3-319-60013-0_182-1.
  3. Felipe I. Anfurrutia, Ainhoa Álvarez, Mikel Larrañaga, and Juan-Miguel López-Gil. Integrating Formative Feedback in Introductory Programming Modules. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 13(1):3-10, 2018. URL: https://doi.org/10.1109/RITA.2018.2801898.
  4. Dhakshina Moorthy Anitha and Dhakshina Moorthy Kavitha. Online coding event as a formative assessment tool in introductory programming and algorithmic courses —A exploration study. Computer Applications in Engineering Education, 28:1580-1590, 2020. Google Scholar
  5. Enrique Barra, Sonsoles López-Pernas, Álvaro Alonso, Juan Fernando Sánchez-Rada, Aldo Gordillo, and Juan Quemada. Automated Assessment in Programming Courses: A Case Study during the COVID-19 Era. Sustainability, 12(18), 2020. URL: https://doi.org/10.3390/su12187451.
  6. Mariette Bengtsson. How to plan and perform a qualitative study using content analysis. NursingPlus Open, 2:8-14, 2016. URL: https://doi.org/10.1016/j.npls.2016.01.001.
  7. Luciana Benotti, Mara Cecilia Martnez, and Fernando Schapachnik. A Tool for Introducing Computer Science with Automatic Formative Assessment. IEEE Transactions on Learning Technologies, 11(2):179-192, 2018. URL: https://doi.org/10.1109/TLT.2017.2682084.
  8. Anis Bey, Patrick Jermann, and Pierre Dillenbourg. A Comparison between Two Automatic Assessment Approaches for Programming: An Empirical Study on MOOCs. Journal of Educational Technology & Society, 21(2):259-272, 2018. URL: http://www.jstor.org/stable/26388406.
  9. Sébastien Combéfis. Automated Code Assessment for Education: Review, Classification and Perspectives on Techniques and Tools. Software, 1(1):3-30, 2022. URL: https://doi.org/10.3390/software1010002.
  10. David T. Conley and Elizabeth M. French. Student Ownership of Learning as a Key Component of College Readiness. American Behavioral Scientist, 58(8):1018-1034, 2014. URL: https://doi.org/10.1177/0002764213515232.
  11. Pedro Delgado-Pérez and Inmaculada Medina-Bulo. Customizable and scalable automated assessment of C/C++ programming assignments. Computer Applications in Engineering Education, 28:1449-1466, 2020. Google Scholar
  12. Suzanne W. Dietrich, Don Goelman, Jennifer Broatch, Sharon M. Crook, Becky Ball, Kimberly Kobojek, and Jennifer Ortiz. Using Formative Assessment for Improving Pedagogy: Reflections on Feedback Informing Database Visualizations. ACM Inroads, 11(4):27-34, November 2020. URL: https://doi.org/10.1145/3430766.
  13. Hagit Gabbay and Anat Cohen. Investigating the effect of Automated Feedback on learning behavior in MOOCs for programming. In Proceedings of the 15th International Conference on Educational Data Mining, pages 376-383. International Educational Data Mining Society, July 2022. URL: https://doi.org/10.5281/zenodo.6853125.
  14. Donn Randy Garrison. E-learning in the 21st century: A community of inquiry framework for research and practice. Third Edition, October 2016. URL: https://doi.org/10.4324/9781315667263.
  15. Ashok Goel and David Joyner. Formative Assessment and Implicit Feedback in Online Learning. In Proceedings of Learning with MOOCs III, Philadelphia, PA, 2016. URL: https://www.davidjoyner.net/blog/formative-assessment-and-implicit-feedback-in-online-learning/.
  16. Rubén González. ICE: An Automated Tool for Teaching Advanced C Programming. International Association for Development of the Information Society, 2017. Google Scholar
  17. Peadar F. Grant. Formative test-driven development for programming practicals. AISHE-J: The All Ireland Journal of Teaching and Learning in Higher Education, 9, 2017. Google Scholar
  18. Beate Grawemeyer, John Halloran, Matthew England, and David Croft. Feedback and Engagement on an Introductory Programming Module. CoRR, abs/2201.01240, 2022. URL: https://arxiv.org/abs/2201.01240.
  19. Shuchi Grover. Toward A Framework for Formative Assessment of Conceptual Learning in K-12 Computer Science Classrooms. In Association for Computing Machinery, SIGCSE '21, pages 31-37, New York, NY, USA, 2021. URL: https://doi.org/10.1145/3408877.3432460.
  20. Sugandha Gupta and Anamika Gupta. E-Assessment Tools for Programming Languages: A Review. In International Conference on Information Technology and Knowledge Management, 2018. Google Scholar
  21. Thomas Hainey, Gavin Baxter, Julie Black, Kenneth Yorke, Julius Bernikas, Natalia Chrzanowska, and Fraser McAulay. Serious games as innovative formative assessment tools for programming in higher education. In ECGBL, 16th European Conference on Games Based Learning, 6 - 7 October 2022, Lisbon, Portugal, June 2022. URL: https://www.academic-conferences.org/conferences/ecgbl/.
  22. Jared Howland, Thomas Wright, Rebecca Boughan, and Brian Roberts. How Scholarly Is Google Scholar? A Comparison to Library Databases. College & Research Libraries, 70:227-234, May 2009. URL: https://doi.org/10.5860/crl.70.3.227.
  23. Julian Jansen, Ana Oprescu, and Magiel Bruntink. The impact of automated code quality feedback in programming education. In Post-proceedings of the Tenth Seminar on Advanced Techniques and Tools for Software Evolution (SATToSE), volume 210, 2017. Google Scholar
  24. Maria Kallia. Assessment in Computer Science courses: A Literature Review. In King’s College London, 2017. Google Scholar
  25. Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. A Systematic Literature Review of Automated Feedback Generation for Programming Exercises. ACM Transactions on Computing Education, 19:1-43, September 2018. URL: https://doi.org/10.1145/3231711.
  26. Barbara Kitchenham and Stuart Charters. Guidelines for performing Systematic Literature Reviews in Software Engineering. Keele University and University of Durham technical report, 2, January 2007. Google Scholar
  27. Melisa Koorsse, Charmain Cilliers, and André P. Calitz. Programming assistance tools to support the learning of IT programming in South African secondary schools. Comput. Educ., 82:162-178, 2015. Google Scholar
  28. Nguyen-Thinh Le. A Classification of Adaptive Feedback in Educational Systems for Programming. Systems, 4(2), 2016. URL: https://doi.org/10.3390/systems4020022.
  29. Alex Lishinski and Aman Yadav. Self-evaluation Interventions: Impact on Self-efficacy and Performance in Introductory Programming. ACM Transactions on Computing Education (TOCE), 21:1-28, 2021. Google Scholar
  30. Dastyni Loksa, Lauren Margulieux, Brett A. Becker, Michelle Craig, Paul Denny, Raymond Pettit, and James Prather. Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use. ACM Trans. Comput. Educ., 22(4), September 2022. URL: https://doi.org/10.1145/3487050.
  31. Jose Manappattukunnel Lukose and Kuttickattu John Mammen. Enhancing Academic Achievement in an Introductory Computer Programming Course through the Implementation of Guided Inquiry-Based Learning and Teaching. In Asia-Pacific Forum on Science Learning and Teaching, volume 19,2, 2018. Google Scholar
  32. Andrew Luxton-Reilly, Brett A. Becker, Yingjun Cao, Roger McDermott, Claudio Mirolo, Andreas Mühling, Andrew Petersen, Kate Sanders, Simon, and Jacqueline Whalley. Developing Assessments to Determine Mastery of Programming Fundamentals. In Association for Computing Machinery, ITiCSE-WGR '17, pages 47-69, New York, NY, USA, 2018. URL: https://doi.org/10.1145/3174781.3174784.
  33. Daniel M. Maggin, Erin Barton, Brian Reichow, Kathleen Lynne Lane, and Karrie A. Shogren. Commentary on the What Works Clearinghouse Standards and Procedures Handbook (v. 4.1) for the Review of Single-Case Research. Remedial and Special Education, 43(6):421-433, 2022. URL: https://doi.org/10.1177/07419325211051317.
  34. Sohail Iqbal Malik and Jo Coldwell-Neilson. A model for teaching an introductory programming course using ADRI. Education and Information Technologies, 22:1089-1120, 2017. Google Scholar
  35. Marina Marchisio, Tiziana Margaria, and Matteo Sacchet. Automatic Formative Assessment in Computer Science: Guidance to Model-Driven Design. In 2020 IEEE 44th Annual Computers, Software, and Applications Conference (COMPSAC), pages 201-206, 2020. Google Scholar
  36. Lauren E. Margulieux, Briana B. Morrison, and Adrienne Decker. Reducing withdrawal and failure rates in introductory programming with subgoal labeled worked examples. International Journal of STEM Education, 7:1-16, 2020. Google Scholar
  37. Timothy Meline. Selecting Studies for Systemic Review: Inclusion and Exclusion Criteria. Contemporary Issues in Communication Science and Disorders, 33(Spring):21-27, 2006. URL: https://doi.org/10.1044/cicsd_33_S_21.
  38. Laura Orozco-Garcia, Carolina Gonzalez, Juan Montano, Cristian Mondragon, and Hendrys Tobar-Munoz. A Formative Assessment Tool to Support Computational Thinking in the Classroom. In 2019 International Conference on Virtual Reality and Visualization (ICVRV), pages 185-188, 2019. URL: https://doi.org/10.1109/ICVRV47840.2019.00043.
  39. Elena C. Papanastasiou. Adaptive Assessment, pages 18-19. Springer Verlag, 2015. Google Scholar
  40. Raymond Pettit, J.D. Homer, K.M. Holcomb, N. Simone, and Susan Mengel. Are automated assessment tools helpful in programming courses? ASEE Annual Conference and Exposition, Conference Proceedings, 122, January 2015. Google Scholar
  41. James E. Prather. Beyond Automated Assessment: Building Metacognitive Awareness in Novice Programmers in CS1. In Nova Southeastern University, 2018. Google Scholar
  42. Yizhou Qian and James Lehman. Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review. ACM Trans. Comput. Educ., 18(1), October 2017. URL: https://doi.org/10.1145/3077618.
  43. Yizhou Qian and James Lehman. Using an automated assessment tool to explore difficulties of middle school students in introductory programming. Journal of Research on Technology in Education, 54:1-17, January 2021. URL: https://doi.org/10.1080/15391523.2020.1865220.
  44. Brian J. Reiser and Iris Tabak. Scaffolding, pages 44-62. Cambridge University Press, United Kingdom, January 2014. URL: https://doi.org/10.1017/CBO9781139519526.005.
  45. Emma Riese and Stefan Stenbom. Experiences of Assessment in Introductory Programming From the Perspective of NonComputer Science Majors. In 2020 IEEE Frontiers in Education Conference (FIE), pages 1-9. IEEE Press, 2020. URL: https://doi.org/10.1109/FIE44824.2020.9274060.
  46. Siti Nurulain Mohd Rum and Maizatul Akmar Binti Ismail. Metacognitive Support Accelerates Computer Assisted Learning for Novice Programmers. J. Educ. Technol. Soc., 20:170-181, 2017. Google Scholar
  47. G. W. Scott. Active engagement with assessment and feedback can improve group-work outcomes and boost student confidence. Higher Education Pedagogies, 2(1):1-13, 2017. URL: https://doi.org/10.1080/23752696.2017.1307692.
  48. Nicole Shanley, Florence Martin, Nicole Collins, Manuel Perez-Quinones, Lynn Ahlgrim-Delzell, David Pugalee, and Ellen Hart. Teaching Programming Online: Design, Facilitation and Assessment Strategies and Recommendations for High School Teachers. TechTrends, 66, April 2022. URL: https://doi.org/10.1007/s11528-022-00724-x.
  49. Sadia Sharmin, Daniel Zingaro, Lisa Zhang, and Clare Brett. Impact of Open-Ended Assignments on Student Self-Efficacy in CS1. In CompEd '19: Proceedings of the ACM Conference on Global Computing Education, pages 215-221, April 2019. URL: https://doi.org/10.1145/3300115.3309532.
  50. Valerie J. Shute. Focus on Formative Feedback. Review of Educational Research, 78(1):153-189, 2008. URL: https://doi.org/10.3102/0034654307313795.
  51. Judith Stanja, Wolfgang Gritz, Johannes Krugel, Anett Hoppe, and Sarah Dannemann. Formative assessment strategies for students' conceptions—The potential of learning analytics. British Journal of Educational Technology, 54(1):58-75, 2023. URL: https://doi.org/10.1111/bjet.13288.
  52. Ricarda Steinmayr, Anne F. Weidinger, Malte Schwinger, and Birgit Spinath. The Importance of Students’ Motivation for Their Academic Achievement – Replicating and Extending Previous Findings. Frontiers in Psychology, 10, 2019. URL: https://doi.org/10.3389/fpsyg.2019.01730.
  53. Qing Sun, Ji Wu, Wenge Rong, and Wenbo Liu. Formative assessment of programming language learning based on peer code review: Implementation and experience report. Tsinghua Science and Technology, 24:423-434, August 2019. URL: https://doi.org/10.26599/TST.2018.9010109.
  54. Elise Trumbull and Andrea A. Lash. Understanding Formative Assessment Insights from Learning Theory and Measurement Theory. In WestEd, 2013. Google Scholar
  55. Fabienne M. van der Kleij, Theodorus Johannes Hendrikus Maria Eggen, Caroline F. Timmers, and Bernard P. Veldkamp. Effects of feedback in a computer-based assessment for learning. Comput. Educ., 58:263-272, 2012. Google Scholar
  56. Jill-Jênn Vie, Fabrice Popineau, Éric Bruillard, and Yolaine Bourda. A Review of Recent Advances in Adaptive Assessment, volume 94, pages 113-142. Studies in Systems, Decision and Control, February 2017. URL: https://doi.org/10.1007/978-3-319-52977-6_4.
  57. Xiao-Ming Wang, Gwo-Jen Hwang, Zi-Yun Liang, and Hsiu-Ying Wang. Enhancing students' computer programming performances, critical thinking awareness and attitudes towards programming: An online peer assessment attempt. Educational Technology and Society, 20:58-68, January 2017. Google Scholar
  58. Stelios Xinogalos, Tomáš Pitner, Miloš Savić, and Mirjana Ivanović. First Programming Language in Introductory Programming Courses, Role of, pages 1-11. Springer International Publishing, Cham, 2019. URL: https://doi.org/10.1007/978-3-319-60013-0_217-1.
  59. Albert Yang, Brendan Flanagan, and Hiroaki Ogata. Adaptive formative assessment system based on computerized adaptive testing and the learning memory cycle for personalized learning. Computers and Education: Artificial Intelligence, 3:100104, October 2022. URL: https://doi.org/10.1016/j.caeai.2022.100104.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact