Document Open Access Logo

A Facet-Based Perspective on Code Comprehension Task Design for Empirical Research

Authors Luisa Bartl , Marvin Wyrich , Sven Apel , Janet Siegmund

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.52.pdf
  • Filesize: 0.57 MB
  • 15 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • General and reference → Measurement
Keywords
  • Code comprehension
  • Construct validity
  • Empirical software engineering

Metrics

Abstract

Code comprehension is an inevitable part of the software development process and a frequent target of empirical research. However, the field lacks a shared definition of the construct. Studies operationalize code comprehension in diverse ways, often without explicitly stating what aspect of understanding they measure. As a result, researchers design study-specific tasks and measures, limiting comparability across studies and hindering cumulative knowledge building.
To initiate a coordinated approach to assess code comprehension, we introduce a facet-based perspective that distinguishes between analytical skills, abstraction, and critical evaluation. We demonstrate the benefit of explicitly stating the targeted facet(s) of code comprehension and aligning task design accordingly. Building on this perspective, we propose a method to guide researchers from research goals to concrete tasks and study design, including considerations for question formulation and response formats. Rather than immediately prescribing a single standardized instrument, our approach takes a first step toward this long-term goal by enabling researchers to design adaptable yet conceptually grounded measurements. By making operationalizations explicit and tasks reusable, we initiate a cumulative process through which the community can iteratively converge on standardized instruments and, ultimately, a shared understanding of code comprehension.

Cite As Get BibTex

Luisa Bartl, Marvin Wyrich, Sven Apel, and Janet Siegmund. A Facet-Based Perspective on Code Comprehension Task Design for Empirical Research. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 52:1-52:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.52

Author Details

Luisa Bartl
  • Chemnitz University of Technology, Germany
Marvin Wyrich
  • Saarland University, Saarbrücken, Germany
Sven Apel
  • Saarland University, Saarbrücken, Germany
Janet Siegmund
  • Chemnitz University of Technology, Germany

Funding

This work has been supported by the European Union under ERC Advanced Grant "Brains On Code" (101052182).

References

  1. Marwen Abbes, Foutse Khomh, Yann-Gaël Guéhéneuc, and Giuliano Antoniol. An Empirical Study of the Impact of Two Antipatterns, Blob and Spaghetti Code, on Program Comprehension. In Europ. Conf. Software Maintenance and Reengineering (CSMR), pages 181-190. IEEE CS, 2011. URL: https://doi.org/10.1109/CSMR.2011.24.
  2. Nahla J. Abid, Bonita Sharif, Natalia Dragan, Hend Alrasheed, and Jonathan I. Maletic. Developer reading behavior while summarizing Java methods: size and context matters. In Proc. Int. Conf. Software Engineering (ICSE), ICSE '19, pages 384-395. IEEE CS, 2019. URL: https://doi.org/10.1109/ICSE.2019.00052.
  3. Shraddha Barke, Michael B James, and Nadia Polikarpova. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proc. ACM Program. Lang., 7(OOPSLA1):85-111, 2023. URL: https://doi.org/10.1145/3586030.
  4. Victor R. Basili and Harlan D. Mills. Understanding and Documenting Programs. IEEE Transactions on Software Engineering, SE-8(3):270-283, 1982. URL: https://doi.org/10.1109/TSE.1982.235255.
  5. Gal Beniamini, Sarah Gingichashvili, Alon Klein Orbach, and Dror G. Feitelson. Meaningful Identifier Names: The Case of Single-Letter Variables. In Proc. Int. Conf. Program Comprehension (ICPC), pages 45-54, 2017. URL: https://doi.org/10.1109/ICPC.2017.18.
  6. Percy Bridgman. The Logic of Modern Physics. The Macmillan Company, 1927. Google Scholar
  7. Ruven Brooks. Towards a Theory of the Comprehension of Computer Programs. International Journal of Man-Machine Studies, 18(6):543-554, 1983. URL: https://doi.org/10.1016/S0020-7373(83)80031-5.
  8. J. Cronbach and P. E. Meehl. Construct Validity in Psychological Tests. Psychological Bulletin, 52(4):281-302, 1955. Google Scholar
  9. Sarah Fakhoury, Devjeet Roy, Yuzhan Ma, Venera Arnaoudova, and Olusola Adesope. Measuring the impact of lexical and structural inconsistencies on developers’ cognitive load during bug localization. Empirical Softw. Eng., 25(3):2140-2178, 2020. URL: https://doi.org/10.1007/S10664-019-09751-4.
  10. Janet Feigenspan, Christian Kästner, Sven Apel, Jörg Liebig, Michael Schulze, Raimund Dachselt, Maria Papendieck, Thomas Leich, and Gunter Saake. Do Background Colors Improve Program Comprehension in the #ifdef Hell? Empirical Softw. Eng., 18(4):699-745, 2013. URL: https://doi.org/10.1007/S10664-012-9208-X.
  11. Dror G Feitelson. Considerations and pitfalls for reducing threats to the validity of controlled experiments on code comprehension. Empirical Software Engineering, 27(6):123, 2022. URL: https://doi.org/10.1007/S10664-022-10160-3.
  12. David J. Gilmore. Models of Debugging. Acta Psychologica, 78(1):151-172, 1991. Google Scholar
  13. Pavlína Wurzel Gonçalves, Pooja Rani, Margaret-Anne Storey, Diomidis Spinellis, and Alberto Bacchelli. Code Review Comprehension: Reviewing Strategies Seen Through Code Comprehension Theories. In Proc. Int. Conf. Program Comprehension (ICPC), pages 589-601. IEEE CS, 2025. URL: https://doi.org/10.1109/ICPC66645.2025.00068.
  14. Dan Gopstein, Jake Iannacone, Yu Yan, Lois DeLong, Yanyan Zhuang, Martin K.-C. Yeh, and Justin Cappos. Understanding misunderstandings in source code. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, pages 129-139, New York, NY, USA, 2017. Association for Computing Machinery. URL: https://doi.org/10.1145/3106237.3106264.
  15. Daniel Graziotin, Per Lenberg, Robert Feldt, and Stefan Wagner. Psychometrics in behavioral software engineering: A methodological introduction with guidelines. ACM Transactions on Software Engineering and Methodology (TOSEM), 31(1):1-36, 2021. URL: https://doi.org/10.1145/3469888.
  16. Latifa Guerrouj, Massimiliano Di Penta, Yann-Gaël Guéhéneuc, and Giuliano Antoniol. An experimental investigation on the effects of context on source code identifiers splitting and expansion. Empirical Softw. Eng., 19(6):1706-1753, 2014. URL: https://doi.org/10.1007/S10664-013-9260-1.
  17. L. Gugerty and G. Olson. Debugging by Skilled and Novice Programmers. In Conf. Human Factors in Computing Systems (CHI), CHI '86, pages 171-174, New York, NY, USA, 1986. ACM Press. Google Scholar
  18. Harold Gulliksen. Theory of Mental Tests. John Wiley & Sons, 1 edition, 1950. Google Scholar
  19. Mohammed Hassan, Kathryn Cunningham, and Craig Zilles. Evaluating Beacons, the Role of Variables, Tracing, and Abstract Tracing for Teaching Novices to Understand Program Intent. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1, pages 329-343. ACM Press, 2023. URL: https://doi.org/10.1145/3568813.3600140.
  20. Felienne Hermans and Efthimia Aivaloglou. Do code smells hamper novice programming? A controlled experiment on Scratch programs. In Proc. Int. Conf. Program Comprehension (ICPC), pages 1-10. IEEE CS, 2016. URL: https://doi.org/10.1109/ICPC.2016.7503706.
  21. Sora Chi-Fang Huang, Zhi-Hong Chen, Yu-Tzu Lin, Martin K. Yeh, and Yi-Wei Chen. Exploring the Link Between Problem-Solving Strategies and Programming Performance: A Comparative Analysis of High and Low Performers. Journal of Educational Computing Research, 64(2):493-523, 2026. Google Scholar
  22. Kang il Park, Jack Johnson, Cole S. Peterson, Nishitha Yedla, Isaac Baysinger, Jairo Aponte, and Bonita Sharif. An Eye Tracking Study Assessing Source Code Readability Rules for Program Comprehension. Empirical Softw. Eng., 29(160), 2024. URL: https://doi.org/10.1007/S10664-024-10532-X.
  23. Cruz Izu and Claudio Mirolo. Comparing Small Programs for Equivalence: A Code Comprehension Task for Novice Programmers. In Annual Conf. Innovation and Technology in Computer Science Education (ITiCSE), ITiCSE '20, pages 466-472, New York, NY, USA, 2020. Association for Computing Machinery. URL: https://doi.org/10.1145/3341525.3387425.
  24. Maria Kallia. The Search for Meaning: Inferential Strategic Reading Comprehension in Programming. In Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 1, pages 1-14. ACM Press, 2023. URL: https://doi.org/10.1145/3568813.3600135.
  25. Martin Konopka, Adam Talian, Jozef Tvarozek, and Pavol Navrat. Data flow metrics in program comprehension tasks. In Proc. Int. Workshop on Eye Movements in Programming, EMIP '18. ACM Press, 2018. Google Scholar
  26. Makrina Viola Kosti, Kostas Georgiadis, Dimitrios A. Adamos, Nikos Laskaris, Diomidis Spinellis, and Lefteris Angelis. Towards an affordable brain computer interface for the assessment of programmers’ mental workload. International Journal of Human-Computer Studies, 115:52-66, 2018. URL: https://doi.org/10.1016/J.IJHCS.2018.03.002.
  27. Helfried Moosbrugger and Augustin Kelava. Testtheorie und Fragebogenkonstruktion. Springer eBook Collection. Springer, Berlin, Heidelberg, 3 edition, 2020. Google Scholar
  28. Sebastian Nielebock, Dariusz Krolikowski, Jacob Krüger, Thomas Leich, and Frank Ortmeier. Commenting source code: is it worth it for small programming tasks? Empirical Softw. Eng., 24(3):1418-1457, 2019. URL: https://doi.org/10.1007/S10664-018-9664-Z.
  29. A. F. Norcio. Indentation, documentation and programmer comprehension. In Conf. Human Factors in Computing Systems (CHI), CHI '82, pages 118-120, New York, NY, USA, 1982. Association for Computing Machinery. Google Scholar
  30. Delano Oliveira, Reydne Bruno, Fernanda Madeiral, and Fernando Castor. Evaluating Code Readability and Legibility: An Examination of Human-centric Studies. In Int. Conf. Software Maintenance & Evolution (ICSME), pages 348-359, 2020. URL: https://doi.org/10.1109/ICSME46990.2020.00041.
  31. Norman Peitek, Sven Apel, Chris Parnin, André Brechmann, and Janet Siegmund. Program Comprehension and Code Complexity Metrics: An fMRI Study. In Proc. Int. Conf. Software Engineering (ICSE), pages 524-536. IEEE CS, 2021. URL: https://doi.org/10.1109/ICSE43902.2021.00056.
  32. Norman Peitek, Janet Siegmund, and Sven Apel. What Drives the Reading Order of Programmers? An Eye Tracking Study. In Proceedings of the International Conference on Program Comprehension (ICPC), pages 342-353. ACM, 2020. URL: https://doi.org/10.1145/3387904.3389279.
  33. Norman Peitek, Janet Siegmund, Chris Parnin, Sven Apel, Johannes C. Hofmeister, and André Brechmann. Simultaneous Measurement of Program Comprehension with fMRI and Eye Tracking: A Case Study. In Int. Symp. Empirical Software Engineering and Measurement (ESEM). ACM Press, 2018. Google Scholar
  34. Nancy Pennington. Stimulus Structures and Mental Representations in Expert Comprehension of Computer Programs. Cognitive Psychology, 19(3):295-341, 1987. Google Scholar
  35. Cristiano Politowski, Foutse Khomh, Simone Romano, Giuseppe Scanniello, Fabio Petrillo, Yann-Gaël Guéhéneuc, and Abdou Maiga. A large scale empirical study of the impact of Spaghetti Code and Blob anti-patterns on program comprehension. Information and Software Technology, 122:106278, 2020. URL: https://doi.org/10.1016/J.INFSOF.2020.106278.
  36. Paul Ralph and Ewan Tempero. Construct Validity in Software Engineering Research and Software Metrics. In Int. Conf. Evaluation and Assessment in Software Engineering (EASE), EASE '18, pages 13-23, New York, NY, USA, 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3210459.3210461.
  37. Guido Salvaneschi, Sven Amann, Sebastian Proksch, and Mira Mezini. An empirical study on program comprehension with reactive programming. In Proc. ACM SIGSOFT Int. Symp. Foundations of Software Engineering (FSE), FSE 2014, pages 564-575, New York, NY, USA, 2014. Association for Computing Machinery. URL: https://doi.org/10.1145/2635868.2635895.
  38. Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-Vásquez, Denys Poshyvanyk, and Rocco Oliveto. Automatically Assessing Code Understandability. IEEE Transactions on Software Engineering, 47(3):595-613, 2021. URL: https://doi.org/10.1109/TSE.2019.2901468.
  39. Todd Sedano. Code Readability Testing, an Empirical Study. In Proc. IEEE Int. Conf. Software Engineering Education and Training (CSEET), pages 111-117, 2016. URL: https://doi.org/10.1109/CSEET.2016.36.
  40. Matheus Segalotto, Willian Bolzan, and Kleinner Farias. Effects of modularization on developers' cognitive effort in code comprehension tasks: A controlled experiment. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering, SBES '23, pages 206-215, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3613372.3613387.
  41. Teresa Shaft and Iris Vessey. The Relevance of Application Domain Knowledge: The Case of Computer Program Comprehension. Information Systems Research, 6(3):286-299, 1995. Google Scholar
  42. Anshul Shah, Thanh Tong, Elena Tomson, Steven Shi, William G. Griswold, and Adalbert Gerald Soosai Raj. Students' Program Comprehension Processes in a Large Code Base. In Proc. Int. Conf. Program Comprehension (ICPC), pages 182-193. IEEE CS, 2025. URL: https://doi.org/10.1109/ICPC66645.2025.00026.
  43. Ben Shneiderman and Richard Mayer. Syntactic/Semantic Interactions in Programmer Behavior: A Model and Experimental Results. International Journal of Computer & Information Science, 8(3):219-238, 1979. URL: https://doi.org/10.1007/BF00977789.
  44. Janet Siegmund, Christian Kästner, Sven Apel, Chris Parnin, Anja Bethmann, Thomas Leich, Gunter Saake, and André Brechmann. Understanding Understanding Source Code with Functional Magnetic Resonance Imaging. In Proc. Int. Conf. Software Engineering (ICSE), pages 378-389. ACM Press, 2014. URL: https://doi.org/10.1145/2568225.2568252.
  45. Janet Siegmund, Norman Peitek, André Brechmann, Chris Parnin, and Sven Apel. Studying Programming in the Neuroage: Just a Crazy Idea? Commu. ACM, 63(6):30-34, 2020. URL: https://doi.org/10.1145/3347093.
  46. Janet Siegmund and Jana Schumann. Confounding Parameters on Program Comprehension: a Literature Survey. Empirical Softw. Eng., 20:1159-1192, 2015. URL: https://doi.org/10.1007/S10664-014-9318-8.
  47. Dag I.K. Sjøberg and Gunnar R. Bergersen. Construct Validity in Software Engineering. IEEE Transactions on Software Engineering, 49(3):1374-1396, 2022. URL: https://doi.org/10.1109/TSE.2022.3176725.
  48. Elliot Soloway and Kate Ehrlich. Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering, SE-10(5):595-609, 1984. URL: https://doi.org/10.1109/TSE.1984.5010283.
  49. Rachel Turner, Michael Falcone, Bonita Sharif, and Alina Lazar. An eye-tracking study assessing the comprehension of c++ and Python source code. In Proc. Symposium Eye Tracking Research and Applications (ETRA), ETRA '14, pages 231-234, New York, NY, USA, 2014. Association for Computing Machinery. URL: https://doi.org/10.1145/2578153.2578218.
  50. Anneliese von Mayrhauser and Marie Vans. Program Comprehension During Software Maintenance and Evolution. Computer, 28(8):44-55, 1995. URL: https://doi.org/10.1109/2.402076.
  51. Daniel Votipka, Seth Rabin, Kristopher Micinski, Jeffrey S Foster, and Michelle L Mazurek. An Observational Investigation of Reverse Engineers’ Processes. In 29th USENIX Security Symposium (USENIX Security 20), pages 1875-1892, 2020. URL: https://www.usenix.org/conference/usenixsecurity20/presentation/votipka-observational.
  52. Stefan Wagner and Marvin Wyrich. Code comprehension confounders: A study of intelligence and personality. IEEE Transactions on Software Engineering, 48(12):4789-4801, 2021. URL: https://doi.org/10.1109/TSE.2021.3127131.
  53. Marvin Wyrich. Source Code Comprehension: A Contemporary Definition and Conceptual Model for Empirical Investigation, 2023. URL: https://doi.org/10.48550/arXiv.2310.11301.
  54. Marvin Wyrich, Justus Bogner, and Stefan Wagner. 40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study. ACM Computing Surveys, 56(4):1-42, 2023. URL: https://doi.org/10.1145/3626522.
  55. Marvin Wyrich, Marvin Munoz Baron, and Justus Bogner. Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent Variables. In Proc. IEEE/ACM Int. Workshop Methodological Issues with Empirical Studies in Software Engineering, pages 42-47, 2024. Google Scholar
  56. Xin Xia, Lingfeng Bao, David Lo, Zhenchang Xing, Ahmed E. Hassan, and Shanping Li. Measuring Program Comprehension: A Large-Scale Field Study with Professionals. IEEE Transactions on Software Engineering, 44(10):951-976, 2018. URL: https://doi.org/10.1109/TSE.2017.2734091.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact