Document Open Access Logo

A Domain-Specific Language for Dynamic White-Box Evaluation of Java Assignments

Authors Afonso B. Caniço , André L. Santos

PDF
Thumbnail PDF

File

OASIcs.ICPEC.2024.2.pdf
  • Filesize: 0.58 MB
  • 13 pages

Document Identifiers

Author Details

Afonso B. Caniço
  • Instituto Universitário de Lisboa (ISCTE-IUL), Portugal
André L. Santos
  • Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR-IUL, Portugal

Cite AsGet BibTex

Afonso B. Caniço and André L. Santos. A Domain-Specific Language for Dynamic White-Box Evaluation of Java Assignments. In 5th International Computer Programming Education Conference (ICPEC 2024). Open Access Series in Informatics (OASIcs), Volume 122, pp. 2:1-2:13, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024)
https://doi.org/10.4230/OASIcs.ICPEC.2024.2

Abstract

Programming exercises involving algorithms typically involve time and spatial constraints. Automated assessments for such implementations are often carried out in a black-box manner or through static analysis of the code, without considering the internal execution properties, which could lead to falsely positive evaluations of students' solutions. We present Witter, a domain-specific language for defining white-box test cases for the Java language. We evaluated programming assignment submissions from a Data Structures and Algorithms course against Witter’s test cases to determine if our approach could offer additional insight regarding incomplete algorithmic behaviour requirements. We found that a significant amount of student solutions fail to meet the desired algorithmic behavior (approx. 21%), despite passing black-box tests. Hence, we conclude that white-box tests are useful to achieve a thorough automated evaluation of this kind of exercises.

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
  • Software and its engineering → Domain specific languages
  • Social and professional topics → Student assessment
Keywords
  • White-box assessment
  • student assessment
  • programming education

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

References

  1. C.L. Aasheim, Paige Rutner, L. Li, and S.R. Williams. Plagiarism and programming: A survey of student attitudes. Journal of Information Systems Education, 23:297-314, January 2012. Google Scholar
  2. Eric Bruneton, Romain Lenglet, and Thierry Coupaye. Asm: A code manipulation tool to implement adaptable systems. In In Adaptable and extensible component systems, 2002. URL: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.5769.
  3. Afonso B. Caniço and André L. Santos. Witter: A library for white-box testing of introductory programming algorithms. In Proceedings of the 2023 ACM SIGPLAN International Symposium on SPLASH-E, SPLASH-E 2023, pages 69-74, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3622780.3623650.
  4. Shigeru Chiba. Load-time structural reflection in java. In Elisa Bertino, editor, ECOOP 2000 - Object-Oriented Programming, pages 313-336, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg. Google Scholar
  5. Sébastien Combéfis. Automated code assessment for education: Review, classification and perspectives on techniques and tools. Software, 1(1):3-30, 2022. URL: https://doi.org/10.3390/software1010002.
  6. Pedro Crispim, Antónia Lopes, and Vasco T. Vasconcelos. Runtime verification for generic classes with congu2. In Proceedings of the 13th Brazilian Conference on Formal Methods: Foundations and Applications, SBMF'10, pages 33-48, Berlin, Heidelberg, 2010. Springer-Verlag. Google Scholar
  7. Michael T. Helmick. Interface-based programming assignments and automatic grading of java programs. SIGCSE Bull., 39(3):63-67, June 2007. URL: https://doi.org/10.1145/1269900.1268805.
  8. David Insa and Josep Silva. Automatic assessment of java code. Computer Languages, Systems & Structures, 53:59-72, 2018. URL: https://doi.org/10.1016/j.cl.2018.01.004.
  9. Hieke Keuning, Johan Jeuring, and Bastiaan Heeren. A systematic literature review of automated feedback generation for programming exercises. ACM Trans. Comput. Educ., 19(1), September 2018. URL: https://doi.org/10.1145/3231711.
  10. Gregor Kiczales, Erik Hilsdale, Jim Hugunin, Mik Kersten, Jeffrey Palm, and William G. Griswold. An overview of aspectj. In Jørgen Lindskov Knudsen, editor, ECOOP 2001 - Object-Oriented Programming, pages 327-354, Berlin, Heidelberg, 2001. Springer Berlin Heidelberg. Google Scholar
  11. José Paulo Leal and Fernando Silva. Mooshak: a web-based multi-site programming contest system. Software: Practice and Experience, 33(6):567-581, 2003. URL: https://doi.org/10.1002/spe.522.
  12. Marcus Messer, Neil C. C. Brown, Michael Kölling, and Miaojing Shi. Automated grading and feedback tools for programming education: A systematic review. ACM Trans. Comput. Educ., December 2023. Just Accepted. URL: https://doi.org/10.1145/3636515.
  13. Samim Mirhosseini, Austin Z. Henley, and Chris Parnin. What is your biggest pain point? an investigation of cs instructor obstacles, workarounds, and desires. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1, SIGCSE 2023, pages 291-297, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3545945.3569816.
  14. José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. Automated assessment in computer science education: A state-of-the-art review. ACM Trans. Comput. Educ., 22(3), June 2022. URL: https://doi.org/10.1145/3513140.
  15. Timur Sağlam, Moritz Brödel, Larissa Schmid, and Sebastian Hahner. Detecting automatic software plagiarism via token sequence normalization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3597503.3639192.
  16. Timur Sağlam, Sebastian Hahner, Larissa Schmid, and Erik Burger. Obfuscation-resilient software plagiarism detection with jplag. In 46th IEEE/ACM International Conference on Software Engineering: Companion Proceedings, ICSE-Companion. Institute of Electrical and Electronics Engineers (IEEE), 2024. URL: https://doi.org/10.1145/3639478.3643074.
  17. Manuel Sánchez, Päivi Kinnunen, Cristóbal Flores, and J. Ángel Velázquez-Iturbide. Student perception and usage of an automated programming assessment tool. Computers in Human Behavior, 31:453-460, February 2014. URL: https://doi.org/10.1016/j.chb.2013.04.001.
  18. Anne Venables and Liz Haywood. Programming students need instant feedback! In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20, ACE '03, pages 267-272, AUS, 2003. Australian Computer Society, Inc. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH Imprint Privacy Contact