Document Open Access Logo

On the Effectiveness of Interpreter-Guided Compiler Testing

Authors Federico Lochbaum , Guillermo Polito

PDF HTML 5

Files

Thumbnail PDF
PDF
OASIcs.Programming.2025.20.pdf
  • Filesize: 0.67 MB
  • 15 pages
HTML5 Logo HTML (experimental)
OASIcs.Programming.2025.20.html

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Compilers
  • Software and its engineering → Software testing and debugging
Keywords
  • Virtual Machines
  • Concolic Testing
  • JIT compilers
  • interpreters
  • Differential Testing
  • Constraint Mutations
  • Compiler Coverage

Metrics

Abstract

Guaranteeing that a compiler behaves correctly is a complex task often approached through test generation and fuzzing. Compiler test generation must not only ensure that a compiler generates code that does not break, but also that it implements the programming language semantics. Recently, interpreter-guided test generation has been proposed to test JIT compilers: Concolic-execution on the interpreter yields test cases for the language semantics which are then validated between differential testing of the interpreter and compiler. In previous work, this solution has been shown to find interpreter/compiler differences. However, little has been said about the effectiveness and the solution limits.
In this paper we study the behavior of this technique, to shed light on future improvements and research. We experiment with this technique on the JIT compiler for the Pharo programming language, on two different backends: ARMv7 and x86. We explore how effective the solution is in terms of compiler coverage and its limitations, and we discuss how future research can overcome them. Moreover, we investigate how this technique combined with random constraint mutations increases backend compiler coverage.

Cite As Get BibTex

Federico Lochbaum and Guillermo Polito. On the Effectiveness of Interpreter-Guided Compiler Testing. In Companion Proceedings of the 9th International Conference on the Art, Science, and Engineering of Programming (Programming 2025). Open Access Series in Informatics (OASIcs), Volume 134, pp. 20:1-20:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.Programming.2025.20

Author Details

Federico Lochbaum
  • Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France
Guillermo Polito
  • Univ. Lille, Inria, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, France

Funding

This work was funded by ANR JCJC Sapper.

References

  1. Aurèle Barrière, Sandrine Blazy, Olivier Flückiger, David Pichardie, and Jan Vitek. Formally verified speculation and deoptimization in a jit compiler. In Conference on Principles of Programming Languages (POPL), January 2021. URL: https://doi.org/10.1145/3434327.
  2. Clément Béra, Eliot Miranda, Marcus Denker, and Stéphane Ducasse. Practical validation of bytecode to bytecode jit compiler dynamic deoptimization. Journal of Object Technology, 15(2):1:1-26, 2016. https://doi.org/10.5381/jot.2016.15.2.a1.
  3. Marcel Böhme, László Szekeres, and Jonathan Metzman. On the reliability of coverage-based fuzzer benchmarking. In Proceedings of the 44th International Conference on Software Engineering, ICSE '22, pages 1621-1633, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3510003.3510230.
  4. Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. An empirical comparison of compiler testing techniques. In Proceedings of the 38th International Conference on Software Engineering, ICSE '16, pages 180-190, New York, NY, USA, 2016. Association for Computing Machinery. URL: https://doi.org/10.1145/2884781.2884878.
  5. Junjie Chen, Jibesh Patra, Michael Pradel, Yingfei Xiong, Hongyu Zhang, Dan Hao, and Lu Zhang. A Survey of Compiler Testing. ACM Computing Surveys, pages 1-36, May 2020. URL: https://dl.acm.org/doi/10.1145/3363562.
  6. Yuting Chen, Ting Su, and Zhendong Su. Deep Differential Testing of JVM Implementations. In International Conference on Software Engineering (ICSE'19), pages 1257-1268. IEEE Press, 2019. URL: https://doi.org/10.1109/ICSE.2019.00127.
  7. Yuting Chen, Ting Su, Chengnian Sun, Zhendong Su, and Jianjun Zhao. Coverage-directed differential testing of JVM implementations. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 85-99. Association for Computing Machinery, June 2016. URL: https://doi.org/10.1145/2908080.2908095.
  8. Peter Dinges and Gul Agha. Targeted test input generation using symbolic-concrete backward execution. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE '14, pages 31-36, New York, NY, USA, 2014. Association for Computing Machinery. URL: https://doi.org/10.1145/2642937.2642951.
  9. Karine Even-Mendoza, Arindam Sharma, Alastair F. Donaldson, and Cristian Cadar. Grayc: Greybox fuzzing of compilers and analysers for c. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, pages 1219-1231, New York, NY, USA, 2023. Association for Computing Machinery. URL: https://doi.org/10.1145/3597926.3598130.
  10. Olivier Flückiger, Gabriel Scherer, Ming-Ho Yee, Aviral Goel, Amal Ahmed, and Jan Vitek. Correctness of speculative optimizations with dynamic deoptimization. In Principles of programming languages (POPL'17), 2017. URL: https://doi.org/10.1145/3158137.
  11. Christian Holler, Kim Herzig, and Andreas Zeller. Fuzzing with code fragments. In 21st USENIX Security Symposium (USENIX Security 12), pages 445-458, Bellevue, WA, August 2012. USENIX Association. URL: https://www.usenix.org/conference/usenixsecurity12/technical-sessions/presentation/holler.
  12. Urs Hölzle, Craig Chambers, and David Ungar. Optimizing dynamically-typed object-oriented languages with polymorphic inline caches. In European Conference on Object-Oriented Programming (ECOOP'91), 1991. URL: https://doi.org/10.1007/BFb0057013.
  13. Sungjae Hwang, Sungho Lee, Jihoon Kim, and Sukyoung Ryu. JUSTGen: Effective Test Generation for Unspecified JNI Behaviors on JVMs. In International Conference on Software Engineering (ICSE'21), pages 1708-1718, 2021. URL: https://doi.org/10.1109/ICSE43902.2021.00151.
  14. Dan Ingalls, Ted Kaehler, John Maloney, Scott Wallace, and Alan Kay. Back to the future: The story of Squeak, a practical Smalltalk written in itself. In Proceedings of Object-Oriented Programming, Systems, Languages, and Applications conference (OOPSLA'97), pages 318-326. ACM Press, November 1997. URL: https://doi.org/10.1145/263700.263754.
  15. Yunho Kim, Youil Kim, Taeksu Kim, Gunwoo Lee, Yoonkyu Jang, and Moonzoo Kim. Automated unit testing of large industrial embedded software using concolic testing. In 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 519-528, 2013. URL: https://doi.org/10.1109/ASE.2013.6693109.
  16. Thijs Klooster, Fatih Turkmen, Gerben Broenink, Ruben Ten Hove, and Marcel Böhme. Continuous fuzzing: A study of the effectiveness and scalability of fuzzing in ci/cd pipelines. In 2023 IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT), pages 25-32, May 2023. URL: https://doi.org/10.1109/SBFT59156.2023.00015.
  17. Bastian Kruck, Stefan Lehmann, Christoph Keßler, Jakob Reschke, Tim Felgentreff, Jens Lincke, and Robert Hirschfeld. Multi-level debugging for interpreter developers. In Companion to Proceedings of the international conference on Modularity, pages 91-93. ACM, 2016. URL: https://doi.org/10.1145/2892664.2892679.
  18. Vu Le, Mehrdad Afshari, and Zhendong Su. Compiler validation via equivalence modulo inputs. In Conference on Programming Language Design and Implementation (PLDI'14), June 2014. URL: https://dl.acm.org/doi/10.1145/2594291.2594334.
  19. Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. Many-core compiler fuzzing. In Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '15, pages 65-76, New York, NY, USA, 2015. Association for Computing Machinery. URL: https://doi.org/10.1145/2737924.2737986.
  20. Rupak Majumdar and Koushik Sen. Hybrid concolic testing. In 29th International Conference on Software Engineering (ICSE'07), pages 416-426, 2007. URL: https://doi.org/10.1109/ICSE.2007.41.
  21. William M McKeeman. Differential Testing for Software. DIGITAL TECHNICAL JOURNAL, 1998. Google Scholar
  22. Eliot Miranda. The cog smalltalk virtual machine. In Proceedings of VMIL 2011, 2011. Google Scholar
  23. Eliot Miranda, Clément Béra, Elisa Gonzalez Boix, and Dan Ingalls. Two decades of Smalltalk VM development: live VM development through simulation tools. In Proceedings of International Workshop on Virtual Machines and Intermediate Languages (VMIL'18), pages 57-66. ACM, 2018. URL: https://doi.org/10.1145/3281287.3281295.
  24. Jihyeok Park, Seungmin An, Dongjun Youn, Gyeongwon Kim, and Sukyoung Ryu. JEST: N+1-Version Differential Testing of Both JavaScript Engines and Specification. In International Conference on Software Engineering (ICSE'21), pages 13-24. IEEE Press, 2021. URL: https://doi.org/10.1109/ICSE43902.2021.00015.
  25. Guillermo Polito, Pablo Tesone, and Stéphane Ducasse. Interpreter-guided Differential JIT Compiler Unit Testing. In Programming Language Design and Implementation (PLDI'22), 2022. Google Scholar
  26. Paul Purdom. A sentence generator for testing parsers. BIT Numerical Mathematics, 12(3):366-375, 1972. URL: https://doi.org/10.1007/BF01932308.
  27. Xiao Qu and Brian Robinson. A case study of concolic testing tools and their limitations. In 2011 International Symposium on Empirical Software Engineering and Measurement, pages 117-126, 2011. URL: https://doi.org/10.1109/ESEM.2011.20.
  28. Chengnian Sun, Vu Le, Qirun Zhang, and Zhendong Su. Toward understanding compiler bugs in gcc and llvm. In Proceedings of the 25th International Symposium on Software Testing and Analysis, ISSTA 2016, pages 294-305, New York, NY, USA, 2016. Association for Computing Machinery. URL: https://doi.org/10.1145/2931037.2931074.
  29. Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. An automatic testing approach for compiler based on metamorphic testing technique. In 2010 Asia Pacific Software Engineering Conference, pages 270-279, 2010. URL: https://doi.org/10.1109/APSEC.2010.39.
  30. David Ungar, Adam Spitz, and Alex Ausch. Constructing a metacircular Virtual machine in an exploratory programming environment. In Companion to Object-Oriented Programming, Systems, Languages, and Applications conference (OOPSLA'05), pages 11-20, New York, NY, USA, 2005. ACM. URL: https://doi.org/10.1145/1094855.1094865.
  31. Christian Wimmer, Michael Haupt, Michael L. Van De Vanter, Mick Jordan, Laurent Daynès, and Douglas Simon. Maxine: An approachable virtual machine for, and in, java. ACM Transaction Architecture Code Optimization, 9(4), January 2013. URL: https://doi.org/10.1145/2400682.2400689.
  32. Dylan Wolff, Marcel Böhme, and Abhik Roychoudhury. Explainable fuzzer evaluation, 2022. URL: https://doi.org/10.48550/arXiv.2212.09519.
  33. Thomas Würthinger, Michael L. Van De Vanter, and Doug Simon. Multi-level virtual machine debugging using the java platform debugger architecture. In Perspectives of Systems Informatics, pages 401-412. Springer, 2010. URL: https://doi.org/10.1007/978-3-642-11486-1_34.
  34. Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and Understanding Bugs in C Compilers. In Programming Language Design and Implementation, PLDI '11, 2011. URL: https://doi.org/10.1145/1993498.1993532.
  35. Guixin Ye, Zhanyong Tang, Shin Hwei Tan, Songfang Huang, Dingyi Fang, Xiaoyang Sun, Lizhong Bian, Haibo Wang, and Zheng Wang. Automated conformance testing for javascript engines via deep compiler fuzzing. In Proceedings of Conference on Programming Language Design and Implementation (PLDI'21), pages 435-450, New York, NY, USA, 2021. ACM. URL: https://doi.org/10.1145/3453483.3454054.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact