Document Open Access Logo

Evaluating Language Models on Cross-Language Code Functional Equivalence

Authors Hui Sun , Anderson Uchôa , Rohit Gheyi , Wesley K. G. Assunção

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.41.pdf
  • Filesize: 1.24 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software verification and validation
  • General and reference → Empirical studies
Keywords
  • Empirical software engineering
  • code functional equivalence
  • generative AI
  • cross-language analysis
  • benchmark
  • failure analysis

Metrics

Abstract

Background. Large Language Models (LLMs) have demonstrated strong performance across a variety of code-understanding tasks, leading many to believe that they can reason about program semantics. A fundamental test of this capability is assessing code functional equivalence. However, existing evaluations primarily focus on single-language settings or rely on synthetically generated code, raising concerns about whether current results reflect true semantic understanding. 

Aims. We investigate whether LLMs can accurately judge functional equivalence across different programming languages in human-written code, a setting that requires deeper reasoning beyond superficial similarity. 

Method. We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python. Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs, selecting GPT-o4-mini as a representative model to assess stability. We then manually analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence, examining the code logic and the generated Chain-of-Thought reasoning. Finally, we categorize these failures and compare them across GPT-o4-mini, Claude-Opus-4.7, and Gemini-3-Flash to determine whether they reflect model-specific issues or broader limitations of state-of-the-art LLMs. 

Results. We identify a difficulty-dependent breakdown in equivalence judgment (harder problems make the model increasingly prone to misclassifying non-equivalent code as equivalent), a model-specific sensitivity to programming language for the best-performing model (particularly a more conservative behavior on Python), and a partial reliance on similarity-based cues. GPT-o4-mini also shows substantial run-to-run instability under identical settings, indicating inconsistent rather than absent capability. Its failures are mainly due to (1) a lack of knowledge relevant to judging code functional equivalence and (2) abstraction-level reasoning failure, such as premature conclusion, semantic-level misunderstanding, and implementation-level error. We also observed that several analyzed failures involved loops and conditional branches. 

Conclusions. Current LLMs do not reliably capture functional equivalence within or across languages. Our findings highlight the need for more realistic benchmarks and suggest improvements through better reasoning-path selection and by addressing reasoning gaps across the identified failure levels.

Cite As Get BibTex

Hui Sun, Anderson Uchôa, Rohit Gheyi, and Wesley K. G. Assunção. Evaluating Language Models on Cross-Language Code Functional Equivalence. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 41:1-41:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.41

Author Details

Hui Sun
  • North Carolina State University, Raleigh, NC, USA
Anderson Uchôa
  • Federal University of Ceará, Itapajé, Ceará, Brazil
  • North Carolina State University, Raleigh, NC, USA
Rohit Gheyi
  • Federal University of Campina Grande, Campina Grande, Paraíba, Brazil
Wesley K. G. Assunção
  • North Carolina State University, Raleigh, NC, USA

Funding

This research was partially supported by FUNCAP (BP6-00241-00276.01.00/25), CNPq (201708/2025-6, 403304/2025-3, 404406/2023-8, 403719/2024-0), FAPESQ-PB (268/2025), CAPES, and Google Cloud Research Credits Program with the award GCP19980904.

Supplementary Materials

References

  1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Unified pre-training for program understanding and generation. In North American chapter of the association for computational linguistics: human language technologies, pages 2655-2668, 2021. Google Scholar
  2. Nafisa Ahmed, Hin Chi Kwok, Mohammad Hamdaqa, and Wesley K. G. Assunção. SMATCH-M-LLM: Semantic similarity in metamodel matching with large language models. In IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR), pages 199-210, 2025. URL: https://doi.org/10.1109/MSR66628.2025.00040.
  3. Wesley K. G. Assunção, Luciano Marchezan, Lawrence Arkoh, Alexander Egyed, and Rudolf Ramler. Contemporary software modernization: Strategies, driving forces, and research opportunities. ACM Trans. Softw. Eng. Methodol., 34(5), May 2025. URL: https://doi.org/10.1145/3708527.
  4. Victor R. Basili, Gianluigi Caldiera, and H. Dieter Rombach. Goal question metric paradigm. Encyclopedia of software engineering, 1(528-532):6, 1994. Google Scholar
  5. Keyuan Cheng, Xudong Shen, Yihao Yang, Tengyue Wang, Yang Cao, Muhammad Asif Ali, Hanbin Wang, Lijie Hu, and Di Wang. Codemenv: Benchmarking large language models on code migration. In Findings of the Association for Computational Linguistics: ACL 2025, pages 2719-2744, 2025. Google Scholar
  6. Jia Feng, Jiachen Liu, Cuiyun Gao, Chun Yong Chong, Chaozheng Wang, Shan Gao, and Xin Xia. Complexcodeeval: A benchmark for evaluating large code models on more complex code. In Automated Software Engineering, pages 1895-1906, 2024. URL: https://doi.org/10.1145/3691620.3695552.
  7. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the association for comp. linguistics, pages 1536-1547, 2020. Google Scholar
  8. D. Flook. Python variable renaming tool. https://github.com/dflook/python-minifier, 2025.
  9. Martin Fowler. Refactoring: improving the design of existing code. Addison-Wesley, 2018. Google Scholar
  10. Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the american statistical association, 32(200):675-701, 1937. Google Scholar
  11. Rohit Gheyi, Rian Melo, Jonhnanthan Oliveira, Márcio Ribeiro, and Baldoino Fonseca. Foundation models as oracles for refactoring correctness detection. Empirical Software Engineering, 2026. Google Scholar
  12. Barney Glaser and Anselm Strauss. Discovery of grounded theory: Strategies for qualitative research. Routledge, 2017. Google Scholar
  13. Greg Guest, Arwen Bunce, and Laura Johnson. How many interviews are enough? an experiment with data saturation and variability. Field methods, 18(1):59-82, 2006. Google Scholar
  14. Ryutaro Inoue and Yoshiki Higo. Improving accuracy of LLM-based code clone detection using functionally equivalent methods. In IEEE/ACIS 22nd International Conference on Software Engineering Research, Management and Applications, pages 24-27. IEEE, 2024. URL: https://doi.org/10.1109/SERA61261.2024.10685589.
  15. Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. Language models for code completion: A practical evaluation. In IEEE/ACM 46th International Conference on Software Engineering, pages 1-13, 2024. URL: https://doi.org/10.1145/3597503.3639138.
  16. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 35(2):1-72, 2026. URL: https://doi.org/10.1145/3747588.
  17. Chaofan Li, Jianlyu Chen, Yingxia Shao, Defu Lian, and Zheng Liu. Towards a generalist code embedding model based on massive data synthesis. Advances in Neural Information Processing Systems, 38, 2026. Google Scholar
  18. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 378(6624):1092-1097, 2022. Google Scholar
  19. Fang Liu, Yang Liu, Lin Shi, Zhen Yang, Li Zhang, Xiaoli Lian, Zhongqi Li, and Yuchi Ma. Beyond functional correctness: Exploring hallucinations in LLM-generated code, 2026. URL: https://arxiv.org/abs/2404.00971.
  20. Ramasamy Maruthamuthu, Dharmesh Dhabliya, Ahmed HR Abbas, Abdullaeva Barno, et al. Advancements in compiler design and optimization techniques. In E3S Web of Conferences, volume 399, page 04047. EDP Sciences, 2023. Google Scholar
  21. Nickil Maveli, Antonio Vergari, and Shay B Cohen. What can large language models capture about code functional equivalence? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6880-6918, 2025. Google Scholar
  22. Tural Mehtiyev and Wesley Assunção. Beyond resolution rates: Behavioral drivers of coding agent success and failure. arXiv preprint arXiv:2604.02547, 2026. URL: https://doi.org/10.48550/arXiv.2604.02547.
  23. Alfred Santa Molison, Marcia Moraes, Glaucia Melo, Fabio Santos, and Wesley K. G. Assunção. Is llm-generated code more maintainable & reliable than human-written code? In ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 151-162, 2025. URL: https://doi.org/10.1109/ESEM64174.2025.00036.
  24. Micheline Bénédicte Moumoula, Abdoul Kader Kaboré, Jacques Klein, and Tegawendé F Bissyandé. The struggles of LLMs in cross-lingual code clone detection. Proceedings of the ACM on Software Engineering, 2(FSE):1023-1045, 2025. URL: https://doi.org/10.1145/3715764.
  25. Steven Muchnick. Advanced compiler design implementation. Morgan kaufmann, 1997. Google Scholar
  26. Stoyan Nikolov, Bernhard Konrad, Moritz Gronbach, Niket Kumar, Ann Yan, Varun Singh, Yaning Liang, and Parthasarathy Ranganathan. A multi-agent ai system for deep learning model migration from TensorFlow to JAX, 2026. URL: https://doi.org/10.48550/arXiv.2603.27296.
  27. Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. Lost in translation: A study of bugs introduced by large language models while translating code. In International Conference on Software Engineering, pages 1-13, 2024. URL: https://doi.org/10.1145/3597503.3639226.
  28. Daniel Perez and Shigeru Chiba. Cross-language clone detection by learning over abstract syntax trees. In Mining Software Repositories, pages 518-528. IEEE, 2019. URL: https://doi.org/10.1109/MSR.2019.00078.
  29. Lutz Prechelt. An empirical comparison of seven programming languages. Computer, 33(10):23-29, 2000. URL: https://doi.org/10.1109/2.876288.
  30. Lutz Prechelt, Guido Malpohl, Michael Philippsen, et al. Finding plagiarisms among a set of programs with jplag. J. Univers. Comput. Sci., 8(11):1016-1038, 2002. Google Scholar
  31. Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks, 2021. URL: https://arxiv.org/abs/2105.12655.
  32. John W Ratcliff and David E Metzener. Pattern matching: The gestalt approach. Dr. Dobb’s Journal, 13(7):46, 1988. Google Scholar
  33. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis, 2020. URL: https://arxiv.org/abs/2009.10297.
  34. Reudismam Rolim, Gustavo Soares, Loris D'Antoni, Oleksandr Polozov, Sumit Gulwani, Rohit Gheyi, Ryo Suzuki, and Björn Hartmann. Learning syntactic program transformations from examples. In International Conference on Software Engineering, pages 404-415. IEEE, 2017. URL: https://doi.org/10.1109/ICSE.2017.44.
  35. Chanchal K Roy, James R Cordy, and Rainer Koschke. Comparison and evaluation of code clone detection techniques and tools: A qualitative approach. Science of computer programming, 74(7):470-495, 2009. URL: https://doi.org/10.1016/J.SCICO.2009.02.007.
  36. Chanchal Kumar Roy and James R Cordy. A survey on software clone detection research. Queen’s School of computing TR, 541(115):64-68, 2007. Google Scholar
  37. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85-105, 2023. URL: https://doi.org/10.1109/TSE.2023.3334955.
  38. Saul Schleimer, Daniel S Wilkerson, and Alex Aiken. Winnowing: local algorithms for document fingerprinting. In Int. conference on Management of data, pages 76-85, 2003. Google Scholar
  39. Vanshika Singh, Caglar Korlu, and Wesley K. G. Assunção. Experiences on using large language models to re-engineer a legacy system at volvo group. In IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 102-112, 2025. URL: https://doi.org/10.1109/SANER64311.2025.00018.
  40. Gustavo Soares, Rohit Gheyi, Dalton Serey, and Tiago Massoni. Making program refactoring safer. IEEE Software, 27:52-57, 2010. URL: https://doi.org/10.1109/MS.2010.63.
  41. C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. Google Scholar
  42. Hui Sun, Anderson Uchôa, Rohit Gheyi, and Wesley K. G. Assunção. Evaluating Language Models on Cross-Language Code Functional Equivalence. In International Symposium on Empirical Software Engineering and Measurement, Munich, Germany, 2026. Replication package available at URL: https://doi.org/10.5281/zenodo.21800077.
  43. Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. Towards a big data curated benchmark of inter-project code clones. In 2014 IEEE international conference on software maintenance and evolution, pages 476-480. IEEE, 2014. Google Scholar
  44. Pankaj Manoharlal Thakur, Kyle Thomson, Wesley Klewerton Guez Assunção, and Bowen Xu. Backporting in robot operating system: Identifying commit purpose and propagation need with large language models. In IEEE/ACM Third International Conference on AI Foundation Models and Software Engineering, pages 40-50. ACM, 2026. URL: https://doi.org/10.1145/3793655.3793730.
  45. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Conference on empirical methods in natural language processing, pages 8696-8708, 2021. Google Scholar
  46. Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, Ke Wang, et al. EquiBench: Benchmarking large language models’ understanding of program semantics via equivalence checking. In NeurIPS 2025 Fourth Workshop on Deep Learning for Code, 2025. Google Scholar
  47. Elise Whitley and Jonathan Ball. Statistics review 6: Nonparametric methods. Critical care, 6(6):509, 2002. Google Scholar
  48. Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, et al. On the evaluation of large language models in unit test generation. In Automated Software Engineering, pages 1607-1619, 2024. Google Scholar
  49. Hao Yu, Wing Lam, Long Chen, Ge Li, Tao Xie, and Qianxiang Wang. Neural detection of semantic code clones via tree-based convolution. In International Conference on Program Comprehension, pages 70-80. IEEE, 2019. URL: https://doi.org/10.1109/ICPC.2019.00021.
  50. Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Int. Conference on Software Engineering, pages 1-12, 2024. URL: https://doi.org/10.1145/3597503.3623316.
  51. Xin Zhou, Kisub Kim, Ting Zhang, Martin Weyssow, Luís F Gomes, Guang Yang, Kui Liu, Xin Xia, and David Lo. Se-jury: An LLM-as-ensemble-judge metric for narrowing the gap with human evaluation in se. In Automated Software Engineering, pages 2606-2618. IEEE, 2025. URL: https://doi.org/10.1109/ASE63991.2025.00214.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact