Document Open Access Logo

A Direction-Aware Study of LLM-Based Code Translation Across Eight Programming Languages

Authors Quanhe Wang , Xiangyun Zhan , Cheng Wen , Bin Yu , Ping Chen , Xingjian Han , Shengchao Qin

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.63.pdf
  • Filesize: 4.33 MB
  • 14 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → General programming languages
  • Computing methodologies → Artificial intelligence
Keywords
  • Large language models
  • code translation
  • multilingual programming
  • empirical software engineering
  • execution-based evaluation

Metrics

Abstract

Large language models (LLMs) are increasingly used for code translation, yet existing evaluations often collapse results across language pairs, thereby obscuring the direction of transfer. We present an emerging empirical study that treats multilingual code translation as a directed transfer problem. Our study uses 100 programming problems, each with reference implementations in eight languages, yielding a complete directed graph of 56 ordered source-target translation directions. Five contemporary LLMs generate candidate translations, which are evaluated through execution in the target language. The results reveal substantial directional asymmetry. In model-averaged pass@5, reversing a language pair changes the translation success rate by as much as 27.4 percentage points. We also observe systematic differences between source and target roles: Python3 and Java achieve higher average performance when used as source languages, whereas JavaScript, Rust, and Golang achieve higher average performance when used as target languages. These findings show that aggregate scores and unordered language-pair averages can conceal practically important transfer behavior. Rather than establishing a definitive model ranking, this study provides preliminary evidence that code-translation evaluations should preserve ordered source-target results, distinguish source and target language roles, and explicitly report directional gaps. This direction-aware perspective provides a more informative basis for subsequent failure analysis and the evaluation of realistic cross-language migration scenarios.

Cite As Get BibTex

Quanhe Wang, Xiangyun Zhan, Cheng Wen, Bin Yu, Ping Chen, Xingjian Han, and Shengchao Qin. A Direction-Aware Study of LLM-Based Code Translation Across Eight Programming Languages. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 63:1-63:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.63

Author Details

Quanhe Wang
  • Shenzhen Institute of Technology, Xidian University, Shenzhen, China
  • School of Computer Science and Technology, Xidian University, Xi'an, China
Xiangyun Zhan
  • Guangzhou Institute of Technology, Xidian University, Guangzhou, China
Cheng Wen
  • Shenzhen Institute of Technology, Xidian University, Shenzhen, China
Bin Yu
  • School of Computer Science and Technology, Xidian University, Xi'an, China
Ping Chen
  • China Electronic Product Reliability and Environmental Testing Research Institute (The fifth electronic research institute of MIIT), Guangzhou, China
Xingjian Han
  • China Electronic Product Reliability and Environmental Testing Research Institute (The fifth electronic research institute of MIIT), Guangzhou, China
Shengchao Qin
  • School of Computer Science and Technology, Xidian University, Xi'an, China

Funding

This work was supported in part by the Basic Research Foundation of Shenzhen City (No. JCYJ20250604184202003), the National Natural Science Foundation of China (Nos. 62302375, 62472339), and the China Postdoctoral Science Foundation funded project (No. 2023M723736), and the Open Project of China Electronic Product Reliability and Environmental Testing Research Institute (The Fifth Electronic Research Institute of MIIT).

Supplementary Materials

References

  1. Stefano Bistarelli, Marco Fiore, Ivan Mercanti, and Marina Mongiello. Usage of large language model for code generation tasks: A review. SN Computer Science, 6(6):673, 2025. URL: https://doi.org/10.1007/S42979-025-04241-5.
  2. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 49(7):3675-3691, 2023. URL: https://doi.org/10.1109/TSE.2023.3267446.
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint, 2021. URL: https://arxiv.org/abs/2107.03374.
  4. Chao Jiang, Dugang Liu, Cheng Wen, Zhiwu Xu, Hua Zheng, Muhammad Sadiq, Jawwad Ahmed Shamsi, Shengchao Qin, and Zhong Ming. Large language models for multilingual code intelligence: A survey. In International Conference on Knowledge Science, Engineering and Management. Springer, 2026. Google Scholar
  5. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 35(2):1-72, 2026. URL: https://doi.org/10.1145/3747588.
  6. Mingsheng Jiao, Tingrui Yu, Xuan Li, Guanjie Qiu, Xiaodong Gu, and Beijun Shen. On the evaluation of neural code translation: Taxonomy and benchmark. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1529-1541. IEEE, 2023. URL: https://doi.org/10.1109/ASE56229.2023.00114.
  7. Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang, Md Rizwan Parvez, and Shafiq Joty. Xcodeeval: An execution-based large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6766-6805, 2024. URL: https://doi.org/10.18653/V1/2024.ACL-LONG.367.
  8. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems, volume 36, 2023. Google Scholar
  9. Zhi Ma, Xiao Liang, Cheng Wen, Rui Chen, Bin Gu, Shengchao Qin, Cong Tian, and Mengfei Yang. Automated ltl specification generation from industrial aerospace requirements. In International Symposium on Formal Methods, pages 670-690. Springer, 2026. URL: https://doi.org/10.1007/978-3-032-26220-2_33.
  10. Zhi Ma, Cheng Wen, Zhexin Su, Xiao Liang, Cong Tian, Shengchao Qin, and Mengfei Yang. Bridging natural language and formal specification - automated translation of software requirements to ltl via hierarchical semantics decomposition using llms. In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering, 2025. Google Scholar
  11. Zhi Ma, Cheng Wen, Bin Yu, and Jie Su. Integrating ensemble learning and large language models for efficient formal verification of ip-based aerospace systems. Information Fusion, 125:103466, 2026. URL: https://doi.org/10.1016/J.INFFUS.2025.103466.
  12. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Unsupervised translation of programming languages. In Advances in Neural Information Processing Systems, volume 33, pages 20601-20611, 2020. Google Scholar
  13. Qingxiao Tao, Tingrui Yu, Xiaodong Gu, and Beijun Shen. Unraveling the potential of large language models in code translation: How far are we? In 2024 31st Asia-Pacific Software Engineering Conference (APSEC), pages 353-362. IEEE, 2024. URL: https://doi.org/10.1109/APSEC65559.2024.00046.
  14. Yanli Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, et al. Repotransbench: A real-world multilingual benchmark for repository-level code translation. IEEE Transactions on Software Engineering, 2025. Google Scholar
  15. Yanlin Wang, Rongyi Ou, Yanli Wang, Mingwei Liu, Jiachi Chen, Ensheng Shi, Xilin Liu, Yuchi Ma, and Zibin Zheng. Effireasontrans: Rl-optimized reasoning for code translation. IEEE Transactions on Software Engineering, 2026. Google Scholar
  16. Cheng Wen, Jialun Cao, Jie Su, Zhiwu Xu, Shengchao Qin, Mengda He, Haokun Li, Shing-Chi Cheung, and Cong Tian. Enchanting program specification synthesis by large language models using static analysis and program verification. In International Conference on Computer Aided Verification. Springer, 2024. Google Scholar
  17. Cheng Wen, Junjie Hu, YiKun Hu, Jie Su, Bin Yu, Dugang Liu, Zhiwu Xu, Weidi Sun, Shengchao Qin, and Cong Tian. Formally specifying the intended behavior of the program: Llm-driven neuro-symbolic program specification synthesis. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 672-682, 2026. URL: https://doi.org/10.18653/V1/2026.ACL-DEMO.66.
  18. Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. Codetransocean: A comprehensive multilingual benchmark for code translation. arXiv preprint, 2023. URL: https://doi.org/10.48550/arXiv.2310.04951.
  19. Yuchen Zhang, Cheng Wen, Zhiwu Xu, Dugang Liu, et al. Enhancing llm-based proof synthesis for rust programs via semantic chunking and hierarchical context expansion. In International Symposium on Theoretical Aspects of Software Engineering, pages 81-100, 2026. Google Scholar
  20. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, pages 5673-5684, 2023. Google Scholar
  21. Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K Reddy. Xlcost: A benchmark dataset for cross-lingual code intelligence. arXiv preprint, 2022. URL: https://doi.org/10.48550/arXiv.2206.08474.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact