2026-10-05T21:32:39Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:27993 2026-10-05T06:44:03Z ddc:004 open_access

An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models Wang, Quanhe Wen, Cheng Liu, Dugang Han, Xingjian Yu, Bin Chen, Ping Qin, Shengchao Tian, Cong Large language models code generation code translation multilingual programming empirical software engineering Background. Large language models (LLMs) can generate executable code, but their reliability across programming languages remains difficult to characterize. Aims. We investigate multilingual code capability under problem alignment, focusing on functional reliability, cross-language consistency, translation behavior, problem and sampling effects, and execution quality. Method. We present MciBench, covering 1,489 programming problems and eight languages, of which 1,466 have execution-validated coverage in all eight languages. We evaluate representative LLMs through multilingual generation and analyze source-code-conditioned translation on 100 randomly selected all-language-covered problems. Results. Performance varies substantially across models and target languages, while source-code-conditioned translation also exhibits marked variation across source-target settings. Problem difficulty and candidate budget materially affect correctness, strong aggregate performance does not necessarily imply cross-language consistency, and accepted programs differ in runtime and memory efficiency. Conclusions. Aggregate pass@k alone is insufficient for characterizing multilingual code capability, motivating evaluation that considers language consistency, problem sensitivity, sampling behavior, translation settings, and post-correctness execution quality. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Quanhe Wang and Cheng Wen and Dugang Liu and Xingjian Han and Bin Yu and Ping Chen and Shengchao Qin and Cong Tian 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.25 urn:nbn:de:0030-drops-279936 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.25 eng https://creativecommons.org/licenses/by/4.0/legalcode