Search Results

Documents authored by Qin, Shengchao

Document
Technical Track Paper
DOI: 10.4230/LIPIcs.ESEM.2026.25
An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models

Authors: Quanhe Wang, Cheng Wen, Dugang Liu, Xingjian Han, Bin Yu, Ping Chen, Shengchao Qin, and Cong Tian

Published in: LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)

Abstract
Background. Large language models (LLMs) can generate executable code, but their reliability across programming languages remains difficult to characterize. Aims. We investigate multilingual code capability under problem alignment, focusing on functional reliability, cross-language consistency, translation behavior, problem and sampling effects, and execution quality. Method. We present MciBench, covering 1,489 programming problems and eight languages, of which 1,466 have execution-validated coverage in all eight languages. We evaluate representative LLMs through multilingual generation and analyze source-code-conditioned translation on 100 randomly selected all-language-covered problems. Results. Performance varies substantially across models and target languages, while source-code-conditioned translation also exhibits marked variation across source-target settings. Problem difficulty and candidate budget materially affect correctness, strong aggregate performance does not necessarily imply cross-language consistency, and accepted programs differ in runtime and memory efficiency. Conclusions. Aggregate pass@k alone is insufficient for characterizing multilingual code capability, motivating evaluation that considers language consistency, problem sensitivity, sampling behavior, translation settings, and post-correctness execution quality.
Cite as

Quanhe Wang, Cheng Wen, Dugang Liu, Xingjian Han, Bin Yu, Ping Chen, Shengchao Qin, and Cong Tian. An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 25:1-25:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026)

Copy BibTex To Clipboard

@InProceedings{wang_et_al:LIPIcs.ESEM.2026.25,
  author =	{Wang, Quanhe and Wen, Cheng and Liu, Dugang and Han, Xingjian and Yu, Bin and Chen, Ping and Qin, Shengchao and Tian, Cong},
  title =	{{An Empirical Study of Problem-Aligned Multilingual Code Generation and Directed Code Translation by Large Language Models}},
  booktitle =	{20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)},
  pages =	{25:1--25:21},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-450-5},
  ISSN =	{1868-8969},
  year =	{2026},
  volume =	{394},
  editor =	{Feldt, Robert and Paasivaara, Maria and Mendez, Daniel and Wagner, Stefan and Bar\'{o}n, Marvin Mu\~{n}oz},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.25},
  URN =		{urn:nbn:de:0030-drops-279936},
  doi =		{10.4230/LIPIcs.ESEM.2026.25},
  annote =	{Keywords: Large language models, code generation, code translation, multilingual programming, empirical software engineering}
}
Document
Emerging Results, Vision & Reflection Track Paper
DOI: 10.4230/LIPIcs.ESEM.2026.63
A Direction-Aware Study of LLM-Based Code Translation Across Eight Programming Languages

Authors: Quanhe Wang, Xiangyun Zhan, Cheng Wen, Bin Yu, Ping Chen, Xingjian Han, and Shengchao Qin

Published in: LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)

Abstract
Large language models (LLMs) are increasingly used for code translation, yet existing evaluations often collapse results across language pairs, thereby obscuring the direction of transfer. We present an emerging empirical study that treats multilingual code translation as a directed transfer problem. Our study uses 100 programming problems, each with reference implementations in eight languages, yielding a complete directed graph of 56 ordered source-target translation directions. Five contemporary LLMs generate candidate translations, which are evaluated through execution in the target language. The results reveal substantial directional asymmetry. In model-averaged pass@5, reversing a language pair changes the translation success rate by as much as 27.4 percentage points. We also observe systematic differences between source and target roles: Python3 and Java achieve higher average performance when used as source languages, whereas JavaScript, Rust, and Golang achieve higher average performance when used as target languages. These findings show that aggregate scores and unordered language-pair averages can conceal practically important transfer behavior. Rather than establishing a definitive model ranking, this study provides preliminary evidence that code-translation evaluations should preserve ordered source-target results, distinguish source and target language roles, and explicitly report directional gaps. This direction-aware perspective provides a more informative basis for subsequent failure analysis and the evaluation of realistic cross-language migration scenarios.
Cite as

Quanhe Wang, Xiangyun Zhan, Cheng Wen, Bin Yu, Ping Chen, Xingjian Han, and Shengchao Qin. A Direction-Aware Study of LLM-Based Code Translation Across Eight Programming Languages. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 63:1-63:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026)

Copy BibTex To Clipboard

@InProceedings{wang_et_al:LIPIcs.ESEM.2026.63,
  author =	{Wang, Quanhe and Zhan, Xiangyun and Wen, Cheng and Yu, Bin and Chen, Ping and Han, Xingjian and Qin, Shengchao},
  title =	{{A Direction-Aware Study of LLM-Based Code Translation Across Eight Programming Languages}},
  booktitle =	{20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026)},
  pages =	{63:1--63:14},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-450-5},
  ISSN =	{1868-8969},
  year =	{2026},
  volume =	{394},
  editor =	{Feldt, Robert and Paasivaara, Maria and Mendez, Daniel and Wagner, Stefan and Bar\'{o}n, Marvin Mu\~{n}oz},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.63},
  URN =		{urn:nbn:de:0030-drops-280313},
  doi =		{10.4230/LIPIcs.ESEM.2026.63},
  annote =	{Keywords: Large language models, code translation, multilingual programming, empirical software engineering, execution-based evaluation}
}

Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact