Search Results

Background. LLM coding assistants are widely used in development, but evidence about their reliability mostly comes from mainstream languages such as Python and Java. Their behaviour on emerging languages, where public code and documentation are scarce, is far less clear. Aims. We ask how capable current LLMs are on Cangjie, an emerging language in the HarmonyOS ecosystem, and which assistive technique is worth its token cost. Method. We build CangjieBench, 248 manually translated tasks derived from HumanEval and ClassEval, run under Text-to-Code and Code-to-Code modalities. We evaluate six LLMs under five methods: direct generation, syntax-constrained prompting, retrieval over documentation, retrieval over code, and agent-based workflows. We report Pass@1, compile rate, token cost, and a seven-category failure taxonomy validated by two LLM annotators (κ ≥ 0.98). Results. Direct generation is unusable on most models, and over four-fifths of failed samples carry foreign syntax that the Cangjie compiler rejects. A 2,146-token syntax reference gives the best accuracy-to-cost trade-off among prompt-based methods. The best agent configuration reaches the highest Pass@1 at 10-120x the token cost of a single prompt, while weaker agents cannot match a syntax-constrained call on the same backbone. A Python reference does not always help: on Code-to-Code it can pull models toward Python idioms and drop compile rates below Text-to-Code. Conclusions. No single method dominates both accuracy and token cost. A practical assistant for an emerging language should ship a concise syntax cheat sheet first, escalate to curated retrieval for API-heavy tasks, and reserve agent loops for tasks that need multi-turn repair. The replication package is at https://github.com/cjhCoder7/CangjieBench.