2026-10-05T21:32:54Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:28046 2026-10-05T06:44:06Z ddc:004 open_access

CWEFT: CWE-Aware Evaluation of Free-Text vs. Typed Prompts Carbonell, Lanaya A. Koyuncu, Anıl Automated vulnerability repair Automated program repair Large language models Prompt engineering Structured prompting Software security Vul4J Adding vulnerability context to a prompt is known to improve LLM repair quality, but the studies that establish this vary two things at once: the domain knowledge injected, such as a CWE label or a fix pattern, and the form that knowledge takes in the prompt. Prompt-engineering research treats format as a variable in its own right, yet in automated vulnerability repair it has never been isolated under a controlled ablation, leaving open whether the wording of guidance matters once its content is fixed. This paper presents CWEFT, a controlled ablation that holds injected repair knowledge constant across a free-text and a typed-schema rendering while decomposing the separate contributions of naming a vulnerability and supplying fix guidance. We construct four prompt conditions that differ by exactly one factor each, grounding all injected knowledge in APR4Vul’s empirically mined fix patterns. Using this design, we generate 504 patches across 42 Java vulnerabilities from Vul4J spanning 15 CWE classes on three frontier models - Claude Sonnet 4.5, GPT-5, and Gemini 2.5 Pro - and score each patch on a Proof-of-Vulnerability-aware hierarchy that separates code that compiles from code that closes the vulnerability and from code that does so without regression. Our emerging results invert a common assumption: naming the CWE class alone achieves the highest correct-fix rate at 19.8%, ahead of the 12.7% unenriched baseline (p = 0.049, exact McNemar), while both richer conditions that add explicit repair guidance fall back to 16.7%. Crucially, the free-text and typed-schema conditions are indistinguishable at 16.7% with perfectly symmetric paired discordance, and with only eight discordant pairs the design cannot rule out a smaller format effect. Across all conditions, repair rate tracks how concrete the vulnerability class is, falling to 3.1% for improper input validation. These initial findings suggest that for frontier models on common vulnerability classes, what a prompt identifies matters more than how elaborately that information is formatted - a direction that warrants higher-powered replication before it is treated as a design principle. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Lanaya A. Carbonell and Anıl Koyuncu 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.78 urn:nbn:de:0030-drops-280469 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.78 eng https://creativecommons.org/licenses/by/4.0/legalcode