2026-10-05T21:32:53Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:28052 2026-10-05T06:44:06Z ddc:004 open_access

Specification Artifacts in Open-Source Pull Request Workflows Do Not Reduce Defects: An Empirical Test of Spec-Driven Development Claims Hill, Brenn Spec-driven development specification artifacts null result defect prediction propensity score matching SZZ AI-assisted development within-author fixed effects Context. Spec-driven development (SDD) tools claim that writing specifications before implementation reduces defects, prevents rework, and improves code quality. No vendor has published empirical evidence for these claims. Method. We test five hypotheses derived from SDD vendor claims against 88,052 pull requests across 119 open-source repositories. What we measure is specification artifacts - overwhelmingly references to tracking issues and tickets - rather than SDD tool output, which no repository in the sample commits. We score 25,209 artifacts on the same quality dimensions SDD tools prescribe, measure rework from subsequent pull request activity, trace defects via the SZZ algorithm, and compare each developer’s spec'd PRs to their own unspec'd PRs. Twelve robustness checks include propensity score matching, complexity stratification, and AI/human subgroup analysis. Results. None of the five hypotheses are supported. After propensity score matching on just-in-time risk profiles, spec'd and unspec'd PRs are indistinguishable in defect introduction (-0.6 percentage points (pp), p = 0.133). Rework shows no protective effect either (+1.2pp, p = 0.001 within-author; +0.5pp, p = 0.146 matched). Specification quality predicts neither defects (p = 0.164) nor rework (p = 0.860), and specifications do not constrain the scope of AI-tagged changes (p = 0.997). The positive raw association between specification artifacts and defects is explained by task complexity: developers spec their hardest work. One subgroup runs the other way - in repositories with no detected AI use, specifications are associated with less rework (-3.3pp, p = 0.014) - but it is one of six subgroup tests and does not survive correction for multiple comparisons. Conclusion. Specification artifacts proxy for task complexity, not quality improvement. SDD tool adoption itself remains untested: no repository in the sample commits such output. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Brenn Hill 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.84 urn:nbn:de:0030-drops-280526 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.84 eng https://creativecommons.org/licenses/by/4.0/legalcode