Search Results

Context. Spec-driven development (SDD) tools claim that writing specifications before implementation reduces defects, prevents rework, and improves code quality. No vendor has published empirical evidence for these claims. Method. We test five hypotheses derived from SDD vendor claims against 88,052 pull requests across 119 open-source repositories. What we measure is specification artifacts - overwhelmingly references to tracking issues and tickets - rather than SDD tool output, which no repository in the sample commits. We score 25,209 artifacts on the same quality dimensions SDD tools prescribe, measure rework from subsequent pull request activity, trace defects via the SZZ algorithm, and compare each developer’s spec'd PRs to their own unspec'd PRs. Twelve robustness checks include propensity score matching, complexity stratification, and AI/human subgroup analysis. Results. None of the five hypotheses are supported. After propensity score matching on just-in-time risk profiles, spec'd and unspec'd PRs are indistinguishable in defect introduction (-0.6 percentage points (pp), p = 0.133). Rework shows no protective effect either (+1.2pp, p = 0.001 within-author; +0.5pp, p = 0.146 matched). Specification quality predicts neither defects (p = 0.164) nor rework (p = 0.860), and specifications do not constrain the scope of AI-tagged changes (p = 0.997). The positive raw association between specification artifacts and defects is explained by task complexity: developers spec their hardest work. One subgroup runs the other way - in repositories with no detected AI use, specifications are associated with less rework (-3.3pp, p = 0.014) - but it is one of six subgroup tests and does not survive correction for multiple comparisons. Conclusion. Specification artifacts proxy for task complexity, not quality improvement. SDD tool adoption itself remains untested: no repository in the sample commits such output.