2026-10-05T21:32:35Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:28014 2026-10-05T06:44:04Z ddc:004 open_access

On the Quality of Vulnerability-Contributing Commit Datasets: A Systematic Review and Exploit-Based Evaluation Carvalho Lopes, Vinicius Zampino, Matthew Garcia, Christian Santos, Joanna C. S. Sejfia, Adriana vulnerability-contributing commits software security dataset quality systematic literature review exploit validation Background. Vulnerability-Contributing Commits (VCCs) are the code changes at which a software system turns from safe to unsafe with respect to a specific vulnerability. Accurate VCC identification is crucial for training vulnerability detection models, yet the reliability of existing VCC datasets remains underexplored. If these VCC datasets contain mislabeled commits, models trained on them may learn incorrect patterns. Aims. We present a two-part study of VCC dataset quality: characterizing how VCCs are defined and identified across the literature, and empirically measuring the accuracy of existing VCC datasets. Method. We review 38 papers and, based on their findings, propose an operational VCC definition based on security state transitions and testable through parent commit validation. We apply this definition in an exploit-based validation study on 6 datasets aggregating 3,105 VCC samples, of which 168 (5.41%) have publicly documented exploits enabling empirical validation. Results. We found significant terminological and definitional inconsistencies: 8 distinct terms with varying definitions, 15.8% of papers lacking explicit definitions, and confusion between commits that contain versus commits that introduce vulnerabilities. Among the 168 validated samples, we observe a 79.2% false positive rate, rising to 88.4% in the subset whose verdicts rest entirely on runtime exploit reproduction, with 47.0% being commits labeled as VCCs although the vulnerability was already exploitable in the parent commit. Conclusions. Existing VCC datasets contain labeling errors detectable through exploit-based parent commit validation. Our findings call for standardized VCC definitions and parent commit validation to distinguish VCC commits from those that merely modify already-vulnerable code. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Vinicius Carvalho Lopes and Matthew Zampino and Christian Garcia and Joanna C. S. Santos and Adriana Sejfia 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.46 urn:nbn:de:0030-drops-280146 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.46 eng https://creativecommons.org/licenses/by/4.0/legalcode