Search Results

Background. Threats to validity (TTV) are a core component of empirical research, yet software engineering studies fail to address them adequately. Published studies often omit validity discussions entirely, report generic concerns disconnected from the study’s actual methodology, or treat validity analysis as a procedural formality rather than a purposeful part of the research process. Aim. This study investigates whether large language models (LLMs) can generate threats to validity that are meaningful, study-specific, and novel, i.e., not reported by the original authors. Method. We conducted an empirical experiment using Gemini 2.5 Flash across 375 papers from ICSE 2025, withholding each paper’s TTV section prior to model input, ensuring the model reasoned from the study’s content rather than reproducing author-reported concerns. Generated threats were evaluated against a four-dimension rubric covering relevance, specificity, clarity, and mitigation quality, and compared against the original threats to determine novelty. All threats were then grouped into a taxonomy of seven categories to characterize the distribution of generated validity concerns. Results. The experiment produced 2,673 threats across the dataset, of which 97% were rated as relevant, specific, and clearly articulated, and 92% received high ratings for mitigation quality. Furthermore, 61% of the generated threats were absent from the original papers, suggesting that the model identified validity concerns beyond those reported by the original authors. Conclusions. LLMs can generate technically sound and previously unreported threats to validity at scale, demonstrating their capacity to support researchers in identifying overlooked methodological risks and strengthening the completeness of validity discussions, while augmenting rather than replacing researcher judgment in the process.