2026-10-05T21:32:41Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:27974 2026-10-05T06:44:02Z ddc:004 open_access

The Influence of Fraudulent AI-Generated Responses on Software Engineering Surveys de Souza Santos, Ronnie Santos, Italo Baldassarre, Maria Teresa Magalhães, Cleyton Wessel, Mairieli LLMs survey threats to validity Background. Large Language Models (LLMs) introduce new concerns regarding fraudulent or AI-assisted participation in software engineering surveys. Aims. This study investigates how suspicious or potentially AI-assisted responses may affect the validity of software engineering survey findings. Method. We conducted a secondary analysis of four software engineering survey datasets using manual identification of suspicious responses, automated AI-generated text detection, descriptive statistical analysis, and thematic analysis. We compared findings obtained from the original and manually cleaned datasets. Results. Quantitative findings generally remained stable after filtering suspicious responses, although some demographic and analytical variables showed moderate variation, affecting the interpretation of specific participant groups and contextual characteristics. In contrast, qualitative findings were more strongly influenced by changes in contextual framing, code prominence, and the nature of the evidence supporting interpretation, shaping how participants' experiences and study contexts were interpreted and characterized. Conclusions. AI-assisted participation may influence software engineering survey findings differently depending on the type of analysis being conducted. The findings reinforce the importance of combining multiple validation procedures, particularly in studies relying on open-ended responses. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Ronnie de Souza Santos and Italo Santos and Maria Teresa Baldassarre and Cleyton Magalhães and Mairieli Wessel 2026 Is Part Of LIPIcs, Volume 394, 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026) InProceedings Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/LIPIcs.ESEM.2026.6 urn:nbn:de:0030-drops-279748 https://drops.dagstuhl.de/entities/document/10.4230/LIPIcs.ESEM.2026.6 eng https://creativecommons.org/licenses/by/4.0/legalcode