Document Open Access Logo

How Many Interviews Are Enough in a Software Engineering Study? Preliminary Findings on Sample Size and Saturation

Authors Ronnie de Souza Santos , Italo Santos , Mauricio Rodrigues Lima , Cleyton Magalhães

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.61.pdf
  • Filesize: 2.35 MB
  • 12 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Human-centered computing → Empirical studies in collaborative and social computing
  • Social and professional topics → Computing industry
  • Social and professional topics → Computing profession
  • Human-centered computing → Collaborative and social computing theory, concepts and paradigms
Keywords
  • qualitative research
  • interviews
  • publications

Metrics

Abstract

Background. Interview-based studies are widely used in empirical software engineering to investigate human, organizational, and socio-technical phenomena, yet interview sample adequacy and saturation are reported inconsistently across the literature. 

Aims. This paper investigates how interview sample adequacy and saturation are reported in empirical software engineering research. 

Method. We analyzed 427 papers published between 2016 and 2025 across major software engineering venues, focusing on interview sample sizes, saturation discussions, and sample adequacy justifications. 

Results. Preliminary findings indicate substantial variation in interview sample sizes, ranging from highly specialized small-sample studies to broader investigations involving large interview datasets. Studies involving 12 or fewer interviewees were common and frequently associated with specialized industrial contexts or constrained organizational access. However, the most common range was 13 to 24 interviewees, suggesting that moderate-sized samples represent the most common configuration in empirical software engineering research. Saturation and sample adequacy justifications were heterogeneous, with many studies relying on implicit or contextual reasoning rather than explicit methodological discussion. 

Conclusions. Our findings provide empirical insights into methodological reporting practices in interview-based software engineering research and contribute to ongoing discussions on qualitative rigor and transparency.

Cite As Get BibTex

Ronnie de Souza Santos, Italo Santos, Mauricio Rodrigues Lima, and Cleyton Magalhães. How Many Interviews Are Enough in a Software Engineering Study? Preliminary Findings on Sample Size and Saturation. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 61:1-61:12, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.61

Author Details

Ronnie de Souza Santos
  • University of Calgary, Canada
Italo Santos
  • University of Hawaii at Manoa, USA
Mauricio Rodrigues Lima
  • University of Hawaii at Manoa, USA
Cleyton Magalhães
  • Universidade Federal Rural de Pernambuco (UFRPE), Brazil

Supplementary Materials

References

  1. Sebastian Baltes and Paul Ralph. Sampling in software engineering research: A critical review and guidelines. Empirical Software Engineering, 27(4):94, 2022. URL: https://doi.org/10.1007/S10664-021-10072-8.
  2. Wasihun Bezabih Bekele and Fikire Yohanes Ago. Sample size for interview in qualitative research in social sciences: A guide to novice researchers. Research in Educational Policy and Management, 4(1):42-50, 2022. Google Scholar
  3. Virginia Braun and Victoria Clarke. To saturate or not to saturate? questioning data saturation as a useful concept for thematic analysis and sample-size rationales. Qualitative research in sport, exercise and health, 13(2):201-216, 2021. Google Scholar
  4. William Cobern and Betty Adams. When interviewing: how many is enough? International Journal of Assessment Tools in Education, 7(1):73-79, 2020. Google Scholar
  5. Ray W Cooksey. Descriptive statistics for summarising data. In Illustrating statistical procedures: Finding meaning in quantitative data, pages 61-139. Springer, 2020. Google Scholar
  6. Daniela S Cruzes and Tore Dyba. Recommended steps for thematic synthesis in software engineering. In 2011 international symposium on empirical software engineering and measurement, pages 275-284. IEEE, 2011. Google Scholar
  7. Shari L Dworkin. Sample size policy for qualitative studies using in-depth interviews. Archives of sexual behavior, 41(6):1319-1320, 2012. Google Scholar
  8. Tore Dybå, Rafael Prikladnicki, Kari Rönkkö, Carolyn Seaman, and Jonathan Sillito. Qualitative research in software engineering. Empirical Software Engineering, 16(4):425-429, 2011. URL: https://doi.org/10.1007/S10664-011-9163-Y.
  9. Kasper Trolle Elmholdt, Michael Gill, and Jeppe Agger Nielsen. “how many interviews do i need?” an examination of interview numbers and sampling moves in qualitative research. Organizational Research Methods, page 10944281261424516, 2026. Google Scholar
  10. Paul Ralph et al. Empirical standards for software engineering research, 2020. URL: https://arxiv.org/abs/2010.03525.
  11. Jill J Francis, Marie Johnston, Clare Robertson, Liz Glidewell, Vikki Entwistle, Martin P Eccles, and Jeremy M Grimshaw. What is an adequate sample size? operationalising data saturation for theory-based interview studies. Psychology and health, 25(10):1229-1245, 2010. Google Scholar
  12. Greg Guest, Arwen Bunce, and Laura Johnson. How many interviews are enough? an experiment with data saturation and variability. Field methods, 18(1):59-82, 2006. Google Scholar
  13. Ashley K Hagaman and Amber Wutich. How many interviews are enough to identify metathemes in multisited and cross-cultural research? another perspective on guest, bunce, and johnson’s (2006) landmark study. Field methods, 29(1):23-41, 2017. Google Scholar
  14. Monique M Hennink, Bonnie N Kaiser, and Vincent C Marconi. Code saturation versus meaning saturation: how many interviews are enough? Qualitative health research, 27(4):591-608, 2017. Google Scholar
  15. Siw Elisabeth Hove and Bente Anda. Experiences from conducting semi-structured interviews in empirical software engineering research. In 11th IEEE International Software Metrics Symposium (METRICS'05), pages 10-pp. IEEE, 2005. Google Scholar
  16. Sonja M Hyrynsalmi, Grischa Liebel, Ronnie de Souza Santos, and Sebastian Baltes. Not real or too soft? on the challenges of publishing interdisciplinary software engineering research. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS), pages 106-117. IEEE, 2025. URL: https://doi.org/10.1109/ICSE-SEIS66351.2025.00016.
  17. Barbara Kitchenham, Stuart Charters, et al. Guidelines for performing systematic literature reviews in software engineering, 2007. Google Scholar
  18. Barbara A Kitchenham, Tore Dyba, and Magne Jorgensen. Evidence-based software engineering. In Proceedings. 26th International Conference on Software Engineering, pages 273-281. IEEE, 2004. Google Scholar
  19. Per Lenberg, Robert Feldt, Lucas Gren, Lars Göran Wallgren Tengberg, Inga Tidefors, and Daniel Graziotin. Qualitative software engineering research: Reflections and guidelines. Journal of Software: Evolution and Process, 36(6):e2607, 2024. URL: https://doi.org/10.1002/SMR.2607.
  20. Kirsti Malterud, Volkert Dirk Siersma, and Ann Dorrit Guassora. Sample size in qualitative interview studies: guided by information power. Qualitative health research, 26(13):1753-1760, 2016. Google Scholar
  21. Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. Systematic mapping studies in software engineering. In 12th international conference on evaluation and assessment in software engineering (EASE). BCS Learning & Development, 2008. Google Scholar
  22. Carolyn B. Seaman. Qualitative methods in empirical studies of software engineering. IEEE Transactions on software engineering, 25(4):557-572, 1999. URL: https://doi.org/10.1109/32.799955.
  23. Klaas-Jan Stol, Paul Ralph, and Brian Fitzgerald. Grounded theory in software engineering research: a critical review and guidelines. In Proceedings of the 38th International conference on software engineering, pages 120-131, 2016. URL: https://doi.org/10.1145/2884781.2884833.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact