Document Open Access Logo

Can LLMs Identify Missing Elements in Requirements Elicitation? An Empirical Evaluation Using Production-Derived User Stories in an Industry Setting

Authors Vinay Kabadi , Abhiram Wuntakal, Naresh Gutha, Lahiri Bellarykar, Xuan Bach D. Le , Patanamon Thongtanunam , Christoph Treude

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.82.pdf
  • Filesize: 0.91 MB
  • 20 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Natural language processing
  • General and reference → Empirical studies
  • Software and its engineering → Requirements analysis
  • Software and its engineering → Software development methods
Keywords
  • Software Engineering
  • Requirement Elicitation
  • Large Language Model (LLM)
  • User Stories

Metrics

Abstract

Business Requirements often fail to fully capture the complete set of system requirements they intend to describe, resulting in under-specification and measurable gaps between documented requirements and production-ready implementations. While experienced business analysts attempt to mitigate these gaps using domain expertise, the resulting artifacts often remain incomplete. 
This study uses Cross-Pollination, a domain-agnostic methodology for transforming incomplete natural language Business Requirements into comprehensive, production-grade functional user stories. To evaluate the completeness of LLM-generated user stories, the study systematically benchmarks them against realistic production artifacts. It further investigates the capability of Large Language Models (LLMs) to bridge the coverage gap between human-authored backlogs and production systems. It identifies the requirements that are overlooked during human-driven elicitation but are recovered through an LLM-assisted approach.
The cross-pollination technique is a structured multi-phase methodology supported by a gap analysis taxonomy and a domain complexity calibration mechanism to establish a minimum baseline for production readiness. The methodology is applied across real-world domains, and a comparative evaluation of analyst-authored, LLM-generated, and production-derived user stories is conducted. Empirical results demonstrate that a substantial proportion of requirement gaps can be systematically identified and addressed through this approach.
The findings indicate that Cross-Pollination provides a reproducible and scalable methodology for translating natural-language requirements into more complete and production-aligned user stories. The results demonstrate that while analysts produce precise requirements, LLM-generated user stories achieve higher recall across all evaluated domains and are effective in uncovering implicit and non-obvious requirements. These findings indicate that an integrated approach, with LLM-assisted elicitation functioning as a complementary mechanism to human expertise, produces more comprehensive and reliable requirement backlogs than either method independently.

Cite As Get BibTex

Vinay Kabadi, Abhiram Wuntakal, Naresh Gutha, Lahiri Bellarykar, Xuan Bach D. Le, Patanamon Thongtanunam, and Christoph Treude. Can LLMs Identify Missing Elements in Requirements Elicitation? An Empirical Evaluation Using Production-Derived User Stories in an Industry Setting. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 82:1-82:20, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.82

Author Details

Vinay Kabadi
  • School of Computing and Information Systems, The University of Melbourne, Australia
Abhiram Wuntakal
  • Trawelli Pty Ltd, Australia
Naresh Gutha
  • Industry Practitioner, Australia
Lahiri Bellarykar
  • Industry Practitioner, Australia
Xuan Bach D. Le
  • School of Computing and Information Systems, The University of Melbourne, Australia
Patanamon Thongtanunam
  • School of Computing and Information Systems, The University of Melbourne, Australia
Christoph Treude
  • School of Computing and Information Systems, Singapore Management University, Singapore

References

  1. Ritu Agarwal and Mohan R Tanniru. Knowledge acquisition using structured interviewing: an empirical investigation. Journal of Management Information Systems, 7(1):123-140, 1990. URL: https://doi.org/10.1080/07421222.1990.11517884.
  2. Yoji Akao. Quality function deployment: integrating customer requirements into product design. Productivity press, 1990. Google Scholar
  3. Bogdan-Alexandru Andrei, Andrei-Cosmin Casu-Pop, Sorin-Catalin Gheorghe, and Costin-Anton Boiangiu. A study on using waterfall and agile methods in software project management. Journal of Information Systems & Operations Management, pages 125-135, 2019. Google Scholar
  4. Chetan Arora, John Grundy, and Mohamed Abdelrazek. Advancing requirements engineering through generative ai: Assessing the role of llms. In Generative AI for Effective Software Development, pages 129-148. Springer, 2024. Google Scholar
  5. Chetan Arora, Mehrdad Sabetzadeh, Lionel Briand, and Frank Zimmer. Automated checking of conformance to requirements templates using natural language processing. IEEE transactions on Software Engineering, 41(10):944-968, 2015. URL: https://doi.org/10.1109/TSE.2015.2428709.
  6. Linden J Ball and Thomas C Ormerod. Putting ethnography to work: the case for a cognitive ethnography of design. International Journal of Human-Computer Studies, 53(1):147-168, 2000. URL: https://doi.org/10.1006/IJHC.2000.0372.
  7. Dipali Baviskar, Swati Ahirrao, Vidyasagar Potdar, and Ketan Kotecha. Efficient automated processing of the unstructured documents using artificial intelligence: A systematic literature review and future directions. Ieee Access, 9:72894-72936, 2021. URL: https://doi.org/10.1109/ACCESS.2021.3072900.
  8. Kent Beck and Ward Cunningham. A laboratory for teaching object oriented thinking. ACM Sigplan Notices, 24(10):1-6, 1989. URL: https://doi.org/10.1145/74877.74879.
  9. Hugh R Beyer and Karen Holtzblatt. Apprenticing with the customer. Communications of the ACM, 38(5):45-52, 1995. URL: https://doi.org/10.1145/203356.203365.
  10. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. Google Scholar
  11. Faiza Allah Bukhsh, Zaharah Allah Bukhsh, and Maya Daneva. A systematic literature review on requirement prioritization techniques and their empirical evaluation. Computer Standards & Interfaces, 69:103389, 2020. URL: https://doi.org/10.1016/J.CSI.2019.103389.
  12. Trevor Butt. George Kelly: The psychology of personal constructs. Bloomsbury Publishing, 2008. Google Scholar
  13. Pär Carlshamre and Joachim Karlsson. A usability-oriented approach to requirements engineering. In Proceedings of the Second International Conference on Requirements Engineering, pages 145-152. IEEE, 1996. URL: https://doi.org/10.1109/ICRE.1996.491439.
  14. OpenCommons contributors. Chaos report on it project outcomes, 2025. URL: https://opencommons.org/CHAOS_Report_on_IT_Project_Outcomes.
  15. Chinedu Alex Ezeigweneme, Aniekan Akpan Umoh, Valentine Ikenna Ilojianya, and Abimbola Oluwatoyin Adegbite. Telecommunications energy efficiency: optimizing network infrastructure for sustainability. Computer Science & IT Research Journal, 5(1):26-40, 2024. Google Scholar
  16. Oluwole Fagbohun, Rachel M Harrison, and Anton Dereventsov. An empirical categorization of prompting techniques for large language models: a practitioner’s guide. arXiv preprint, 2024. URL: https://doi.org/10.48550/arXiv.2402.14837.
  17. D Méndez Fernández, Stefan Wagner, Marcos Kalinowski, Michael Felderer, Priscilla Mafra, Antonio Vetrò, Tayana Conte, M-T Christiansson, Des Greer, Casper Lassenius, et al. Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice. Empirical software engineering, 22(5):2298-2338, 2017. URL: https://doi.org/10.1007/S10664-016-9451-7.
  18. Joseph A Goguen and Charlotte Linde. Techniques for requirements elicitation. In [1993] Proceedings of the IEEE International Symposium on Requirements Engineering, pages 152-164. IEEE, 1993. URL: https://doi.org/10.1109/ISRE.1993.324822.
  19. Ellen Gottesdiener. Requirements by collaboration: workshops for defining needs. Addison-Wesley Professional, 2002. Google Scholar
  20. Abbie Griffin and John R Hauser. The voice of the customer. Marketing science, 12(1):1-27, 1993. Google Scholar
  21. Peter Haumer, Klaus Pohl, and Klaus Weidenhaupt. Requirements elicitation and validation with real world scenes. IEEE transactions on Software Engineering, 24(12):1036-1054, 2002. Google Scholar
  22. Arshia Hemmat, Mohammadreza Sharbaf, Shekoufeh Kolahdouz-Rahimi, Kevin Lano, and Sobhan Y Tehrani. Research directions for using llm in software requirement engineering: A systematic review. Frontiers in Computer Science, 7:1519437, 2025. Google Scholar
  23. David N Hinkle. The change of personal constructs from the viewpoint of a theory of implications. unpublished doctoral dissertation, Ohio State University, 1965. Google Scholar
  24. Karen Holtzblatt and Hugh R Beyer. Requirements gathering: the human factor. Communications of the ACM, 38(5):31-32, 1995. Google Scholar
  25. Haowen Hou, Fei Ma, Binwen Bai, Xinxin Zhu, and Fei Yu. Enhancing and accelerating large language models via instruction-aware contextual compression. arXiv preprint, 2024. URL: https://arxiv.org/abs/2408.15491.
  26. Kaicheng Huang, Fanyu Wang, Yutan Huang, and Chetan Arora. Prompt engineering for requirements engineering: A literature review and roadmap. In 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW), pages 548-557. IEEE, 2025. URL: https://doi.org/10.1109/REW66121.2025.00081.
  27. Azham Hussain, Emmanuel OC Mkpojiogu, and Fazillah Mohmad Kamal. The role of requirements in the success or failure of software projects. International Review of Management and marketing, 6(7):306-311, 2016. Google Scholar
  28. Richard A Krueger. Focus groups: A practical guide for applied research. Sage publications, 2014. Google Scholar
  29. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020. Google Scholar
  30. Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. arXiv preprint, 2023. URL: https://arxiv.org/abs/2305.13269.
  31. Anthony Lowe and Keith Ridgway. Uk user’s guide to quality function deployment. Engineering Management Journal, 10(3):147-155, 2000. Google Scholar
  32. Christopher D Manning. Introduction to information retrieval. Syngress Publishing,, 2008. Google Scholar
  33. Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba-Nabende. Prompt engineering in large language models. In International conference on data intelligence and cognitive informatics, pages 387-402. Springer, 2023. Google Scholar
  34. Karen L McGraw and Karen Harbison-Briggs. Knowledge acquisition: Principles and guidelines. Prentice-Hall, Inc., 1989. Google Scholar
  35. Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. Getting access to what goes on in people’s heads? reflections on the think-aloud technique. In Proceedings of the second Nordic conference on Human-computer interaction, pages 101-110, 2002. URL: https://doi.org/10.1145/572020.572033.
  36. Bashar Nuseibeh, Anthony Finkelstein, and Jeff Kramer. Method engineering for multi-perspective software development. Information and software technology, 38(4):267-274, 1996. URL: https://doi.org/10.1016/0950-5849(95)01054-8.
  37. Oladapo Adeboye Popoola, Henry Ejiga Adama, Chukwuekem David Okeke, and Abiodun Emmanuel Akinoso. Advancements and innovations in requirements elicitation: Developing a comprehensive conceptual model. World Journal of Advanced Research and Reviews, 22(01):1209-1220, 2024. Google Scholar
  38. Colin Potts, Kenji Takahashi, and Annie I. Anton. Inquiry-based requirements analysis. IEEE software, 11(2):21-32, 1994. URL: https://doi.org/10.1109/52.268952.
  39. Aqsa Rasheed, Bushra Zafar, Tehmina Shehryar, Naila Aiman Aslam, Muhammad Sajid, Nouman Ali, Saadat Hanif Dar, and Samina Khalid. Requirement engineering challenges in agile software development. Mathematical Problems in Engineering, 2021(1):6696695, 2021. Google Scholar
  40. Juliet Richardson, Thomas C Ormerod, and Andrew Shepherd. The role of task analysis in capturing requirements for interface design. Interacting with computers, 9(4):367-384, 1998. URL: https://doi.org/10.1016/S0953-5438(97)00036-2.
  41. James Robertson and Suzanne Robertson. Volere. Requirements Specification Templates, 2000. Google Scholar
  42. Suzanne Robertson and James Robertson. Mastering the requirements process: Getting requirements right. Addison-wesley, 2012. Google Scholar
  43. Colette Rolland, Carine Souveyet, and Camille Ben Achour. Guiding goal modeling using scenarios. IEEE transactions on software engineering, 24(12):1055-1071, 1998. URL: https://doi.org/10.1109/32.738339.
  44. Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint, 1, 2024. URL: https://doi.org/10.48550/arXiv.2402.07927.
  45. Yerik Afrianto Singgalen. Coastal and marine tourism monitoring system design using rapid application development (rad). J. Inf. Syst. Res, 5(2):468-479, 2024. Google Scholar
  46. Ian Sommerville, Peter Sawyer, and Stephen Viller. Viewpoints for requirements elicitation: a practical approach. In Proceedings of IEEE International Symposium on Requirements Engineering: RE'98, pages 74-81. IEEE, 1998. URL: https://doi.org/10.1109/ICRE.1998.667811.
  47. Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11-21, 1972. Google Scholar
  48. Tjerk Spijkman, Bente Molenkamp, Steffen Beudeker, Sietse Overbeek, and Fabiano Dalpiaz. Llm-assisted requirements engineering in agile mdd: Industry insights and validation. In 2025 IEEE 33rd International Requirements Engineering Conference (RE), pages 366-377. IEEE, 2025. URL: https://doi.org/10.1109/RE63999.2025.00043.
  49. Alistair G Sutcliffe, Neil AM Maiden, Shailey Minocha, and Darrel Manuel. Supporting scenario-based requirements engineering. IEEE Transactions on software engineering, 24(12):1072-1088, 1998. URL: https://doi.org/10.1109/32.738340.
  50. June Verner, Jennifer Sampson, and Narciso Cerpa. What factors lead to software project failure? In 2008 Second International Conference on Research Challenges in Information Science, pages 71-80, 2008. URL: https://doi.org/10.1109/RCIS.2008.4632095.
  51. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022. Google Scholar
  52. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022. Google Scholar
  53. Mohammad Amin Zadenoori, Jacek Dąbrowski, Waad Alhoshan, Liping Zhao, and Alessio Ferrari. Large language models (llms) for requirements engineering (re): A systematic literature review. arXiv preprint, 2025. URL: https://doi.org/10.48550/arXiv.2509.11446.
  54. Aya Zaki-Ismail, Mohamed Osama, Mohamed Abdelrazek, John Grundy, and Amani Ibrahim. Rcm-extractor: an automated nlp-based approach for extracting a semi formal representation model from natural language requirements. Automated Software Engineering, 29(1):10, 2022. URL: https://doi.org/10.1007/S10515-021-00312-Y.
  55. XiangRui Zhang, XueJie Du, HaoYu Chen, Yongzhong He, Wenjia Niu, and Qiang Li. Automatically generating rules of malicious software packages via large language model. In 2025 55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pages 734-747. IEEE, 2025. URL: https://doi.org/10.1109/DSN64029.2025.00072.
  56. Liping Zhao, Waad Alhoshan, Alessio Ferrari, Keletso J Letsholo, Muideen A Ajagbe, Erol-Valeriu Chioasca, and Riza T Batista-Navarro. Natural language processing for requirements engineering: A systematic mapping study. ACM Computing Surveys (CSUR), 54(3):1-41, 2021. URL: https://doi.org/10.1145/3444689.
  57. Didar Zowghi and Chad Coulin. Requirements elicitation: A survey of techniques, approaches, and tools. In Engineering and managing software requirements, pages 19-46. Springer, 2005. Google Scholar
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact