Document Open Access Logo

Towards Reliable AI-Assisted Behavioral Modeling: Evaluating Prompting, Retrieval, and Repair Strategies for UML State Transition Diagram Generation

Authors Mussammat Maimuna Faria , Mashiat Amin Farin , Yasin Sazid , Ahmedul Kabir

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.19.pdf
  • Filesize: 3.22 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software design engineering
Keywords
  • State Transition Diagrams
  • Behavioral Software Modeling

Metrics

Abstract

Background. Generating behavioral models like state transition diagrams from natural language requirements presents challenges in requirements analysis and design. Traditional NLP and ML methods struggle to maintain the correct execution flow and structural consistency. Recent advances in LLMs offer new opportunities, though their effectiveness in generating behavioral models, particularly concerning prompting, retrieval, and repair strategies, remains largely unexamined. 

Aims. This paper evaluates how LLM-based generation strategies (zero-shot, one-shot, few-shot, RAG) affect state transition diagram quality (correctness, completeness, understandability, terminological alignment) and whether iterative repair improves validity. 

Method. We built a dataset of 80 requirement–diagram pairs from software engineering textbooks. We generated PlantUML code for each requirement using four prompting strategies across four open-source LLMs. To isolate the effect of different knowledge sources, we conducted a RAG analysis study comparing four retrieval corpus configurations (full, examples-only, rules-only, theory-only). We also assessed an iterative repair workflow using automated evaluation. We evaluated syntactic validity via PlantUML validation and structural validity via rule-based analysis, supplementing with human evaluation of quality dimensions. We produced 1,169 PlantUML outputs, of which 120 validated outputs were manually evaluated. 

Results. Few-shot prompting improved syntactic validity from 28.7% to 93.5% on average over zero-shot. Iterative repair significantly improved structural validity, especially for DeepSeek (from 22.2% to 88.9%), with most repairs succeeding within two iterations. Human evaluation showed understandability was rated higher than correctness, indicating a quality-perception gap. 

Conclusions. We recommend few-shot prompting with iterative repair for AI-assisted behavioral modeling. Our findings reveal critical trade-offs: retrieval improves syntax but risks hallucination, model architecture matters more than strategy, and automated validation cannot substitute for human correctness assessment. This is the first empirical study systematically evaluating prompting, retrieval, and repair in LLM-based UML state transition diagram generation from natural language.

Cite As Get BibTex

Mussammat Maimuna Faria, Mashiat Amin Farin, Yasin Sazid, and Ahmedul Kabir. Towards Reliable AI-Assisted Behavioral Modeling: Evaluating Prompting, Retrieval, and Repair Strategies for UML State Transition Diagram Generation. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 19:1-19:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.19

Author Details

Mussammat Maimuna Faria
  • Institute of Information Technology, University of Dhaka, Dhaka, Bangladesh
Mashiat Amin Farin
  • Computer Science Department, University of Texas at Dallas, Richardson, TX, USA
Yasin Sazid
  • Computer Science Department, University of Texas at Dallas, Richardson, TX, USA
Ahmedul Kabir
  • Institute of Information Technology, University of Dhaka, Dhaka, Bangladesh

Funding

  • Faria, Mussammat Maimuna: Supported by a fellowship from the ICT Division, Government of Bangladesh (Memo No. 56.00.0000.000.052.33.0001.25-178, dated 9 June 2026).

Supplementary Materials

References

  1. Sharif Ahmed, Arif Ahmed, and Nasir U Eisty. Automatic transformation of natural to unified modeling language: A systematic review. In 2022 IEEE/ACIS 20th International Conference on Software Engineering Research, Management and Applications (SERA), pages 112-119. IEEE, 2022. URL: https://doi.org/10.1109/SERA54885.2022.9806783.
  2. Jim Arlow and Ila Neustadt. UML 2 and the Unified Process: Practical Object-Oriented Analysis and Design. Addison-Wesley Professional, Boston, MA, 2nd edition, 2005. Google Scholar
  3. Felix Bachmann, Len Bass, and Mark Klein. Preliminary design of arche: A software architecture design assistant. Technical Report CMU/SEI-2003-TR-021, Software Engineering Institute, Carnegie Mellon University, 2003. URL: https://doi.org/10.1184/R1/6582149.v1.
  4. Lenz Belzner, Thomas Gabor, and Martin Wirsing. Large language model assisted software engineering: Prospects, challenges, and a case study. In Bridging the Gap Between AI and Reality: First International Conference, AISoLA 2023, Crete, Greece, October 23–28, 2023, Proceedings, pages 355-374, 2023. URL: https://doi.org/10.1007/978-3-031-46002-9_23.
  5. Simon Bennett, Steve McRobb, and Ray Farmer. Object-oriented systems analysis and design using UML. McGraw Hill Higher Education, 2006. Google Scholar
  6. Islem Bouzenia and Michael Pradel. Understanding software engineering agents: A study of thought-action-result trajectories. In Proceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025), 2025. arXiv preprint arXiv:2506.18824. URL: https://doi.org/10.48550/arXiv.2506.18824.
  7. Humberto Cervantes, Rick Kazman, and Yuanfang Cai. An llm-assisted approach to designing software architectures using add, 2025. URL: https://doi.org/10.48550/arXiv.2506.22688.
  8. Kua Chen, Yujing Yang, Boqi Chen, José Antonio Hernández López, Gunter Mussbacher, and Dániel Varró. Automated domain modeling with large language models: A comparative study. In 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS), pages 162-172, 2023. URL: https://doi.org/10.1109/MODELS58315.2023.00037.
  9. Federico Ciccozzi, Ivano Malavolta, and Bran Selic. Execution of uml models: a systematic review of research and practice. Softw. Syst. Model., 18(3):2313-2360, June 2019. URL: https://doi.org/10.1007/s10270-018-0675-4.
  10. Maria Stella De Biase, Simona Bernardi, Stefano Marrone, José Merseguer, and Angelo Palladino. Completion of sysml state machines from given-when-then requirements. Software and Systems Modeling, 23(6):1455-1491, 2024. URL: https://doi.org/10.1007/S10270-024-01228-3.
  11. Malinda Dilhara, Abhiram Bellur, Timofey Bryksin, and Danny Dig. Unprecedented code change automation: The fusion of llms and transformation by example. Proceedings of the ACM on Software Engineering, 1(FSE):631-653, 2024. URL: https://doi.org/10.1145/3643755.
  12. Matteo Esposito, Xiaozhou Li, Sergio Moreschini, Noman Ahmad, Tomas Cerny, Karthik Vaidhyanathan, Valentina Lenarduzzi, and Davide Taibi. Generative ai for software architecture. applications, challenges, and future directions, 2025. URL: https://doi.org/10.48550/arXiv.2503.13310.
  13. Mussammat Maimuna Faria, Mashiat Amin Farin, Yasin Sazid, and Ahmedul Kabir. mashiat0808/State_Transition_Diagram_LLM_workflow. Software, (visited on 2026-09-21). URL: https://github.com/mashiat0808/State_Transition_Diagram_LLM_workflow
    Software Heritage Logo archived version
    full metadata available at: https://doi.org/10.4230/artifacts.28071
  14. Mussammat Maimuna Faria, Mashiat Amin Farin, Yasin Sazid, and Ahmedul Kabir. State transition diagram llm workflow, 2026. URL: https://github.com/mashiat0808/State_Transition_Diagram_LLM_workflow.
  15. Nick Feng, Lina Marsso, Sinem Getir Yaman, Isobel Standen, Yesugen Baatartogtokh, Reem Ayad, Victória Oldemburgo de Mello, Beverley Townsend, Hanne Bartels, Ana Cavalcanti, Radu Calinescu, and Marsha Chechik. Normative requirements operationalization with large language models. In Proceedings of the 2024 IEEE 32nd International Requirements Engineering Conference (RE 2024), pages 129-141, 2024. URL: https://doi.org/10.1109/re59067.2024.00022.
  16. Alessio Ferrari, Sallam Abualhaija, and Chetan Arora. Model generation with llms: From requirements to uml sequence diagrams, 2024. URL: https://arxiv.org/abs/2404.06371.
  17. Jeff Garland and Richard Anthony. Large-scale software architecture: a practical guide using UML. John Wiley & Sons, 2003. Google Scholar
  18. Soham Ghosh and Gaurav Mittal. Advancing engineering research through context-aware and knowledge graph-based retrieval-augmented generation. Frontiers in Artificial Intelligence, 8:1697169, 2025. URL: https://doi.org/10.3389/FRAI.2025.1697169.
  19. Polydoros Giannouris and Sophia Ananiadou. Nomad: A multi-agent llm system for uml class diagram generation from natural language requirements. In Proceedings of the 14th International Conference on Model-Based Software and Systems Engineering, pages 257-264, 2026. URL: https://doi.org/10.5220/0014301900004058.
  20. Hassan Gomaa. Software modeling and design: UML, use cases, patterns, and software architectures. Cambridge University Press, 2011. Google Scholar
  21. Patrick Grässle, Henriette Baumann, and Philippe Baumann. UML 2.0 in Action: A Project-Based Tutorial. Packt Publishing, 2005. Google Scholar
  22. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The llama 3 herd of models. arXiv preprint, 2024. URL: https://arxiv.org/abs/2407.21783.
  23. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint, 2025. URL: https://arxiv.org/abs/2501.12948.
  24. Mohammad Kasra Habib, Daniel Graziotin, and Stefan Wagner. Reqbrain: Task-specific instruction tuning of llms for ai-assisted requirements generation, 2025. URL: https://doi.org/10.48550/arXiv.2505.17632.
  25. M.P.E. Heimdahl and N.G. Leveson. Completeness and consistency in hierarchical state-based requirements. IEEE Transactions on Software Engineering, 22(6):363-377, 1996. URL: https://doi.org/10.1109/32.508311.
  26. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8), December 2024. URL: https://doi.org/10.1145/3695988.
  27. James Ivers and Ipek Ozkaya. Will generative ai fill the automation gap in software architecting? In 2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C), pages 41-45, 2025. URL: https://doi.org/10.1109/ICSA-C65153.2025.00014.
  28. Munima Jahan, Zahra Shakeri Hossein Abad, and Behrouz Far. Generating sequence diagram from natural language requirements. In 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), pages 39-48. IEEE, 2021. URL: https://doi.org/10.1109/REW53955.2021.00012.
  29. Munima Jahan, Mohammad Mahdi Hassan, Reza Golpayegani, Golshid Ranjbaran, Chanchal Roy, Banani Roy, and Kevin Schneider. Automated derivation of uml sequence diagrams from user stories: Unleashing the power of generative ai vs. a rule-based approach. In Proceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems, pages 138-148, 2024. URL: https://doi.org/10.1145/3640310.3674081.
  30. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL: https://doi.org/10.48550/arXiv.2310.06825.
  31. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. ACM Transactions on Software Engineering and Methodology, 35(2):1-72, 2026. URL: https://doi.org/10.1145/3747588.
  32. Rodi Jolak, Maxime Savary-Leblanc, Manuela Dalibor, Andreas Wortmann, Regina Hebig, Juraj Vincur, Ivan Polasek, Xavier Le Pallec, Sébastien Gérard, and Michel RV Chaudron. Software engineering whispers: The effect of textual vs. graphical software design descriptions on software design communication. Empirical software engineering, 25(6):4427-4471, 2020. URL: https://doi.org/10.1007/S10664-020-09835-6.
  33. Craig Larman. Applying UML and patterns: an introduction to object oriented analysis and design and iterative development. Prentice Hall PTR, Upper Saddle River, NJ, USA, 2004. Google Scholar
  34. Taslim Mahbub, Dana Dghaym, Aadhith Shankarnarayanan, Taufiq Syed, Salsabeel Shapsough, and Imran Zualkernan. Can gpt-4 aid in detecting ambiguities, inconsistencies, and incompleteness in requirements analysis? a comprehensive case study. IEEE Access, 12:171972-171992, 2024. URL: https://doi.org/10.1109/ACCESS.2024.3464242.
  35. Russ Miles and Kim Hamilton. Learning UML 2.0: a pragmatic introduction to UML. O'Reilly Media, Inc., 2006. Google Scholar
  36. Jackson Nguyen, Rui En Koe, Fanyu Wang, Chetan Arora, and Alessio Ferrari. Class model generation from requirements using large language models. arXiv preprint, 2026. URL: https://doi.org/10.48550/arXiv.2603.09100.
  37. Bashar Nuseibeh and Steve Easterbrook. Requirements engineering: a roadmap. In Proceedings of the Conference on The Future of Software Engineering, pages 35-46, 2000. URL: https://doi.org/10.1145/336512.336523.
  38. Object Management Group. Unified modeling language (UML) version 2.5.1. Standard formal/17-12-05, Object Management Group, December 2017. URL: https://www.omg.org/spec/UML/2.5.1.
  39. Leonid Ozirkovskyy, Bohdan Volochiy, Oleksandr Shkiliuk, Mykhailo Zmysnyi, and Pavlo Kazan. Functional safety analysis of safety-critical system using state transition diagram. Radioelectronic and Computer Systems, 0(2):145-158, 2022. URL: https://doi.org/10.32620/reks.2022.2.12.
  40. PlantUML Team. Plantuml language reference guide. URL: https://plantuml.com/guide.
  41. Roger S Pressman. Software engineering: a practitioner’s approach. Palgrave macmillan, 2005. Google Scholar
  42. Paul Ralph, Nauman bin Ali, Sebastian Baltes, et al. Empirical standards for software engineering research, 2020. URL: https://arxiv.org/abs/2010.03525.
  43. Aqsa Rasheed, Bushra Zafar, Tehmina Shehryar, Naila Aiman Aslam, Muhammad Sajid, Nouman Ali, Saadat Hanif Dar, and Samina Khalid. Requirement engineering challenges in agile software development. Mathematical Problems in Engineering, 2021(1):6696695, 2021. Google Scholar
  44. Pascal Roques. UML in practice: the art of modeling software systems demonstrated through worked examples and solutions. John Wiley & Sons, 2006. Google Scholar
  45. Nimra Shehzadi, Javed Ferzund, Rubia Fatima, and Adnan Riaz. Automatic complexity analysis of uml class diagrams using visual question answering (vqa) techniques. Software, 4(4):22, 2025. URL: https://doi.org/10.3390/SOFTWARE4040022.
  46. Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. Between lines of code: Unraveling the distinct patterns of machine and human programmers. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, pages 1628-1639, 2025. URL: https://doi.org/10.1109/ICSE55347.2025.00005.
  47. Ian Sommerville. Software Engineering. Pearson, 9 edition, 2011. Google Scholar
  48. Suriya Sundaramoorthy. UML diagramming: a case study approach. Auerbach Publications, 2022. Google Scholar
  49. Brennan Swick, Sean Donegan, Andrew Gillman, and Michael Groeber. Human planning of robot actions through llm-guided state machine synthesis. In 2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN), pages 430-437, 2024. URL: https://doi.org/10.1109/RO-MAN60168.2024.10731325.
  50. Valerio Terragni, Annie Vella, Partha Roop, and Kelly Blincoe. The future of ai-driven software engineering. ACM Trans. Softw. Eng. Methodol., 34(5), May 2025. URL: https://doi.org/10.1145/3715003.
  51. Beian Wang, Chong Wang, Peng Liang, Bing Li, and Cheng Zeng. How llms aid in uml modeling: An exploratory study with novice analysts, 2024. URL: https://doi.org/10.48550/arXiv.2404.17739.
  52. Chengjie Wang, Jingzheng Wu, Xiang Ling, Tianyue Luo, and Chen Zhao. Correct code, vulnerable dependencies: A large scale measurement study of llm-specified library versions. In Proceedings of the International Conference on Software Engineering (ICSE 2026), 2026. arXiv preprint arXiv:2605.06279. URL: https://doi.org/10.48550/arXiv.2605.06279.
  53. Haoyu Wang, Christopher M. Poskitt, and Jun Sun. Agentspec: Customizable runtime enforcement for safe and reliable llm agents. In Proceedings of the 48th International Conference on Software Engineering (ICSE 2026), 2026. arXiv preprint arXiv:2503.18666. URL: https://doi.org/10.48550/arXiv.2503.18666.
  54. Yuchen Wang, Shangxin Guo, and Chee Wei Tan. From code generation to software testing: Ai copilot with context-based rag. IEEE Software, 42(4):34-42, 2025. URL: https://doi.org/10.1109/MS.2025.3549628.
  55. Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt, 2023. URL: https://doi.org/10.48550/arXiv.2302.11382.
  56. Chi Xiao, Daniel Ståhl, and Jan Bosch. Uml sequence diagram generation: A multi-model, multi-domain evaluation. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 272-283, 2025. URL: https://doi.org/10.1109/ICSE-SEIP66354.2025.00030.
  57. Aashish Yadavally and Tien N. Nguyen. From seed to scope: Reasoning to identify change impact sets. In Proceedings of the 48th IEEE/ACM International Conference on Software Engineering, ICSE '26, page To Appear, 2026. Google Scholar
  58. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint, 2024. URL: https://arxiv.org/abs/2412.15115.
  59. Tao Yue, Lionel C Briand, and Yvan Labiche. atoucan: an automated framework to derive uml analysis models from use case models. ACM Transactions on Software Engineering and Methodology (TOSEM), 24(3):1-52, 2015. URL: https://doi.org/10.1145/2699697.
  60. Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. Towards an understanding of large language models in software engineering tasks. Empirical Software Engineering, 30(2):50, 2025. URL: https://doi.org/10.1007/S10664-024-10602-0.
  61. Shaohong Zhong, Andrea Scarinci, and Alice Cicirello. Natural language processing for systems engineering: Automatic generation of systems modelling language diagrams. Knowledge-Based Systems, 259:110071, 2023. URL: https://doi.org/10.1016/J.KNOSYS.2022.110071.
  62. Taohong Zhu, Lucas C. Cordeiro, and Youcheng Sun. Reqinone: A large language model-based agent for software requirements specification generation, 2025. URL: https://doi.org/10.48550/arXiv.2508.09648.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact