Document Open Access Logo

Why Do LLMs Fail at OCL Generation? A Graph Reasoning Perspective

Authors Hamza Attarwala , Moataz Chouchen , Mohammad Hamdaqa , Omar Alam

PDF

File

Thumbnail PDF
PDF
LIPIcs.ESEM.2026.48.pdf
  • Filesize: 0.73 MB
  • 21 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Software and its engineering → Unified Modeling Language (UML)
Keywords
  • Object Constraint Language (OCL)
  • Unified Modelling Language (UML)
  • Model-Driven Engineering (MDE)
  • Large Language Models (LLM)
  • Graph Reasoning

Metrics

Abstract

Background. Large Language Models (LLMs) are increasingly used to generate Object Constraint Language (OCL) constraints from natural language specifications and UML class diagrams. However, existing work mainly focuses on improving accuracy, with limited understanding of why these models fail. 

Aims. This study investigates factors associated with LLM failures in OCL generation by examining the task from a graph-reasoning perspective. 

Method. We conduct an empirical evaluation using the PathOCL dataset across six LLMs. We analyze the relationship between OCL correctness and UML structural properties (e.g., navigation depth and model complexity), lexical similarity, prompt ordering strategies, and graph-aware prompting. 

Results. We find that, within the evaluated dataset and models, OCL generation correctness is negatively associated with navigation depth and structural complexity. Lexical similarity provides limited explanatory power, while textual ordering is associated with differences in performance. Graph-based prompting yields partial improvements but does not eliminate errors involving structural navigation. 

Conclusions. The findings are consistent with structural demands being an important contributor to OCL generation difficulty, while not establishing graph reasoning as the sole or primary cause of failure.

Cite As Get BibTex

Hamza Attarwala, Moataz Chouchen, Mohammad Hamdaqa, and Omar Alam. Why Do LLMs Fail at OCL Generation? A Graph Reasoning Perspective. In 20th International Symposium on Empirical Software Engineering and Measurement (ESEM 2026). Leibniz International Proceedings in Informatics (LIPIcs), Volume 394, pp. 48:1-48:21, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2026) https://doi.org/10.4230/LIPIcs.ESEM.2026.48

Author Details

Hamza Attarwala
  • Polytechnique Montréal, Montréal, QC, Canada
Moataz Chouchen
  • Concordia University, Montréal, Canada
Mohammad Hamdaqa
  • Polytechnique Montréal, Montréal, QC, Canada
Omar Alam
  • Trent University, Peterborough, ON, Canada

Supplementary Materials

References

  1. Seif Abukhalaf, Mohammad Hamdaqa, and Foutse Khomh. On codex prompt engineering for ocl generation: an empirical study. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 148-157. IEEE, 2023. URL: https://doi.org/10.1109/MSR59073.2023.00033.
  2. Seif Abukhalaf, Mohammad Hamdaqa, and Foutse Khomh. Pathocl: path-based prompt augmentation for ocl generation with gpt-4. In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, pages 108-118, 2024. URL: https://doi.org/10.1145/3650105.3652290.
  3. Shaukat Ali, Muhammad Zohaib Iqbal, Andrea Arcuri, and Lionel C Briand. Generating test data from ocl constraints with search techniques. IEEE Transactions on Software Engineering, 39(10):1376-1402, 2013. URL: https://doi.org/10.1109/TSE.2013.17.
  4. Omar Raad Alsammak and Ashraf Abdulmunim Abdulmajeed. Effective analysis and classification of uml class diagrams: Literature review. In AIP Conference Proceedings, volume 3393, page 060011. AIP Publishing LLC, 2026. Google Scholar
  5. Anthropic. Claude sonnet 4.5. https://www.anthropic.com/news/claude-sonnet-4-5, 2025. Accessed: Nov. 13, 2025.
  6. Imran Sarwar Bajwa, Behzad Bordbar, and Mark G Lee. Ocl constraints generation from natural language specification. In 2010 14th IEEE International Enterprise Distributed Object Computing Conference, pages 204-213. IEEE, 2010. URL: https://doi.org/10.1109/EDOC.2010.33.
  7. Jordi Cabot, Robert Clarisó, and Daniel Riera. On the verification of uml/ocl class diagrams using constraint programming. Journal of Systems and Software, 93:1-23, 2014. URL: https://doi.org/10.1016/J.JSS.2014.03.023.
  8. Jordi Cabot, David Delgado, and Lola Burgueño. Combining ocl and natural language: a call for a community effort. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, pages 908-912, 2022. URL: https://doi.org/10.1145/3550356.3561542.
  9. Jordi Cabot and Martin Gogolla. Object constraint language (ocl): a definitive guide. In International school on formal methods for the design of computer, communication and software systems, pages 58-90. Springer, 2012. URL: https://doi.org/10.1007/978-3-642-30982-3_3.
  10. Jordi Cabot and Ernest Teniente. Constraint support in mda tools: A survey. In European Conference on Model Driven Architecture-Foundations and Applications, pages 256-267. Springer, 2006. URL: https://doi.org/10.1007/11787044_20.
  11. Ziwei Chai, Tianjie Zhang, Liang Wu, Kaiqiang Han, Xiaohai Hu, Xuanwen Huang, and Yang Yang. Graphllm: Boosting graph reasoning ability of large language model. IEEE Transactions on Big Data, 2025. Google Scholar
  12. Google DeepMind. Gemini 2.5 pro. https://deepmind.google/en/models/gemini/pro, 2025. Accessed: Nov. 13, 2025.
  13. DeepSeek-AI. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. URL: https://doi.org/10.48550/arXiv.2412.19437.
  14. Enrico Franconi, Alessandro Mosca, Xavier Oriol, Guillem Rull, and Ernest Teniente. Ocl fo: first-order expressive ocl constraints for efficient integrity checking. Software & Systems Modeling, 18(4):2655-2678, 2019. URL: https://doi.org/10.1007/S10270-018-0688-Z.
  15. Yuyao Ge, Shenghua Liu, Baolong Bi, Yiwei Wang, Lingrui Mei, Wenjie Feng, Lizhe Chen, and Xueqi Cheng. Can graph descriptive order affect solving graph problems with llms? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6404-6420, 2025. URL: https://doi.org/10.18653/V1/2025.ACL-LONG.321.
  16. Martin Gogolla, Fabian Büttner, and Mark Richters. Use: A uml-based specification environment for validating uml and ocl. Science of Computer Programming, 69(1-3):27-34, 2007. URL: https://doi.org/10.1016/J.SCICO.2007.01.013.
  17. Haoyu Han, Yaochen Xie, Hui Liu, Xianfeng Tang, Sreyashi Nag, William Headden, Yang Li, Chen Luo, Shuiwang Ji, Qi He, et al. Reasoning with graphs: Structuring implicit knowledge to enhance llms reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25698-25714, 2025. Google Scholar
  18. James W Hardin and Joseph M Hilbe. Generalized estimating equations. chapman and hall/CRC, 2002. Google Scholar
  19. Karsten Hölscher, Paul Ziemann, and Martin Gogolla. On translating uml models into graph transformation systems. Journal of Visual Languages & Computing, 17(1):78-105, 2006. URL: https://doi.org/10.1016/J.JVLC.2005.11.001.
  20. Zhongzhan Huang, Guoming Ling, Shanshan Zhong, Hefeng Wu, and Liang Lin. Minilongbench: the low-cost long context understanding benchmark for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11442-11460, 2025. URL: https://doi.org/10.18653/V1/2025.ACL-LONG.560.
  21. Kevin Chenhao Li, Vahid Zolfaghari, Nenad Petrovic, Fengjunjie Pan, and Alois Knoll. Optimizing retrieval augmented generation for object constraint language. arXiv preprint arXiv:2505.13129, 2025. URL: https://doi.org/10.48550/arXiv.2505.13129.
  22. Qianlong Li, Chen Huang, Shuai Li, Yuanxin Xiang, Deng Xiong, and Wenqiang Lei. Graphotter: Evolving llm-based graph reasoning for complex table question answering. In Proceedings of the 31st International Conference on Computational Linguistics, pages 5486-5506, 2025. URL: https://aclanthology.org/2025.coling-main.368/.
  23. Christoph Mayr-Dorn, Anmol Bilal, Cosmina Ratiu, and Alexander Egyed. Generating quality assurance constraints from natural language with llms. Journal of Software: Evolution and Process, 37(11):e70062, 2025. URL: https://doi.org/10.1002/SMR.70062.
  24. Meta AI. Introducing LLaMA 4: Advancing multimodal intelligence. https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 2024. Accessed: 2025-12-29.
  25. Object Management Group. Object constraint language (ocl), version 2.4. Technical report, OMG, 2014. URL: https://www.omg.org/spec/OCL/2.4/.
  26. OpenAI. Gpt-5. https://chat.openai.com, 2025. Accessed: Nov. 13, 2025.
  27. Lorenzo Pacchiardi, Lucy G Cheke, and José Hernández-Orallo. 100 instances is all you need: predicting the success of a new llm on unseen data by testing on a few instances. arXiv preprint arXiv:2409.03563, 2024. URL: https://doi.org/10.48550/arXiv.2409.03563.
  28. Fengjunjie Pan, Vahid Zolfaghari, Long Wen, Nenad Petrovic, Jianjie Lin, and Alois Knoll. Generative ai for ocl constraint generation: Dataset collection and llm fine-tuning. In 2024 IEEE International Symposium on Systems Engineering (ISSE), pages 1-8. IEEE, 2024. Google Scholar
  29. Miao Peng, Nuo Chen, Zongrui Suo, and Jia Li. Rewarding graph reasoning process makes llms more generalized reasoners. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 2257-2268, 2025. URL: https://doi.org/10.1145/3711896.3737109.
  30. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, pages 34303-34326, 2024. URL: https://proceedings.mlr.press/v235/maia-polo24a.html.
  31. Revanth Rameshkumar, Jimson Huang, Yunxin Sun, Fei Xia, and Abulhair Saparov. Reasoning models reason well, until they don't. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 936-956, 2025. URL: https://doi.org/10.18653/V1/2025.IJCNLP-LONG.52.
  32. Arnaud Roques. Plantuml. https://plantuml.com/, 2026. Accessed: 2026-05-09.
  33. Hanan Abdulwahab Siala and Kevin Lano. Using llms to extract ocl specifications from java and python programs: an empirical study. OCL '25, STAF, 2025. Google Scholar
  34. Mathias Soeken, Robert Wille, Mirco Kuhlmann, Martin Gogolla, and Rolf Drechsler. Verifying uml/ocl models using boolean satisfiability. In 2010 Design, Automation & Test in Europe Conference & Exhibition (DATE 2010), pages 1341-1344. IEEE, 2010. URL: https://doi.org/10.1109/DATE.2010.5457017.
  35. Li Tan, Zongyuan Yang, and Jinkui Xie. Ocl constraints automatic generation for uml class diagram. In 2010 IEEE International Conference on Software Engineering and Service Sciences, pages 392-395. IEEE, 2010. Google Scholar
  36. Anton Tsitsulin, Bryan Perozzi, Bahare Fatemi, and Jonathan J Halcrow. Graph reasoning with llms (greal). In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6424-6425, 2024. URL: https://doi.org/10.1145/3637528.3671448.
  37. Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36:30840-30861, 2023. Google Scholar
  38. xAI. Grok-4-fast. https://docs.x.ai/docs/models/grok-4-fast-reasoning, 2025. Accessed: Nov. 13, 2025.
  39. Yilong Yang, Yibo Liu, Tianshu Bao, Weiru Wang, Nan Niu, and Yongfeng Yin. Deepocl: A deep neural network for object constraint language generation from unrestricted nature language. CAAI Transactions on Intelligence Technology, 9(1):250-263, 2024. URL: https://doi.org/10.1049/CIT2.12207.
  40. Zike Yuan, Ming Liu, Hui Wang, and Bing Qin. Ma-gts: A multi-agent framework for solving complex graph problems in real-world applications. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19297-19315, 2025. Google Scholar
  41. Qifan Zhang, Nuo Chen, Zehua Li, Miao Peng, Jing Tang, and Jia Li. Improving LLMs` generalized reasoning abilities by graph problems. In Second Conference on Language Modeling, 2025. URL: https://openreview.net/forum?id=6vMRcaYbU7.
  42. Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. Can llm graph reasoning generalize beyond pattern memorization? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2289-2305, 2024. URL: https://doi.org/10.18653/V1/2024.FINDINGS-EMNLP.127.
  43. Paul Ziemann, Karsten Hölscher, and Martin Gogolla. From uml models to graph transformation systems. Electronic Notes in Theoretical Computer Science, 127(4):17-33, 2005. URL: https://doi.org/10.1016/J.ENTCS.2004.10.025.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2026 Schloss Dagstuhl – LZI GmbH Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact