Document Open Access Logo

Using Multi-Modal LLMs to Create Models for Fault Diagnosis (Short Paper)

Authors Silke Merkelbach , Alexander Diedrich , Anna Sztyber-Betley , Louise Travé-Massuyès , Elodie Chanthery , Oliver Niggemann , Roman Dumitrescu

PDF

File

Thumbnail PDF
PDF
OASIcs.DX.2024.31.pdf
  • Filesize: 0.74 MB
  • 15 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Knowledge representation and reasoning
Keywords
  • Fault Diagnosis
  • Large Language Models
  • LLMs
  • Physical Modelling
  • Process Industry
  • P&IDs

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Creating models that are usable for fault diagnosis is hard. This is especially true for cyber-physical systems that are subject to architectural changes and may need to be adapted to different product variants intermittently. We therefore can no longer rely on expert-defined and static models for many systems. Instead, models need to be created more cheaply and need to adapt to different circumstances. In this article we present a novel approach to create physical models for process industry systems using multi-modal large language models (i.e ChatGPT). We present a five-step prompting approach that uses a piping and instrumentation diagram (P&ID) and natural language prompts as its input. We show that we are able to generate physical models of three systems of a well-known benchmark. We further show that we are able to diagnose faults for all of these systems by using the Fault Diagnosis Toolbox. We found that while multi-modal large language models (MLLMs) are a promising method for automated model creation, they have significant drawbacks.

Cite As Get BibTex

Silke Merkelbach, Alexander Diedrich, Anna Sztyber-Betley, Louise Travé-Massuyès, Elodie Chanthery, Oliver Niggemann, and Roman Dumitrescu. Using Multi-Modal LLMs to Create Models for Fault Diagnosis (Short Paper). In 35th International Conference on Principles of Diagnosis and Resilient Systems (DX 2024). Open Access Series in Informatics (OASIcs), Volume 125, pp. 31:1-31:15, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024) https://doi.org/10.4230/OASIcs.DX.2024.31

Author Details

Silke Merkelbach
  • Fraunhofer IEM, Paderborn, Germany
Alexander Diedrich
  • Helmut-Schmidt-University, Hamburg, Germany
Anna Sztyber-Betley
  • Warsaw University of Technology, Poland
Louise Travé-Massuyès
  • LAAS-CNRS, University of Toulouse, France
Elodie Chanthery
  • LAAS-CNRS, INSA, University of Toulouse, France
Oliver Niggemann
  • Helmut-Schmidt-University, Hamburg, Germany
Roman Dumitrescu
  • Advanced Systems Engineering, Paderborn University, Germany

Funding

  • Merkelbach, Silke: This work is supported by the German Federal Ministry for Economic Affairs and Climate Action under grant 03EN4004B.
  • Sztyber-Betley, Anna: This project was partially funded by a research grant from the Scientific Council of the Discipline of Automation, Electronics, Electrical Engineering and Space Technologies of Warsaw University of Technology granted in 2023.
  • Travé-Massuyès, Louise: The work is supported by ANITI through the French "Investing for the Future – P3IA" program under the Grant agreement n^oANR-19-P3IA-0004.
  • Chanthery, Elodie: The work is supported by ANITI through the French "Investing for the Future – P3IA" program under the Grant agreement n^oANR-19-P3IA-0004.

Acknowledgements

This work has benefited from participation in Dagstuhl Seminar 24031 "Fusing Causality, Reasoning, and Learning for Fault Management and Diagnosis".

Supplementary Materials

References

  1. Luis A Aguirre. An introduction to nonlinear system identification. In Lectures on Nonlinear Dynamics, pages 133-154. Springer, 2023. Google Scholar
  2. Lukas Schulze Balhorn, Marc Caballero, and Artur M Schweidtmann. Toward autocorrection of chemical process flowsheets using large language models. arXiv preprint arXiv:2312.02873, 2023. URL: https://doi.org/10.48550/arXiv.2312.02873.
  3. Kaja Balzereit, Alexander Diedrich, Jonas Ginster, Stefan Windmann, and Oliver Niggemann. An ensemble of benchmarks for the evaluation of ai methods for fault handling in cpps. In 19th IEEE International Conference on Industrial Informatics, November 2021. Google Scholar
  4. J-Ph Cassar and M Staroswiecki. A structural approach for the design of failure detection and identification systems. IFAC Proceedings Volumes, 30(6):841-846, 1997. Google Scholar
  5. Cody James Christopher and Alban Grastien. Critical observations in model-based diagnosis. Artificial Intelligence, page 104116, 2024. URL: https://doi.org/10.1016/J.ARTINT.2024.104116.
  6. Deutsches Institut für Normung e.V. (DIN). DIN EN ISO 10628-2: Flow diagrams for process plants - part 2: Graphical symbols. DIN standard, DIN, 2012. Google Scholar
  7. Alexander Diedrich, Lukas Moddemann, and Oliver Niggemann. Learning system descriptions for cyber-physical systems. In Proceedings of 12th IFAC Symposium on Fault Detection, Supervision and Safety for Technical Processes, 2024. Google Scholar
  8. Marius-Constantin Dinu, Claudiu Leoveanu-Condrei, Markus Holzleitner, Werner Zellinger, and Sepp Hochreiter. Symbolicai: A framework for logic-based approaches combining generative models and solvers. arXiv preprint arXiv:2402.00854, 2024. URL: https://doi.org/10.48550/arXiv.2402.00854.
  9. Teresa Escobet, Anibal Bregon, Belarmino Pulido, and Vicenç Puig. Fault Diagnosis of Dynamic Systems. Springer, 2019. Google Scholar
  10. Erik Frisk, Mattias Krysander, and Daniel Jung. A toolbox for analysis and design of model based diagnosis systems for large scale models. IFAC-PapersOnLine, 50(1):3287-3293, 2017. Google Scholar
  11. Constantin Hildebrandt, Sebastian Törsleff, Birte Caesar, and Alexander Fay. Ontology building for cyber-physical systems: A domain expert-centric approach. In 2018 IEEE 14th international conference on automation science and engineering (CASE), pages 1079-1086. IEEE, 2018. URL: https://doi.org/10.1109/COASE.2018.8560465.
  12. Edwin Hirtreiter, Lukas Schulze Balhorn, and Artur M Schweidtmann. Toward automatic generation of control structures for process flow diagrams with large language models. AIChE Journal, 70(1):e18259, 2024. Google Scholar
  13. Sungmin Kang, Gabin An, and Shin Yoo. A preliminary evaluation of llm-based fault localization. arXiv preprint arXiv:2308.05487, 2023. URL: https://doi.org/10.48550/arXiv.2308.05487.
  14. Shota Kato, Chunpu Zhang, and Manabu Kano. Simple algorithm for judging equivalence of differential-algebraic equation systems. Scientific reports, 13(1):11534, 2023. Google Scholar
  15. Mattias Krysander, Jan Åslund, and Mattias Nyberg. An efficient algorithm for finding minimal overconstrained subsystems for model-based diagnosis. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 38(1):197-206, 2008. URL: https://doi.org/10.1109/TSMCA.2007.909555.
  16. Shouvik Mani, Michael A. Haddad, Dan Constantini, Willy Douhard, Qiwei Li, and Louis Poirier. Automatic Digitization of Engineering Diagrams using Deep Learning and Graph Search. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 673-679. IEEE, June 2020. URL: https://doi.org/10.1109/CVPRW50498.2020.00096.
  17. Silke Merkelbach, Alexander Diedrich, Sebastian Von Enzberg, Oliver Niggemann, and Roman Dumitrescu. Towards the generation of models for fault diagnosis of cps using vqa models. ML4CPS 2024 – Machine Learning for Cyber Physical Systems Conference, 2014. Google Scholar
  18. Lukas Moddemann, Henrik Sebastian Steude, Alexander Diedrich, and Oliver Niggemann. Discret2di - deep learning based discretization for model-based diagnosis. In Proceedings of 12th IFAC Symposium on Fault Detection, Supervision and Safety for Technical Processes, 2024. Google Scholar
  19. Oluwatosin Ogundare, Gustavo Quiros Araya, Ioannis Akrotirianakis, and Ankit Shukla. Resiliency analysis of llm generated models for industrial automation. arXiv preprint arXiv:2308.12129, 2023. URL: https://doi.org/10.48550/arXiv.2308.12129.
  20. OpenAI. Chatgpt 4 vision preview. version gpt-4-1106-vision-preview. url: https://platform.openai.com/docs/api-reference, 2023. Google Scholar
  21. LIU Peifeng, Lu Qian, Xingwei Zhao, and Bo Tao. Joint knowledge graph and large language model for fault diagnosis and its application in aviation assembly. IEEE Transactions on Industrial Informatics, 2024. Google Scholar
  22. CG Pérez-Zuniga, E Chanthery, L Travé-Massuyès, and J Sotomayor. Fault-driven structural diagnosis approach in a distributed context. IFAC-PapersOnLine, 50(1):14254-14259, 2017. Google Scholar
  23. Swantje Plambeck, Aaron Bracht, Nemanja Hranisavljevic, and Goerschwin Fey. Famos-fast model learning for hybrid cyber-physical systems using decision trees. In Proceedings of the 27th ACM International Conference on Hybrid Systems: Computation and Control, pages 1-10, 2024. URL: https://doi.org/10.1145/3641513.3650131.
  24. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. Llm-sr: Scientific equation discovery via programming with large language models. arXiv preprint arXiv:2404.18400, 2024. URL: https://doi.org/10.48550/arXiv.2404.18400.
  25. Arka Sinha, Johannes Bayer, and Syed Saqib Bukhari. Table localization and field value extraction in piping and instrumentation diagram images. In 2019 international conference on document analysis and recognition workshops (ICDARW), volume 1, pages 26-31. IEEE, 2019. URL: https://doi.org/10.1109/ICDARW.2019.00010.
  26. Louise Travé-Massuyès. Bridging control and artificial intelligence theories for diagnosis: A survey. Engineering Applications of Artificial Intelligence, 27:1-16, 2014. URL: https://doi.org/10.1016/J.ENGAPPAI.2013.09.018.
  27. Louise Travé-Massuyes, Teresa Escobet, and Xavier Olive. Diagnosability analysis based on component-supported analytical redundancy relations. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 36(6):1146-1160, 2006. URL: https://doi.org/10.1109/TSMCA.2006.878984.
  28. Yonghao Wu, Zheng Li, Jie M Zhang, Mike Papadakis, Mark Harman, and Yong Liu. Large language models in fault localisation. arXiv preprint arXiv:2308.15276, 2023. URL: https://doi.org/10.48550/arXiv.2308.15276.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact