Document Open Access Logo

FLEX: Fault Localization and Explanation Using Open-Source Large Language Models in Powertrain Systems (Short Paper)

Authors Herbert Muehlburger , Franz Wotawa

PDF
Thumbnail PDF

File

OASIcs.DX.2024.25.pdf
  • Filesize: 3.34 MB
  • 14 pages

Document Identifiers

Author Details

Herbert Muehlburger
  • Institute of Software Technology, Graz University of Technology, Austria
Franz Wotawa
  • Institute of Software Technology, Graz University of Technology, Austria

Funding

This study was conducted within the framework of the project ARCHIMEDES, receiving funding from the Key Digital Technologies Joint Undertaking (KDT JU) - the Public-Private Partnership for research, development, and innovation under Horizon Europe – and National Authorities under Grant Agreement No 101112295 funded by the European Union and the FFG under Grant Agreement No FO999899377.

Cite As Get BibTex

Herbert Muehlburger and Franz Wotawa. FLEX: Fault Localization and Explanation Using Open-Source Large Language Models in Powertrain Systems (Short Paper). In 35th International Conference on Principles of Diagnosis and Resilient Systems (DX 2024). Open Access Series in Informatics (OASIcs), Volume 125, pp. 25:1-25:14, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2024) https://doi.org/10.4230/OASIcs.DX.2024.25

Abstract

Cyber-physical systems (CPS) are critical to modern infrastructure, but are vulnerable to faults and anomalies that threaten their operational safety. In this work, we evaluate the use of open-source Large Language Models (LLMs), such as Mistral 7B, Llama3.1:8b-instruct-fp16, and others to detect anomalies in two distinct datasets: battery management and powertrain systems. Our methodology utilises retrieval-augmented generation (RAG) techniques, incorporating a novel two-step process where LLMs first infer operational rules from normal behavior before applying these rules for fault detection. During the experiments, we found that the original prompt design yielded strong results for the battery dataset but required modification for the powertrain dataset to improve performance. The adjusted prompt, which emphasises rule inference, significantly improved anomaly detection for the powertrain dataset. Experimental results show that models like Mistral 7B achieved F1-scores up to 0.99, while Llama3.1:8b-instruct-fp16 and Gemma 2 reached perfect F1-scores of 1.0 in complex scenarios. These findings demonstrate the impact of effective prompt design and rule inference in improving LLM-based fault detection for CPS, contributing to increased operational resilience.

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software testing and debugging
  • Software and its engineering → Software fault tolerance
  • Computing methodologies → Machine learning
  • Software and its engineering
  • Computing methodologies → Natural language processing
Keywords
  • Fault detection
  • anomaly detection
  • powertrain systems
  • large language models
  • open-source LLMs

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Supplementary Materials

References

  1. Marah Abdin, Jyoti Aneja, and Hany Awadalla. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, August 2024. URL: https://doi.org/10.48550/arXiv.2404.14219.
  2. Abhaya Abhaya and Bidyut Kr. Patra. An efficient method for autoencoder based outlier detection. Expert Systems with Applications, page 118904, September 2022. URL: https://doi.org/10.1016/j.eswa.2022.118904.
  3. Mistral AI-Team. Mistral NeMo. https://mistral.ai/news/mistral-nemo/, July 2024. Google Scholar
  4. Joseph Osedome Boi-Ukeme. A Robust Discrete Event Method for the Design of Cyber-Physical Systems, January 2023. Google Scholar
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, July 2020. https://arxiv.org/abs/2005.14165, URL: https://doi.org/10.48550/arXiv.2005.14165.
  6. Jaime Carbonell and Jade Goldstein. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 335-336, Melbourne Australia, August 1998. ACM. URL: https://doi.org/10.1145/290941.291025.
  7. Marco Cook, Cory Paterson, Angelos K. Marnerides, and Dimitrios Pezaros. Anomaly Diagnosis in Cyber-Physical Systems. In ICC 2022 - IEEE International Conference on Communications, pages 5445-5450, May 2022. URL: https://doi.org/10.1109/ICC45855.2022.9838968.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. https://arxiv.org/abs/1810.04805, URL: https://doi.org/10.48550/arXiv.1810.04805.
  9. Abhimanyu Dubey, Abhinav Jauhri, and Pandey. Llama 3 Herd of Models, August 2024. URL: https://arxiv.org/abs/2407.21783.
  10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, October 2023. URL: https://doi.org/10.48550/arXiv.2310.06825.
  11. Sudha Krishnamurthy, Soumik Sarkar, and Ashutosh Tewari. Scalable Anomaly Detection and Isolation in Cyber-Physical Systems Using Bayesian Networks. In ASME 2014 Dynamic Systems and Control Conference. American Society of Mechanical Engineers Digital Collection, December 2014. URL: https://doi.org/10.1115/DSCC2014-6365.
  12. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, M. Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. ArXiv, May 2020. Google Scholar
  13. Herbert Muehlburger and Franz Wotawa. A Passive Testing Approach using a Semi-Supervised Intrusion Detection Model for SCADA Network Traffic. In 2022 IEEE International Conference On Artificial Intelligence Testing (AITest), pages 42-47, August 2022. URL: https://doi.org/10.1109/AITest55621.2022.00015.
  14. Herbert Muehlburger and Franz Wotawa. FaultLines: Evaluating the efficacy of open-source large language models for fault detection in cyber-physical systems. In 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), Shanghai, China, July 2024. IEEE Computer Society. Google Scholar
  15. Herbert Muehlburger and Franz Wotawa. Making Systems Fail-Aware: A Semi-Supervised Machine Learning Approach for Identifying Failures by Learning the Correct Behavior of a System. IFAC-PapersOnLine, 58(4):7-12, January 2024. URL: https://doi.org/10.1016/j.ifacol.2024.07.185.
  16. Herbert Mühlburger and Franz Wotawa. muehlburger/dx-2024-flex. Software, (visited on 2024-11-14). URL: https://github.com/muehlburger/dx-2024-flex
    Software Heritage Logo archived version
    full metadata available at: https://doi.org/10.4230/artifacts.22522
  17. Jonathan Pan, Swee Liang Wong, and Yidi Yuan. RAGLog: Log Anomaly Detection using Retrieval Augmented Generation, November 2023. URL: https://doi.org/10.48550/arXiv.2311.05261.
  18. Vasso Reppa, Marios M. Polycarpou, and Christos G. Panayiotou. Distributed Sensor Fault Diagnosis for a Network of Interconnected Cyberphysical Systems. IEEE Transactions on Control of Network Systems, 2(1):11-23, March 2015. URL: https://doi.org/10.1109/TCNS.2014.2367362.
  19. Naeem Seliya, Taghi M. Khoshgoftaar, and Jason Van Hulse. A Study on the Relationships of Classifier Performance Metrics. In 2009 21st IEEE International Conference on Tools with Artificial Intelligence, pages 59-66, November 2009. URL: https://doi.org/10.1109/ICTAI.2009.25.
  20. Gemma Team, Morgane Riviere, and Shreya Pathak. Gemma 2: Improving Open Language Models at a Practical Size, August 2024. URL: https://doi.org/10.48550/arXiv.2408.00118.
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. arXiv:1706.03762 [cs], December 2017. URL: https://arxiv.org/abs/1706.03762.
  22. Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in Time Series: A Survey, May 2023. URL: https://arxiv.org/abs/2202.07125.
  23. Franz Wotawa, David Kaufmann, Adil Amukhtar, Iulia Nica, Florian Klück, Hermann Felbinger, Petr Blaha, Matus Kozovsky, Zdenek Havranek, and Martin Dosedel. Real-Time Predictive Maintenance - Model-Based, Simulation-Based and Machine Learning Based Diagnosis, pages 63-81. River Publishers, New York, 1 edition, September 2022. URL: https://doi.org/10.1201/9781003337232-7.
  24. Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large Language Models can Learn Rules, October 2023. URL: https://doi.org/10.48550/arXiv.2310.07064.
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2024 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact