Document Open Access Logo

The DX Competition 2025 and Its Benchmarks (DX Competition)

Authors Ingo Pill , Daniel Jung , Eldin Kurudzija , Anna Sztyber-Betley , Michał Syfert , Kai Dresia , Günther Waxenegger-Wilfing , Johan de Kleer

PDF

File

Thumbnail PDF
PDF
OASIcs.DX.2025.14.pdf
  • Filesize: 4.16 MB
  • 19 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Causal reasoning and diagnostics
Keywords
  • Diagnosis
  • Algorithms
  • Evaluation

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Fault diagnosis has been addressed in many research communities, leading to a variety of available fault diagnosis techniques. Deciding as a user which fault diagnosis methods are suitable for a specific application is thus a nontrivial task. Benchmarks can provide the community with a holistic understanding of the landscape of newly developed and available fault diagnosis methods when making this decision. After a long hiatus, we revived the DX Competition with three fault diagnosis benchmarks: SLIDe, LUMEN, and LiU-ICE. The purpose of the benchmarks is to inspire fault diagnosis research with challenging problems in cyber-physical systems relevant for industry. The benchmarks share a common code structure and we used similar performance metrics in order to simplify the adaptation of diagnosis system solutions to the different case studies.

Cite As Get BibTex

Ingo Pill, Daniel Jung, Eldin Kurudzija, Anna Sztyber-Betley, Michał Syfert, Kai Dresia, Günther Waxenegger-Wilfing, and Johan de Kleer. The DX Competition 2025 and Its Benchmarks (DX Competition). In 36th International Conference on Principles of Diagnosis and Resilient Systems (DX 2025). Open Access Series in Informatics (OASIcs), Volume 136, pp. 14:1-14:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.DX.2025.14

Author Details

Ingo Pill
  • Institute of Software Engineering and Artificial Intelligence, TU Graz, Austria
Daniel Jung
  • Department of Electrical Engineering, Linköping University, Sweden
Eldin Kurudzija
  • Institute of Space Propulsion, German Aerospace Center (DLR), Köln, Germany
Anna Sztyber-Betley
  • Warsaw University of Technology, Poland
Michał Syfert
  • Warsaw University of Technology, Poland
Kai Dresia
  • Institute of Space Propulsion, German Aerospace Center (DLR), Lampoldhausen, Germany
Günther Waxenegger-Wilfing
  • Institute of Space Propulsion, German Aerospace Center (DLR), Hardthausen am Kocher, Germany
  • Institute of Computer Science, University of Würzburg, Germany
Johan de Kleer
  • c-infinity, Mountain View, CA, USA

Acknowledgements

We would like to thank all of our colleagues who contributed to making the DX Competition 2025 happen and who worked with us on the benchmarks. This includes in particular Erik Frisk, Mattias Krysander, Tobias Lindell, Tobias Traudt, Jan Deeken, Justin Hardi, Stefan Schlechtriem, Michael Börner, Dmitry Suslov, Robson dos Santos Hahn, Sebastian Klein, Wolfgang Armbruster, Jan Haemisch, Christopher Groll, Max Axel Müller, and Vincent Bareiß.

Supplementary Materials

References

  1. J. L. Augustin and O. Niggemann. Graph Structural Residuals: A Learning Approach to Diagnosis, 2023. URL: https://doi.org/10.48550/arXiv.2308.06961.
  2. K. Balzereit, A. Diedrich, J. Ginster, S. Windmann, and O. Niggemann. An ensemble of benchmarks for the evaluation of AI methods for fault handling in CPPS. In 2021 IEEE 19th Int. Conf. on Industrial Informatics (INDIN), pages 1-6. IEEE, 2021. Google Scholar
  3. M. Bartyś, R. Patton, M. Syfert, S. de las Heras, and J. Quevedo. Introduction to the DAMADICS actuator FDI benchmark study. Control Engineering Practice, 14(6):577-596, 2006. Google Scholar
  4. T. W. Bickmore. Real-Time Sensor Data Validation. Contractor Report, NASA-CR-195295, 1994. Google Scholar
  5. P. Chatterjee, J. Campos, R. Abreu, and S. Roy. Augmenting Automated Spectrum Based Fault Localization for Multiple Faults. In 32nd Int. Joint Conf. on Artificial Intelligence (IJCAI-23), pages 3140-3148, August 2023. Google Scholar
  6. M. Davidson and J. Stephens. Advanced health management system for the space shuttle main engine. In 40th AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit, 2004. Google Scholar
  7. J. de Kleer and B. C. Williams. Diagnosis with Behavioral Modes. In 11th Int. Joint Conf. on Artificial Intelligence (IJCAI), pages 1324-1330, 1989. Google Scholar
  8. J. Downs and E. Vogel. A plant-wide industrial process control problem. Computers & chemical engineering, 17(3):245-255, 1993. Google Scholar
  9. K. Dresia, M. Börner, W. Armbruster, S. Klein, T. Traudt, D. Suslov, J. Hardi, G. Waxenegger-Wilfing, and J. C. Deeken. Design and control challenges for the LUMEN LOX/LNG expander-bleed rocket engine. In 34th Int. Symposium on Space Technology and Science (ISTS), 2023. Google Scholar
  10. K. Dresia, T. Traudt, M. Börner, D. Suslov, W. Armbruster, R. dos Santos Hahn, E. Kurudzija, C. Groll, M. A. Müller, S. Klein, J. Hämisch, J. Deeken, J. Hardi, and S. Schlechtriem. Hot-fire testing and system analysis of the LUMEN liquid upper stage demonstrator engine. In 3rd Int. Conf. on Flight Vehicles, Aerothermodynamics and Re-entry (FAR), 2025. Google Scholar
  11. J. Ehrhardt, M. Ramonat, R. Heesch, K. Balzereit, A. Diedrich, and O. Niggemann. An AI benchmark for diagnosis, reconfiguration & planning. In 2022 IEEE 27th Int. Conf. on Emerging Technologies and Factory Automation (ETFA), pages 1-8, 2022. Google Scholar
  12. L. Eriksson. Modeling and control of turbocharged SI and DI engines. Oil & Gas Science and Technology-Revue de l'IFP, 62(4):523-538, 2007. Google Scholar
  13. A. Feldman, J. de Kleer, T. Kurtoglu, S. Narasimhan, S. Poll, D. Garcia, L. Kuhn, and A. van Gemund. The diagnostic competitions. AI Magazine, 35(2):49-54, 2014. URL: https://doi.org/10.1609/AIMAG.V35I2.2532.
  14. A. Feldman, G. Provan, and A. van Gemund. The Lydia approach to combinational model-based diagnosis. Proc. Int. Workshop on Principles of Diagnosis, 9:403-408, 2009. Google Scholar
  15. E. Frisk, M. Krysander, and D. Jung. A toolbox for analysis and design of model based diagnosis systems for large scale models. IFAC-PapersOnLine, 50(1):3287-3293, 2017. Google Scholar
  16. A. Grastien and P. Kan-John. Wizards of Oz description of the 2009 DXC entry. Proc. Int. Workshop on Principles of Diagnosis, 9:409-413, 2009. Google Scholar
  17. D. Jung, E. Frisk, and M. Krysander. The LiU-ICE benchmark-an industrial fault diagnosis case study. arXiv preprint, 2024. URL: https://arxiv.org/abs/2408.13269.
  18. D. Jung, E. Frisk, M.s Krysander, A. Sztyber-Betley, F. Corrini, A. Arici, N. Anselmi, M. Mazzoleni, J. Xu, S. Mo, Z. Xu, C. Yang, Z. Du, H. Safaeipour, M. Forouzanfar, V. Mirahi, A. Pinnarelli, V. Puig, Q. Deng, Y. Liu, J. Liu, H. Ke, W. Zhu, S. Merkelbach, M. Ahang, and H. Najjaran. A fault diagnosis benchmark of technical systems with incomplete data - six solutions. Control Engineering Practice, 2025. to appear. Google Scholar
  19. D. Jung, H. Khorasgani, E. Frisk, M. Krysander, and G. Biswas. Analysis of fault isolation assumptions when comparing model-based design approaches of diagnosis systems. IFAC-PapersOnLine, 48(21):1289-1296, 2015. Google Scholar
  20. D. Jung and M. Krysander. Assumption-based Design of Hybrid Diagnosis Systems: Analyzing Model-based and Data-driven Principles. In Annual Conf. of the PHM Society, 2024. Google Scholar
  21. T. Kurtoglu, S. Narasimhan, S. Poll, D. Garcia, L. Kuhn, J. de Kleer, and A. Feldman. Second international diagnostic competition (dxc’10), 2010. Google Scholar
  22. T. Kurtoglu, S. Narasimhan, S. Poll, D. Garcia, L. Kuhn, J. de Kleer, A. van Gemund, and A. Feldman. First international diagnosis competition-DXC’09. Proc. Int. Workshop on Principles of Diagnosis DX, 9:383-396, 2009. Google Scholar
  23. E. Kurudzija, K. Dresia, J. Martin, T. Traudt, J. C. Deeken, and G. Waxenegger-Wilfing. Virtual sensing for fault detection within the LUMEN fuel turbopump test campaign. In 9th Edition of the Space Propulsion Conference, Glasgow, Scotland., May 2024. Google Scholar
  24. K. Lunde, R. Lunde, and B. Münker. Model-based failure analysis with rodon. In ECAI 2006, pages 647-651. IOS Press, 2006. Google Scholar
  25. I. Matei, M. Zhenirovskyy, J. de Kleer, and A. Feldman. Classification-based Diagnosis Using Synthetic Data from Uncertain Models. Annual Conference of the PHM Society, 10(1), 2018. Google Scholar
  26. O. J. Mengshoel. Designing resource-bounded reasoners using bayesian networks: System health monitoring and diagnosis. In Proc. of the 18th Int. Workshop on Principles of Diagnosis (dx-07), pages 330-337, 2007. Google Scholar
  27. A. Metodi, R. Stern, M. Kalech, and M. Codish. Compiling Model-Based Diagnosis to Boolean Satisfaction. In 26th AAAI Conf. on Artificial Intelligence, pages 793-799, 2012. Google Scholar
  28. L. Moddemann, H. Steude, A. Diedrich, I. Pill, and O. Niggemann. Extracting Knowledge using Machine Learning for Anomaly Detection and Root-Cause Diagnosis. In 29th IEEE Int. Conf. on Emerging Technologies and Factory Automation (ETFA), 2024. to appear. Google Scholar
  29. S. Narasimhan and L. Brownston. HyDE - A general framework for stochastic and hybrid modelbased diagnosis. Proc. Int. Workshop on Principles of Diagnosis, 7:162-169, 2007. Google Scholar
  30. I. Nica, I. Pill, T. Quaritsch, and F. Wotawa. The Route to Success - A Performance Comparison of Diagnosis Algorithms. In 23rd International Joint Conference on Artificial Intelligence, pages 1039-1045, 2013. Google Scholar
  31. P. Odgaard and J. Stoustrup. Results of a wind turbine FDI competition. IFAC Proceedings Volumes, 45(20):102-107, 2012. Google Scholar
  32. P. Odgaard, J. Stoustrup, and M. Kinnaert. Fault-tolerant control of wind turbines: A benchmark model. IEEE Transactions on control systems Technology, 21(4):1168-1182, 2013. URL: https://doi.org/10.1109/TCST.2013.2259235.
  33. I. Pill and T. Quaritsch. Behavioral diagnosis of LTL specifications at operator level. In 23rd Int. Joint Conf. on Artificial Intelligence, pages 1053-1059, 2013. Google Scholar
  34. I. Pill and T. Quaritsch. RC-Tree: A variant avoiding all the redundancy in Reiter’s minimal hitting set algorithm. In IEEE Int. Symp. on Software Reliability Engineering Workshops (ISSREW), pages 78-84, 2015. Google Scholar
  35. I. Pill, T. Quaritsch, and F. Wotawa. Parse tree structure in LTL requirements diagnosis. In 2015 IEEE Int. Symp. on Software Reliability Engineering Workshops, pages 100-107, 2015. Google Scholar
  36. I. Pill and F. Wotawa. Spectrum-Based Fault Localization for Logic-Based Reasoning. In 2018 IEEE Int. Symposium on Software Reliability Engineering Workshops (ISSREW), pages 192-199, 2018. Google Scholar
  37. I. Pill and F. Wotawa. Exploiting observations from combinatorial testing for diagnostic reasoning. In 30th Int. Workshop on Principles of Diagnosis, 2019. Google Scholar
  38. I. Pill and F. Wotawa. Extending Automated FLTL Test Oracles with Diagnostic Support. In IEEE Int. Symp.on Software Reliability Engineering Workshops, pages 354-361, 2019. Google Scholar
  39. I. Pill and F. Wotawa. Computing Multi-Scenario Diagnoses. In 31st Int. Workshop on Principles of Diagnosis, 2020. Google Scholar
  40. S. Poll, J. de Kleer, R. Abreau, M. Daigle, A. Feldman, D. Garcia, and A Sweet. Third international diagnostics competition-DXC’11. In Proc. of the 22nd Int. Workshop on Principles of Diagnosis, pages 267-278, 2011. Google Scholar
  41. S. Poll, A. Patterson-Hine, J. Camisa, D. Garcia, D. Hall, C. Lee, O. Mengshoel, C. Neukom, D. Nishikawa, J. Ossenfort, et al. Advanced diagnostics and prognostics testbed. In DX Int. Workshop on Principles of Diagnosis, pages 178-185, 2007. Google Scholar
  42. R. Reiter. A Theory of Diagnosis from First Principles. Art. Intelligence, 32(1):57-95, 1987. URL: https://doi.org/10.1016/0004-3702(87)90062-2.
  43. I. Roychoudhury, G. Biswas, and X. Koutsoukos. Designing distributed diagnosers for complex continuous systems. IEEE Trans. on Automation Science and Engineering, 6(2):277-290, 2009. URL: https://doi.org/10.1109/TASE.2008.2009094.
  44. P. Simontacchia, R. Blasi, Edeline E., S. Sagnier, , A. Espinosa-Ramos, J. Breteau, and P. Altenhöfer. PROMETHEUS: Precursor of new low-cost rocket engine family. In Proc. of the 8th European Conf. for Aeronatuics and Space Sciences (EUCASS), 2019. Google Scholar
  45. W. A. Smith and R. B. Randall. Rolling element bearing diagnostics using the case western reserve university data: A benchmark study. Mechanical Systems and Signal Processing, 64-65:100-131, 2015. Google Scholar
  46. SpaceX. Falcon User’s Guide. Space Exploration Technologies Corp., September 2021. (visited on 05/09/2025). URL: https://www.spacex.com/media/falcon-users-guide-2021-09.pdf.
  47. A. Sweet, A. Feldman, S. Narasimhan, M. Daigle, and S. Poll. Fourth international diagnostic competition-DXC’13. In Proc. of the 24th Int. Workshop on Principles of Diagnosis, pages 224-229, 2013. Google Scholar
  48. M. Syfert. Cyber-attack scenarios for super-heaters system, March 2023. URL: https://doi.org/10.5281/zenodo.7612269.
  49. M. Syfert, P. Wnuk, A. Sztyber-Betley, and M. Pobocha. The Model of Ongoing Diagnosis of Process Faults and Detection of Cybernetic Attacks for a Steam Line. Acta Physica Polonica A, 146(4):438, 2024. Google Scholar
  50. A. Sztyber, E. Chanthery, and L. Travé-Massuyès. Benchmark for fault diagnosis of water distribution network. In 34rd Int. Workshop on Principle of Diagnosis - DX 2023, pages 1-8, 2023. Google Scholar
  51. A. Sztyber, E. Chanthery, L. Travé-Massuyès, and C. G. Pérez-Zuñiga. Water network benchmarks for structural analysis algorithms in fault diagnosis. In 33rd Int. Workshop on Principle of Diagnosis - DX 2022, 2022. Google Scholar
  52. A. Sztyber, Z. Górecka, J. M. Kościelny, and M. Syfert. Controller modelling as a tool for cyber-attacks detection. In Z. Kowalczuk, editor, Intelligent and Safe Computer Systems in Control and Diagnostics, pages 100-111, Cham, 2023. Springer International Publishing. Google Scholar
  53. A. Sztyber-Betley, M. Syfert, J. M. Kościelny, and Z. Górecka. Controller Cyber-Attack Detection and Isolation. Sensors, 23(5), 2023. URL: https://doi.org/10.3390/S23052778.
  54. A. E. Tischer and R. C. Glover. Studies and Analyses of the Space Shuttle Main Engine. Contractor Report, NASA-CR-183593, 1987. Google Scholar
  55. T. Traudt, W. Armbruster, C.r Groll, R. H. Dos Santos Hahn, K. Dresia, M. Börner, S. Klein, D. Suslov, E. Kurudzija, J. Haemisch, M. A. Müller, J. C. Deeken, J. Hardi, and S. Schlechtriem. LUMEN, the test bed for rocket engine components: Results of the acceptance tests and overview on the engine test preparation. In 9th Edition of the Space Propulsion Conference, May 2024. Google Scholar
  56. D. Vranješ, J. Ehrhardt, R. Heesch, L. Moddemann, H. S. Steude, and O. Niggemann. Design Principles for Falsifiable, Replicable and Reproducible Empirical Machine Learning Research. In I. Pill, A. Natan, and F. Wotawa, editors, 35th Int. Conf. on Principles of Diagnosis and Resilient Systems (DX 2024), volume 125 of Open Access Series in Informatics (OASIcs), pages 7:1-7:13. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2024. URL: https://doi.org/10.4230/OASICS.DX.2024.7.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact