Document Open Access Logo

Assessing Diagnosis Algorithms: Of Sampling, Baselines, Metrics and Oracles

Authors Ingo Pill , Johan de Kleer

PDF

File

Thumbnail PDF
PDF
OASIcs.DX.2025.5.pdf
  • Filesize: 0.65 MB
  • 19 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Computing methodologies → Causal reasoning and diagnostics
Keywords
  • Model-based Diagnosis
  • Diagnosis
  • Algorithms

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

Assessing and comparing diagnosis algorithms is a surprisingly complex challenge. We have to make decisions ranging from identifying the implications of the chosen baseline, via defining and ensuring a representative sampling strategy, to the choice of metric best suited to capture the computational, probing, or repair costs as well as the deviations from the baseline. We discuss several aspects of the overall challenge, identify related issues, and evaluate a special economic metric.

Cite As Get BibTex

Ingo Pill and Johan de Kleer. Assessing Diagnosis Algorithms: Of Sampling, Baselines, Metrics and Oracles. In 36th International Conference on Principles of Diagnosis and Resilient Systems (DX 2025). Open Access Series in Informatics (OASIcs), Volume 136, pp. 5:1-5:19, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.DX.2025.5

Author Details

Ingo Pill
  • Institute of Software Engineering and Artificial Intelligence, TU Graz, Austria
Johan de Kleer
  • c-infinity, Mountain View, CA, USA

References

  1. R. Abreu and A. J. C. van Gemund. A low-cost approximate minimal hitting set algorithm and its application to model-based diagnosis. In 8th Symposium on Abstraction, Reformulation, and Approximation, SARA, 2009. URL: http://www.aaai.org/ocs/index.php/SARA/SARA09/paper/view/834.
  2. J. Biteus, M. Nyberg, and E. Frisk. An algorithm for computing the diagnoses with minimal cardinality in a distributed system. Engineering Applications of Artificial Intelligence, 21(2): 269-276, 2008. URL: https://doi.org/10.1016/j.engappai.2007.03.006.
  3. A. Boussif and M. Ghazel. Diagnosability analysis of input/output discrete-event systems using model-checking. IFAC-PapersOnLine, 48(7):71-78, 2015. URL: https://doi.org/10.1016/j.ifacol.2015.06.475.
  4. J. de Kleer and B. C. Williams. Diagnosing multiple faults. Artificial Intelligence, 32(1):97-130, 1987. URL: https://doi.org/10.1016/0004-3702(87)90063-4.
  5. J. de Kleer and B. C. Williams. Diagnosis with Behavioral Modes. In 11th International Joint Conference on Artificial Intelligence (IJCAI), pages 1324-1330, 1989. Google Scholar
  6. Johan de Kleer. Hitting set algorithms for model-based diagnosis. In 22nd International Workshop on Principles of Diagnosis (DX’11), 2011. Google Scholar
  7. P. Delgado-Pérez and F. Chicano. An experimental and practical study on the equivalent mutant connection: An evolutionary approach. Information and Software Technology, 124:106317, 2020. URL: https://doi.org/10.1016/j.infsof.2020.106317.
  8. A. Feldman, T. Kurtoglu, S. Narasimhan, S. Poll, D. Garcia, J. de Kleer, L. Kuhn, and A. van Gemund. Empirical evaluation of diagnostic algorithm performance using a generic framework. International Journal of Prognostics and Health Management, pages 1-28, 2010. Google Scholar
  9. A. Feldman, I. Pill, F. Wotawa, I. Matei, and J. de Kleer. Efficient model-based diagnosis of sequential circuits. In 34th AAAI Conference on Artificial Intelligence (AAAI'20), pages 2814-2821. AAAI Press, 2020. URL: https://doi.org/10.1609/AAAI.V34I03.5670.
  10. R. Greiner, B. A. Smith, and R. W. Wilkerson. A Correction to the Algorithm in Reiter’s Theory of Diagnosis. Artificial Intelligence, 41(1):79-88, 1989. URL: https://doi.org/10.1016/0004-3702(89)90079-9.
  11. A. Hou. A theory of measurement in diagnosis from first principles. Artificial Intelligence, 65(2):281-328, February 1994. URL: https://doi.org/10.1016/0004-3702(94)90019-1.
  12. I. Matei, M. Zhenirovskyy, J. de Kleer, and A. Feldman. Classification-based Diagnosis Using Synthetic Data from Uncertain Models. Annual Conference of the PHM Society, 10(1), 2018. Google Scholar
  13. A. Metodi, R. Stern, M. Kalech, and M. Codish. Compiling Model-Based Diagnosis to Boolean Satisfaction. In 26th AAAI Conference on Artificial Intelligence, pages 793-799, 2012. Google Scholar
  14. E. Muškardin, I. Pill, and F. Wotawa. CatIO - A Framework for Model-Based Diagnosis of Cyber-Physical Systems. In D. Helic, G. Leitner, M. Stettinger, A. Felfernig, and Z. W. Raś, editors, Foundations of Intelligent Systems, pages 267-276, 2020. Google Scholar
  15. I. Nica, I. Pill, T. Quaritsch, and F. Wotawa. The Route to Success - A Performance Comparison of Diagnosis Algorithms. In 23rd International Joint Conference on Artificial Intelligence, pages 1039-1045, 2013. Google Scholar
  16. I.-D. Nica and F. Wotawa. ConDiag - Computing minimal diagnoses using a constraint solver. In 23rd International Workshop on Principles of Diagnosis, 2012. Google Scholar
  17. I. Pill and J. de Kleer. Challenges for Model-Based Diagnosis. In 35th International Conference on Principles of Diagnosis and Resilient Systems (DX 2024), volume 125 of Open Access Series in Informatics (OASIcs), pages 6:1-6:20, 2024. URL: https://doi.org/10.4230/OASIcs.DX.2024.6.
  18. I. Pill and T. Quaritsch. Behavioral diagnosis of LTL specifications at operator level. In 23rd Int. Joint Conf. on Artificial Intelligence, pages 1053-1059, 2013. Google Scholar
  19. I. Pill and T. Quaritsch. RC-Tree: A variant avoiding all the redundancy in Reiter’s minimal hitting set algorithm. In IEEE Int. Symp. on Software Reliability Engineering Workshops (ISSREW), pages 78-84, 2015. Google Scholar
  20. I. Pill, I. Rubil, F. Wotawa, and M. Nica. SIMULTATE: A Toolset for Fault Injection and Mutation Testing of Simulink Models. In IEEE 9th International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 168-173, 2016. Google Scholar
  21. I. Pill and F. Wotawa. Spectrum-Based Fault Localization for Logic-Based Reasoning . In 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pages 192-199, 2018. URL: https://doi.org/10.1109/ISSREW.2018.00006.
  22. I. Pill and F. Wotawa. On Using an I/O Model for Creating an Abductive Diagnosis Model via Combinatorial Exploration, Fault Injection, and Simulation. In 29th International Workshop on Principles of Diagnosis (DX’18), 2018. Google Scholar
  23. I. Pill and F. Wotawa. Exploiting observations from combinatorial testing for diagnostic reasoning. In 30th Int. Workshop on Principles of Diagnosis, 2019. Google Scholar
  24. I. Pill and F. Wotawa. Computing Multi-Scenario Diagnoses. In 31st International Workshop on Principles of Diagnosis, DX ; Conference date: 26-09-2020, 2020. URL: http://dx-2020.org/.
  25. R. Reiter. A Theory of Diagnosis from First Principles. Artificial Intelligence, 32(1):57-95, 1987. URL: https://doi.org/10.1016/0004-3702(87)90062-2.
  26. L. Ye, P. Dague, D. Longuet, L. B. Briones, and A. Madalinski. Fault manifestability verification for discrete event systems. In 22nd European Conf. on Artificial Intelligence, pages 1718-1719, 2016. URL: https://doi.org/10.3233/978-1-61499-672-9-1718.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact