Document Open Access Logo

New Fault Domains for Conformance Testing of Finite State Machines

Authors Frits Vaandrager , Ivo Melse

PDF

File

Thumbnail PDF
PDF
LIPIcs.CONCUR.2025.34.pdf
  • Filesize: 1.15 MB
  • 22 pages

Document Identifiers

Related Versions

Subject Classification

ACM Subject Classification
  • Theory of computation
Keywords
  • conformance testing
  • finite state machines
  • Mealy machines
  • apartness
  • observation tree
  • fault domains
  • k-A-complete test suites

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads
    0
    Metadata Views

Abstract

A fault domain reflects a tester’s assumptions about faults that may occur in an implementation and that need to be detected during testing. A fault domain that has been widely studied in the literature on black-box conformance testing is the class of finite state machines (FSMs) with at most m states. Numerous strategies for generating test suites have been proposed that guarantee fault coverage for this class. These so-called m-complete test suites grow exponentially in m-n, where n is the number of states of the specification, so one can only run them for small values of m-n. But the assumption that m-n is small is not realistic in practice. In his seminal paper from 1964, Hennie raised the challenge to design checking experiments in which the number of states may increase appreciably. In order to solve this long-standing open problem, we propose (much larger) fault domains that capture the assumption that all states in an implementation can be reached by first performing a sequence from some set A (typically a state cover for the specification), followed by k arbitrary inputs, for some small k. The number of states of FSMs in these fault domains grows exponentially in k. We present a sufficient condition for k-A-completeness of test suites with respect to these fault domains. Our condition implies k-A-completeness of two prominent m-complete test suite generation strategies, the Wp and HSI methods. Thus these strategies are complete for much larger fault domains than those for which they were originally designed, and thereby solve Hennie’s challenge. We show that three other prominent m-complete methods (H, SPY and SPYH) do not always generate k-A-complete test suites.

Cite As Get BibTex

Frits Vaandrager and Ivo Melse. New Fault Domains for Conformance Testing of Finite State Machines. In 36th International Conference on Concurrency Theory (CONCUR 2025). Leibniz International Proceedings in Informatics (LIPIcs), Volume 348, pp. 34:1-34:22, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/LIPIcs.CONCUR.2025.34

Author Details

Frits Vaandrager
  • Radboud University, Nijmegen, The Netherlands
Ivo Melse
  • Radboud University, Nijmegen, The Netherlands

Funding

Supported by NWO project OCENW.M.23.155, Evidence-Driven Black-Box Checking (EVI).

Acknowledgements

This article grew out of earlier work of the first author [Vaandrager, 2024; Frits W. Vaandrager et al., 2022] and the bachelor thesis of the second author [Melse, 2024]. An early version was made available on arXiv [Frits W. Vaandrager et al., 2024]. Many thanks to Kirill Bogdanov, Paul Fiterău-Broştean, Dennis Hendriks, Bálint Kocsis, Bram Pellen, Jurriaan Rot, and the anonymous reviewers for their insightfull feedback on earlier versions.

References

  1. D. Angluin. Learning regular sets from queries and counterexamples. Inf. Comput., 75(2):87-106, 1987. URL: https://doi.org/10.1016/0890-5401(87)90052-6.
  2. W. Y. L. Chan, C. T. Vuong, and M. R. Otp. An improved protocol test generation procedure based on UIOs. In Proceedings of the ACM Symposium on Communications Architectures & Protocols, September 19-22, 1989, SIGCOMM '89, pages 283-294, New York, NY, USA, 1989. Association for Computing Machinery. URL: https://doi.org/10.1145/75246.75274.
  3. T.S. Chow. Testing software design modeled by finite-state machines. IEEE Transactions on Software Engineering, 4(3):178-187, 1978. URL: https://doi.org/10.1109/TSE.1978.231496.
  4. Adenilso da Silva Simão, Alexandre Petrenko, and Nina Yevtushenko. Generating reduced tests for FSMs with extra states. In Manuel Núñez, Paul Baker, and Mercedes G. Merayo, editors, Testing of Software and Communication Systems, 21st IFIP WG 6.1 International Conference, TESTCOM 2009 and 9th International Workshop, FATES 2009, Eindhoven, The Netherlands, November 2-4, 2009. Proceedings, volume 5826 of Lecture Notes in Computer Science, pages 129-145. Springer, 2009. URL: https://doi.org/10.1007/978-3-642-05031-2_9.
  5. Rita Dorofeeva, Khaled El-Fakih, Stéphane Maag, Ana R. Cavalli, and Nina Yevtushenko. FSM-based conformance testing methods: A survey annotated with experimental evaluation. Information & Software Technology, 52(12):1286-1297, 2010. URL: https://doi.org/10.1016/j.infsof.2010.07.001.
  6. Rita Dorofeeva, Khaled El-Fakih, and Nina Yevtushenko. An improved conformance testing method. In Farn Wang, editor, Formal Techniques for Networked and Distributed Systems - FORTE 2005, 25th IFIP WG 6.1 International Conference, Taipei, Taiwan, October 2-5, 2005, Proceedings, volume 3731 of Lecture Notes in Computer Science, pages 204-218. Springer, 2005. URL: https://doi.org/10.1007/11562436_16.
  7. Feodor F. Dragan and Arne Leitert. On the minimum eccentricity shortest path problem. Theoretical Computer Science, 694:66-78, 2017. URL: https://doi.org/10.1016/j.tcs.2017.07.004.
  8. P. Fiterău-Broştean, R. Janssen, and F.W. Vaandrager. Combining model learning and model checking to analyze TCP implementations. In S. Chaudhuri and A. Farzan, editors, Proceedings 28th International Conference on Computer Aided Verification (CAV'16), Toronto, Ontario, Canada, volume 9780 of Lecture Notes in Computer Science, pages 454-471. Springer, 2016. URL: https://doi.org/10.1007/978-3-319-41540-6_25.
  9. S. Fujiwara, G. von Bochmann, F. Khendek, M. Amalou, and A. Ghedamsi. Test selection based on finite state models. IEEE Transactions on Software Engineering, 17(6):591-603, 1991. URL: https://doi.org/10.1109/32.87284.
  10. Maciej Gazda and Robert M. Hierons. Model independent refusal trace testing. Science of Computer Programming, 239:103173, 2025. URL: https://doi.org/10.1016/j.scico.2024.103173.
  11. Herman Geuvers and Bart Jacobs. Relating apartness and bisimulation. Logical Methods in Computer Science, Volume 17, Issue 3, July 2021. URL: https://doi.org/10.46298/lmcs-17(3:15)2021.
  12. F. C. Hennie. Fault detecting experiments for sequential circuits. In Proceedings of the Fifth Annual Symposium on Switching Circuit Theory and Logical Design, pages 95-110, 1964. URL: https://doi.org/10.1109/SWCT.1964.8.
  13. Falk Howar and Bernhard Steffen. Active automata learning in practice. In Amel Bennaceur, Reiner Hähnle, and Karl Meinke, editors, Machine Learning for Dynamic Software Analysis: Potentials and Limits: International Dagstuhl Seminar 16172, Dagstuhl Castle, Germany, April 24-27, 2016, Revised Papers, pages 123-148. Springer International Publishing, 2018. Google Scholar
  14. Felix Hübner, Wen-ling Huang, and Jan Peleska. Experimental evaluation of a novel equivalence class partition testing strategy. Softw. Syst. Model., 18(1):423-443, 2019. URL: https://doi.org/10.1007/S10270-017-0595-8.
  15. Bálint Kocsis and Jurriaan Rot. Complete test suites for automata in monoidal closed categories. In Parosh Aziz Abdulla and Delia Kesner, editors, Foundations of Software Science and Computation Structures - 28th International Conference, FoSSaCS 2025, Held as Part of the International Joint Conferences on Theory and Practice of Software, ETAPS 2025, Hamilton, ON, Canada, May 3-8, 2025, Proceedings, volume 15691 of Lecture Notes in Computer Science, pages 198-219. Springer, 2025. URL: https://doi.org/10.1007/978-3-031-90897-2_10.
  16. Loes Kruger, Sebastian Junges, and Jurriaan Rot. Small test suites for active automata learning. In Bernd Finkbeiner and Laura Kovács, editors, Tools and Algorithms for the Construction and Analysis of Systems - 30th International Conference, TACAS 2024, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2024, Luxembourg City, Luxembourg, April 6-11, 2024, Proceedings, Part II, volume 14571 of Lecture Notes in Computer Science, pages 109-129. Springer, 2024. URL: https://doi.org/10.1007/978-3-031-57249-4_6.
  17. D. Lee and M. Yannakakis. Testing finite-state machines: State identification and verification. IEEE Transactions on Computers, 43(3):306-320, 1994. URL: https://doi.org/10.1109/12.272431.
  18. D. Lee and M. Yannakakis. Principles and methods of testing finite state machines - a survey. Proceedings of the IEEE, 84(8):1090-1123, 1996. URL: https://doi.org/10.1109/5.533956.
  19. Gang Luo, Alexandre Petrenko, and Gregor von Bochmann. Selecting test sequences for partially-specified nondeterministic finite state machines. In Tadanori Mizuno, Teruo Higashino, and Norio Shiratori, editors, Protocol Test Systems: 7th workshop 7th IFIP WG 6.1 international workshop on protocol text systems, pages 95-110, Boston, MA, 1995. Springer US. URL: https://doi.org/10.1007/978-0-387-34883-4_6.
  20. T. Maarse. Active Mealy Machine Learning using Action Refinements. Master thesis, Radboud University Nijmegen, August 2020. URL: https://www.cs.ru.nl/F.Vaandrager/thesis-2019_NWI-IMC048_s4416295.pdf.
  21. I. Melse. Sufficient Conditions for k-Completeness Using Observation Trees and Apartness. Bachelor thesis, ICIS, Radboud University Nijmegen, April 2024. URL: https://www.cs.ru.nl/bachelors-theses/2024/Ivo_Melse___s1088677___Sufficient_Conditions_for_k-completeness_using_Observation_Trees_and_Apartness.pdf.
  22. J. Moerman. Nominal Techniques and Black Box Testing for Automata Learning. PhD thesis, Radboud University Nijmegen, July 2019. Google Scholar
  23. E.F. Moore. Gedanken-experiments on sequential machines. In Automata Studies, volume 34 of Annals of Mathematics Studies, pages 129-153. Princeton University Press, 1956. Google Scholar
  24. Daniel Neider, Rick Smetsers, Frits W. Vaandrager, and Harco Kuppens. Benchmarks for automata learning and conformance testing. In Tiziana Margaria, Susanne Graf, and Kim G. Larsen, editors, Models, Mindsets, Meta: The What, the How, and the Why Not?, volume 11200 of Lecture Notes in Computer Science, pages 390-416. Springer, 2018. URL: https://doi.org/10.1007/978-3-030-22348-9_23.
  25. Doron Peled, Moshe Y. Vardi, and Mihalis Yannakakis. Black box checking. In Jianping Wu, Samuel T. Chanson, and Qiang Gao, editors, Proceedings FORTE, volume 156 of IFIP Conference Proceedings, pages 225-240. Kluwer, 1999. Google Scholar
  26. A. Petrenko, T. Higashino, and T. Kaji. Handling redundant and additional states in protocol testing. In A. Cavalli and S. Budkowski, editors, Proceedings of the 8th International Workshop on Protocol Test Systems IWPTS '95, Paris, France, pages 307-322, 1995. Google Scholar
  27. Alexandre Petrenko and Nina Yevtushenko. Conformance tests as checking experiments for partial nondeterministic FSM. In Wolfgang Grieskamp and Carsten Weise, editors, Formal Approaches to Software Testing, 5th International Workshop, FATES 2005, Edinburgh, UK, July 11, 2005, Revised Selected Papers, volume 3997 of Lecture Notes in Computer Science, pages 118-133. Springer, 2005. URL: https://doi.org/10.1007/11759744_9.
  28. Alexandre Petrenko, Nina Yevtushenko, Alexandre Lebedev, and Anindya Das. Nondeterministic state machines in protocol conformance testing. In Omar Rafiq, editor, Protocol Test Systems, VI, Proceedings of the IFIP TC6/WG6.1 Sixth International Workshop on Protocol Test systems, Pau, France, 28-30 September, 1993, volume C-19 of IFIP Transactions, pages 363-378. North-Holland, 1993. Google Scholar
  29. Robert Sachtleben. Unifying frameworks for complete test strategies. Science of Computer Programming, 237:103135, 2024. URL: https://doi.org/10.1016/j.scico.2024.103135.
  30. Adenilso Simão, Alexandre Petrenko, and Nina Yevtushenko. On reducing test length for FSMs with extra states. Softw. Test. Verification Reliab., 22(6):435-454, 2012. URL: https://doi.org/10.1002/STVR.452.
  31. Wouter Smeenk, Joshua Moerman, Frits W. Vaandrager, and David N. Jansen. Applying automata learning to embedded control software. In Michael J. Butler, Sylvain Conchon, and Fatiha Zaïdi, editors, Formal Methods and Software Engineering - 17th International Conference on Formal Engineering Methods, ICFEM 2015, France, 2015, Proceedings, volume 9407 of Lecture Notes in Computer Science, pages 67-83. Springer, 2015. URL: https://doi.org/10.1007/978-3-319-25423-4_5.
  32. Michal Soucha and Kirill Bogdanov. SPYH-method: An improvement in testing of finite-state machines. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops, Västerås, Sweden, April 9-13, 2018, pages 194-203. IEEE Computer Society, 2018. URL: https://doi.org/10.1109/ICSTW.2018.00050.
  33. A. S. Troelstra and H. Schwichtenberg. Basic Proof Theory. Cambridge Tracts in Theoretical Computer Science. Cambridge University Press, 2 edition, 2000. URL: https://doi.org/10.1017/CBO9781139168717.
  34. Frits Vaandrager. Model learning. Communications of the ACM, 60(2):86-95, February 2017. URL: https://doi.org/10.1145/2967606.
  35. Frits Vaandrager. A new perspective on conformance testing based on apartness. In Venanzio Capretta, Robbert Krebbers, and Freek Wiedijk, editors, Logics and Type Systems in Theory and Practice: Essays Dedicated to Herman Geuvers on The Occasion of His 60th Birthday, pages 225-240, Cham, 2024. Springer Nature Switzerland. URL: https://doi.org/10.1007/978-3-031-61716-4_15.
  36. Frits W. Vaandrager, Paul Fiterau-Brostean, and Ivo Melse. Completeness of FSM test suites reconsidered. CoRR, abs/2410.19405, 2024. URL: https://doi.org/10.48550/arXiv.2410.19405.
  37. Frits W. Vaandrager, Bharat Garhewal, Jurriaan Rot, and Thorsten Wißmann. A new approach for active automata learning based on apartness. In Dana Fisman and Grigore Rosu, editors, Tools and Algorithms for the Construction and Analysis of Systems - 28th International Conference, TACAS 2022, Munich, Germany, April 2-7, 2022, Proceedings, Part I, volume 13243 of Lecture Notes in Computer Science, pages 223-243. Springer, 2022. URL: https://doi.org/10.1007/978-3-030-99524-9_12.
  38. Petra van den Bos, Ramon Janssen, and Joshua Moerman. n-Complete test suites for IOCO. Softw. Qual. J., 27(2):563-588, 2019. URL: https://doi.org/10.1007/S11219-018-9422-X.
  39. M.P. Vasilevskii. Failure diagnosis of automata. Cybernetics and System Analysis, 9(4):653-665, 1973. (Translated from Kibernetika, No. 4, pp. 98-108, July-August, 1973.). URL: https://doi.org/10.1007/BF01068590.
  40. Nan Yang, Kousar Aslam, Ramon R. H. Schiffelers, Leonard Lensink, Dennis Hendriks, Loek Cleophas, and Alexander Serebrenik. Improving model inference in industry by combining active and passive learning. In Xinyu Wang, David Lo, and Emad Shihab, editors, 26th IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2019, Hangzhou, China, February 24-27, 2019, pages 253-263. IEEE, 2019. URL: https://doi.org/10.1109/SANER.2019.8668007.
  41. N. V. Yevtushenko and A. F. Petrenko. Synthesis of test experiments in some classes of automata. Autom. Control Comput. Sci., 24(4):50-55, April 1991. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact