On Assessing ML Model Robustness: A Methodological Framework (Academic Track)

Authors Afef Awadid, Boris Robert



PDF
Thumbnail PDF

File

OASIcs.SAIA.2024.1.pdf
  • Filesize: 1.29 MB
  • 10 pages

Document Identifiers

Author Details

Afef Awadid
  • IRT SystemX, Palaiseau, France
Boris Robert
  • IRT Saint Exupéry, Toulouse, France

Acknowledgements

We would like to extend special thanks to the robustness team of the confiance.ai research program, composed of AI engineers, for producing the deliverables that served as the foundation for this paper.

Cite As Get BibTex

Afef Awadid and Boris Robert. On Assessing ML Model Robustness: A Methodological Framework (Academic Track). In Symposium on Scaling AI Assessments (SAIA 2024). Open Access Series in Informatics (OASIcs), Volume 126, pp. 1:1-1:10, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/OASIcs.SAIA.2024.1

Abstract

Due to their uncertainty and vulnerability to adversarial attacks, machine learning (ML) models can lead to severe consequences, including the loss of human life, when embedded in safety-critical systems such as autonomous vehicles. Therefore, it is crucial to assess the empirical robustness of such models before integrating them into these systems. ML model robustness refers to the ability of an ML model to be insensitive to input perturbations and maintain its performance. Against this background, the Confiance.ai research program proposes a methodological framework for assessing the empirical robustness of ML models. The framework encompasses methodological processes (guidelines) captured in Capella models, along with a set of supporting tools. This paper aims to provide an overview of this framework and its application in an industrial setting.

Subject Classification

ACM Subject Classification
  • Software and its engineering → Software verification and validation
Keywords
  • ML model robustness
  • assessment
  • framework
  • methodological processes
  • tools

Metrics

  • Access Statistics
  • Total Accesses (updated on a weekly basis)
    0
    PDF Downloads

References

  1. ISO/IEC 22989:2022. Information technology — artificial intelligence — artificial intelligence concepts and terminology, 2022. Google Scholar
  2. ISO/IEC TR 24029-1. Artificial intelligence (ai)—assessment of the robustness of neural networks—part 1: Overview, 2021. Google Scholar
  3. ISO/IEC TR 25059. Iso/iec 25059:2023 - systems and software engineering - systems and software quality requirements and evaluation (square) - quality model for ai-based systems, 2023. Google Scholar
  4. Abraham Chan, Arpan Gujarati, Karthik Pattabiraman, and Sathish Gopalakrishnan. The fault in our data stars: studying mitigation techniques against faulty training data in machine learning applications. In 2022 52nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pages 163-171. IEEE, 2022. URL: https://doi.org/10.1109/DSN53405.2022.00027.
  5. Gregory Chance, Dhaminda B Abeywickrama, Beckett LeClair, Owen Kerr, and Kerstin Eder. Assessing trustworthiness of autonomous systems. arXiv preprint arXiv:2305.03411, 2023. URL: https://doi.org/10.48550/arXiv.2305.03411.
  6. Mahyar Fazlyab, Manfred Morari, and George J Pappas. Probabilistic verification and reachability analysis of neural networks via semidefinite programming. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 2726-2731. IEEE, 2019. URL: https://doi.org/10.1109/CDC40024.2019.9029310.
  7. Y Guo. Globally robust stability analysis for stochastic cohen-grossberg neural networks with impulse control and time-varying delays. Ukrainian Mathematical Journal, 69(8):1049-106, 2017. Google Scholar
  8. Chengqiang Huang, Zheng Hu, Xiaowei Huang, and Ke Pei. Statistical certification of acceptable robustness for neural networks. In Artificial Neural Networks and Machine Learning-ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part I 30, pages 79-90. Springer, 2021. URL: https://doi.org/10.1007/978-3-030-86362-3_7.
  9. He Huang, Yuzhong Qu, and Han-Xiong Li. Robust stability analysis of switched hopfield neural networks with time-varying delay under uncertainty. Physics Letters A, 345(4-6):345-354, 2005. Google Scholar
  10. Uyeong Jang, Xi Wu, and Somesh Jha. Objective metrics and gradient descent algorithms for adversarial examples in machine learning. In Proceedings of the 33rd Annual Computer Security Applications Conference, pages 262-277, 2017. URL: https://doi.org/10.1145/3134600.3134635.
  11. Mohd Javaid, Abid Haleem, Ibrahim Haleem Khan, and Rajiv Suman. Understanding the potential applications of artificial intelligence in agriculture sector. Advanced Agrochem, 2(1):15-30, 2023. Google Scholar
  12. Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, pages 97-117. Springer, 2017. URL: https://doi.org/10.1007/978-3-319-63387-9_5.
  13. Natan Levy and Guy Katz. Roma: A method for neural network robustness measurement and assessment. In International Conference on Neural Information Processing, pages 92-105. Springer, 2022. URL: https://doi.org/10.1007/978-981-99-1639-9_8.
  14. Ping Li, Fang Xiong, Xibei Huang, and Xiaojun Wen. Construction and optimization of vending machine decision support system based on improved c4. 5 decision tree. Heliyon, 10(3), 2024. Google Scholar
  15. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083, 2017. URL: https://arxiv.org/abs/1706.06083.
  16. Mohamed-Iliasse Mahraz, Loubna Benabbou, and Abdelaziz Berrado. Machine learning in supply chain management: A systematic literature review. International Journal of Supply and Operations Management, 9(4):398-416, 2022. Google Scholar
  17. Dietmar PF Möller. Machine learning and deep learning. In Guide to Cybersecurity in Digital Transformation: Trends, Methods, Technologies, Applications and Best Practices, pages 347-384. Springer, 2023. Google Scholar
  18. Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773, 2017. Google Scholar
  19. Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, et al. Adversarial robustness toolbox v1. 0.0. arXiv preprint arXiv:1807.01069, 2018. Google Scholar
  20. Yixin Nie, Yicheng Wang, and Mohit Bansal. Analyzing compositionality-sensitivity of nli models. In Proceedings of the AAAI conference on artificial intelligence, pages 6867-6874, 2019. URL: https://doi.org/10.1609/AAAI.V33I01.33016867.
  21. Wenjie Ruan, Min Wu, Youcheng Sun, Xiaowei Huang, Daniel Kroening, and Marta Kwiatkowska. Global robustness evaluation of deep neural networks with provable guarantees for the hamming distance. IJCAI-19, 2019. Google Scholar
  22. Jie Wang, Jun Ai, Minyan Lu, Haoran Su, Dan Yu, Yutao Zhang, Junda Zhu, and Jingyu Liu. A survey of neural network robustness assessment in image recognition. arXiv preprint arXiv:2404.08285, 2024. URL: https://doi.org/10.48550/arXiv.2404.08285.
  23. Stefan Webb, Tom Rainforth, Yee Whye Teh, and M Pawan Kumar. A statistical approach to assessing neural network robustness. arXiv preprint arXiv:1811.07209, 2018. URL: https://arxiv.org/abs/1811.07209.
  24. Maurice Weber. Probabilistic Robustness Guarantees for Machine Learning Systems. PhD thesis, ETH Zurich, 2023. Google Scholar
  25. Lily Weng, Pin-Yu Chen, Lam Nguyen, Mark Squillante, Akhilan Boopathy, Ivan Oseledets, and Luca Daniel. Proven: Verifying robustness of neural networks with a probabilistic approach. In International Conference on Machine Learning, pages 6727-6736. PMLR, 2019. URL: http://proceedings.mlr.press/v97/weng19a.html.
  26. Matthew Wicker, Luca Laurenti, Andrea Patane, and Marta Kwiatkowska. Probabilistic safety for bayesian neural networks. In Conference on uncertainty in artificial intelligence, pages 1198-1207. PMLR, 2020. URL: http://proceedings.mlr.press/v124/wicker20a.html.
  27. Cong Xu, Wensheng Chen, Mingkuan Lin, Jianli Lu, Yunghsiao Chung, Jiahui Zou, Ciliang Yang, et al. Applications and challenges of hybrid artificial intelligence in chip age testing: a comprehensive review. Journal of Artificial Intelligence Practice, 6(3):70-75, 2023. Google Scholar
Questions / Remarks / Feedback
X

Feedback for Dagstuhl Publishing


Thanks for your feedback!

Feedback submitted

Could not send message

Please try again later or send an E-mail