AI Certification: Empirical Investigations into Possible Cul-De-Sacs and Ways Forward

The first study reviewed activities surrounding the safeguarding and certification of AI systems over recent years. Collaborations with industry partners in Germany across various domains (automotive, finance, and food manufacturing) and interviews with certification experts, employees, and developers led to the publication of a framework for documenting AI applications along four major development steps [2]. The feedback from practitioners highlighted five main points:

The AI Assessment Catalogue [6] is widely used in Germany and regarded as a de facto standard, but its length (approximately 160 pages) poses practical challenges. It is primarily designed for final assessments, not for providing support during the development phase.

There is some uncertainty about the Assessment Catalog, as practitioners are concerned about its compliance with the AI Act and its extensive length, though it remains the primary tool for developing safe AI.

Partners expressed high hopes for future standards, as they are eagerly awaiting new standards for safe AI development, showing interest in best practices for data collection and model selection, despite there being some concern about the practical usefulness of these standards.

Need for implementation details, technical tools, and clearly defined performance thresholds was often expressed, as partners desire more concrete guidance, including specific tools and methods, actionable instructions, and specific information to facilitate the development process.

Trust-Building is a key motivation to pursue certification of AI systems. Affected individuals want to be included in the development process and understand the design choices made.

The second study focused on experts in XAI and AI certification to explore expectations regarding the use of XAI in this area [3]. Through 15 qualitative interviews, it was found that XAI can help to identify errors such as biases within AI models. However, it cannot comprehensively answer the question, “Is this AI model safe to use?”

Surrounding the use of XAI, some common themes (also in part echoing the practitioners’ view of the first study) emerged: Experts missed clear guidance and standardization on how and when to use XAI. They employed a variety of XAI methods to address specific problems, sometimes successfully (especially as a communication tool between experts) and sometimes unsuccessfully (when data relationships were unknown or too complex). Additionally, XAI methods face several challenges:

• $\mathrm{■}$

  Difficulty in implementation or use.

• $\mathrm{■}$

  Growing complexity of AI systems, such as large language models (LLMs).

• $\mathrm{■}$

  Need to adapt XAI methods to different data types, use cases and AI models, including multi-modal ones.

• $\mathrm{■}$

  Difficulty (or impossibility) in objectively assessing the quality of explanations.

These challenges overall led the experts to believe that XAI is unlikely to provide comprehensive solutions to AI certification in the near future. While some hope was levied towards new XAI-approaches, the most promising methods mentioned regarding AI certification (apart from just spotting ML model biases) were formal verification methods that provide (statistical) guarantees for AI properties and AI approaches incorporating transparent decision-making by design, such as neuro-symbolic approaches.