Evaluating Dimensions of AI Transparency:
A Comparative Study of Standards, Guidelines, and the EU AI Act

Transparency has been identified as one of the key features for trustworthy AI by the international expert community [1, 3, 4, 5, 6, 8, 10, 16, 21]. However, the existing body of literature – ranging from academic papers, policy documents, recommendations, to regulations, and standards – lacks a standardized, widely-accepted definition of transparency. The documents provide varying interpretations, focusing on different dimensions of transparency such as traceability of data origin, explainability of algorithmic decisions, disclosure of the system’s abilities and limitations, and assuring user awareness about them interacting with a machine, among others. This fragmentation of interpretations leads to differing requirements for achieving transparency in AI systems, posing a significant challenge for standardization and compliance assurance with respect to AI quality.

This paper is a pilot study aiming at establishing a methodology to highlight both discrepancies and commonalities across key documents, providing a tool for policy makers to identify relevant features that need to be addressed to produce effective standards for AI transparency within a specific regulatory framework. For our analysis, we focused on selected standards and guidelines of high relevance within the European framework published by prominent German, European and international organizations during the drafting phase of the European AI Act. The AI Act itself was also included as an important reference.

By considering further documents representing other perspectives and fields of interests, our methodology has the potential to scale up to other theoretical, practical, and regulatory frameworks with differing geographical focus. This includes additional transparency dimensions being defined by other documents.

We compared transparency definitions and requirements in several pivotal documents from the field of AI regulation and standardization shown in Tab. 1. Besides the AI Act itself [7], considered sources include central papers with respect to the development of the AI Act, i.e., the HLEG GL [6], and OECD [18], already available documents from standardization, i.e., ISO 22989 [12] and IEEE 7000 or [11], as well as further guidelines and internationally recognized white papers in this direction, i.e., VDE 90012 [22] and Fraunhofer GL [20]. The selection of the documents was based on discussions that were conducted in the context of the German Standardization Roadmap on Artificial Intelligence [4] and its subsequent activities. According to its character as a pilot study, this paper did not include a comprehensive analysis of potentially relevant documents in this field, but focused on this specific selection. In the following, we present the documents in detail.

In this section, we define and delimit our evaluation criteria. The goal of the process was to achieve a comparison that indicates whether or not the documents incorporate or express a specific notion or scope of transparency. Based on the criteria defined below, the particular documents were evaluated. The evaluation had three possible results:

1. 1.

   yes – the document distinctly adopts the notion of transparency expressed by the criterion;

2. 2.

   no – the document does not refer to the notion of transparency expressed by the criterion;

3. 3.

   unclear – the document does not explicitly introduce the notion of transparency expressed by the criterion, but may contain implicit support or alignment with it.

The criteria were developed according to discussions conducted as a follow-up of the German Standardization Roadmap on Artificial Intelligence [4]. This was performed in an iterative approach where relevant aspects were collected from the included documents and the criteria were consolidated accordingly before the evaluation process was started. The authors consider the list of criteria deduced by the analysis of the above-mentioned documents to be exhaustive, meaning that no other additional generic transparency criteria were found in the reviewed documents. The authors do not exclude that new transparency criteria can be deduced, by analyzing further documents. This is part of the iterative approach and the authors recommend screening for additional transparency criteria when expanding this evaluation methodology to an extended set of documents.

For each paper, at least two members of the authors group performed the analysis. In case of a disagreement, the particular rating was discussed in the overall group of authors, who finally decided about the classification. For practical reasons, not all authors could be involved every time. For achieving a reliable consensus, at least five of the authors had to be involved into the final decision where the two authors which analyzed the document needed to be included. In cases where no full agreement could be achieved, the item was assigned to the “unclear” category. Note, however, that the evaluation category “unclear” refers to the way a specific transparency criterion is presented within a document and not to the modalities of agreement among authors.

The following list describes the used criteria in a systematic way. All criteria start with the phrase “Does the document distinctly…”, followed by the criterion, such as “… set transp. requirements relating to the design or development stage of AI?”.

The term “distinctly” as opposed to “explicitly” is chosen deliberately to include clearly implied intentions, since the choice of words and voice differs considerably between documents. For example, IEEE 7000 [11] states:

System requirements for machine learning systems may include quantitative and qualitative data-oriented specifications that include identifications for collection of data, data formats, diversity, ranges of data, …

This clearly implies that IEEE 7000 considers transparency requirements relating to the design and development stage of AI, since most commonly for machine learning systems, data will be used for training and testing during these stages – even if no explicit reference to the design and development stage of the life cycle is made. Such clearly implied intentions are, for this study, considered equivalent to explicit statements, with particular decision principles given in the following subsections.

In this process, it should be understood that the task of comparing documents that widely differ in focus, authors’ background, scope and intended impact cannot be strictly formal and performed under sharply defined criteria while still extracting meaningful results. The goal of this approach is to adequately reflect the conceptual ideas incorporated into the respective documents, requiring to some extent a margin of discretion and interpretation. The same should be applied by the readers who are to realize that the following will not replace a formal, thorough study of any individual document, possibly with technical and legal expertise, in particular when, for example, assessing a system with respect to a given standard or legal regulation, such as the AI Act. Furthermore, this motivates the aforementioned choice of three instead of just two possible evaluation results, namely “yes” and “no” when no considerable ambiguity was found, and “unclear” else (cf. Tab. 3).

For the description of the requirements, we use the coding scheme laid out in Tab. 2. For example, the code TR-STG-OPS (“Does the document distinctly set transp. requirements relating to the operation stage of AI?”) is composed of the supergroup TR for “transparency”, the group STG for life cycle stage-related criteria, and the criterion OPS for the operation stage.

Table 3 provides an overview of our analysis of criteria for AI transparency dimensions across the documents described above. The key findings from our review are presented, subsequently. For each entry, a final consensus could be found in the group of authors, according to the defined evaluation process. In some cases, the discussions showed that an ambiguity in the interpretation of the requirements persisted. Thus, the particular criterion had to be rated as “unclear”, since the definitions in the corresponding document were not clear enough or could not be interpreted in a sufficiently consistent way.

Transparency is a subject of discussion in all of the documents, emphasizing its importance in AI development. However, in some documents its definition is unclear or not provided, i.e., in OECD, VDE 90012, and the Fraunhofer GL.

Explainability of system outputs is included as an essential dimension of transparency in nearly all the documents. The only document not mentioning this aspect, namely IEEE 7000, is also the sole source in our collection not specific to AI but to system design in general. In certain documents, such as the Fraunhofer GL, HLEG GL, and the AI Act, traceability and communication are additionally mentioned as integral parts of transparency.

Transparency is explicitly demanded during both the development and operational stages across nearly all documents, except for OECD. The end of life stage, in particular in terms of transparency obligations as a task in the retirement stage, are only made explicit in ISO 22989. Furthermore, no concrete measures are expressed for this stage, even when this stage was included in the document. As previously stated, it should also be noted that the inclusion of this stage as part of the life cycle is disputed.

All documents claim that transparency requirements must be target group-specific. However, the definition of target groups is often unclear or limited.

There are significant differences regarding which target groups and use cases are considered. End users are consistently included, and also indirectly affected stakeholders / non-users are explicitly mentioned in nearly all documents, with the exception of ISO 22989. Transparency towards operators is widely addressed as well, although not always explicitly, i.e., in the HLEG GL and ISO 22989. Transparency towards developers is absent in OECD and addressed unclearly in the HLEG GL. For the other documents, adequate information needs to be provided to the developers in order to achieve sufficient transparency. Transparency for testing or auditing organizations is explicitly mentioned just in four out of seven documents, while transparency for regulators and public authorities is only addressed in a dedicated way in the AI Act and VDE 90012. More complex scenarios for the implementation of AI systems involving different actors are usually not considered. This means that most of the documents refer to a situation with a single manufacturer of the AI system. Only the AI Act explicitely discusses more complex scenarios, where, e.g., a general purpose AI (GPAI) model is developed by one manufacturer and then integrated into a downstream application by another manufacturer. In these cases, the AI Act includes transparency obligations to be fulfilled by the manufacturer of the GPAI model.

Notable differences were recognized regarding the specific kind of information that should be made transparent. Transparency concerning code, models, and training data is consistently addressed across documents. As already mentioned, transparency with respect to the explanation of outcomes is required in most of the documents, except IEEE 7000. Similarly, information regarding an AI system’s intended purpose and its predictable consequences is considered a significant transparency aspect in nearly all the documents, with the exception of VDE 90012 regarding the intended purpose and of ISO 22989 regarding the predictability of consequences. Information concerning the limitations and error modes of AI systems is generally addressed in the examined sources, even though some documents do not explicitly present this as a dimension of transparency. Finally, other aspects such as the reference to the state of the art, to the business model and operator interests, as well as the disclosure of technical parameters of the system (e.g., “weights” and “features”) are not mentioned as transparency obligations in many of the documents.

Most of the documents also require that an AI system should reveal that the user is interacting with an AI system. This requirement is only absent in ISO 22989 and unclear in IEEE 7000. Finally, it can be recognized that many entries remain marked with a grey circle. This shows that a number of topics remains ambiguous in the documents.

The presented results underline that transparency is regarded as an important ethical value in AI regulation and standardization literature. Also, there is a wide range of criteria that are considered as important requirements for AI transparency across different regulatory frameworks and guidelines. This variety of criteria reflects the fact that AI transparency is thought to have many dimensions, each of which demanding specific, tailored requirements to be addressed. Even though many transparency dimensions address overlapping technical or societal aspects, they allow for independent implementation of transparency measures. For example, informing users that they are engaging with an AI system, explaining to auditing organizations or authorities how the system processes inputs to reach outputs, and providing access to information about the AI model and training data can be executed separately using distinct approaches.

When considering the full extent of what defines transparency within the respective documents, considerable disparities become evident. No two documents share a completely compatible conception of this ethical value. A contributing factor is that the considered documents define the criteria for transparency – to a large degree – implicitly through the measures proposed, rather than comprehensively specifying concepts and goals first and deriving adequate measures systematically. Moreover, different kinds of documents have different scopes and focuses; therefore they tend to emphasize transparency aspects that are more relevant for their purposes. For instance, some of them include more technical specifications regarding appropriate measures of system transparency, while others, like the EU AI Act, focus more on high-level requirements related to AI transparency, explicitly leaving the task of defining concrete technical measures to domain-specific, technical standards.

The analyzed documents reflect that transparency basically deals with the question concerning which pieces of information should be provided to which stakeholder to deploy AI safely and responsibly. Indeed, transparency is generally considered a driver for trustworthiness, enabling meaningful and informed human interaction with the system. In order to do so, it is necessary to consider the system’s intended purpose, which defines the specific technical solutions, user groups, and use cases for the AI system. The provision of information needs to be aligned with the expertise of the involved persons as well as their particular needs. For example, there is a considerable difference between laypersons, technical experts, and actors with specific domain, application or regulatory knowledge. Additionally, the adequacy of information depends on the particular role of the actors. For example, a developer needs different information in comparison to a user, provider, auditing organization, or public authority.

This also goes for cases where a manufacturer includes an AI model from a third party provider in a new downstream application. In particular, such more complex scenarios were intensively discussed in the legislative phase of the AI Act [7] in the context of AI models without a specific / narrow intended purpose. The AI Act defines these types of models as general purpose AI (GPAI) models and describes obligations, which information has to be made available by the third-party provider of the GPAI model in order to achieve sufficient transparency for the manufacturer of the downstream application. This substantially extends the requirements, since the ethical impact can usually only be rated when the context of the application is clear. Thus, a new line of discussions was addressed in the AI Act in this regard compared to the other documents. This was due to the fact that most of these documents were older, i.e., written between 2019 and 2022, where high-impact large language models and other generative AI systems were not yet on the market.

The latest developments of the debate around transparency for GPAI models exemplify the highly dynamic nature of AI technology, which is leading to challenges regarding AI standardization and regulation. As an ethical value guiding the deployment of AI systems, transparency is a fundamental prerequisite for the achievement of other ethical principles. Indeed, a lack of awareness of the system impact and a limited understanding of its capabilities and limitations could hinder different actors along the AI value chain in the responsible use of a system, causing ethical issues ranging from safety risk to data misuse or group discrimination. Moving from these premises, the development of the AI Act [7] and the associated New Legislative Framework of the European Union are indicative of the goal to establish clearly-defined, internationally accepted rules and standards to enable ethical, value-centered use and implementation of AI systems. To this end, the exact definition of major terms, the operationalizability of concepts, and an adequate translation into concrete requirements for all relevant stakeholders along the AI value chain plays a pivotal role. The dimensions and criteria presented in the current paper are considered to serve as a starting point for systematically collecting and comparing requirements for AI transparency. In particular, this may apply to the identification of discrepancies in current standardization documents and guidelines as well as to a systematic compilation of key aspects in future development steps.

At the same time, the presented study has limitations to be considered. For practical reasons, the study was performed as a pilot study and limited to seven well-known documents in the AI context. However, a broader look that also includes scientific positions could help to establish a more thorough perspective in this very dynamic field. Additionally, the currently developed standards that include aspects of transparency should be addressed using the presented approach. For example, this may refer to important standards like ISO/IEC 42001 [13] regarding management systems for AI, or ISO/IEC 12792 [14], which directly addresses a taxonomy for transparency of AI systems. Furthermore, the development towards harmonized standards and guidelines for the implementation of the AI Act should be taken into account.

It must be noted that the considered and compared documents were released in the time span between 2019 [6] to 2024 [7]. These past five years have been characterized by an unprecedented level of disruption in AI technology, such as the development of large language models and generative AI models for image and video synthesis. These new types of models blur the line between human and AI capabilities – accompanied by a substantial shift in the perception of AI systems, their potentials, societal impacts and risks, and regulatory requirements. Hence, the documents, even though seemingly released in quick succession, must already in part be viewed in their individual “historical” context, explaining, for example that the AI Act [7] heavily addresses the challenges of GPAI, while in IEEE 7000 [11], many transparency requirements common for machine learning are absent. This explains in part the heterogeneity of the analyzed documents, such as concerning requirements with respect to GPAI models.

Further on, our analysis encountered challenges due to the inherent complexity and occasional ambiguity of the sources. The fulfillment of specific criteria was not always explicitly stated in a single, easily identifiable section, necessitating an interpretative evaluation of the overall narrative of the document. To maintain objectivity and ensure the accuracy of our classifications, our research team implemented a systematic review process. This involved meticulous discussion and careful cross-referencing of the documents to ensure our evaluations were anchored in the text. This systematic approach, guided by the framework set out in our methodology, aimed to limit the influence of subjective judgments and provided an accurate representation of each document’s stance on transparency requirements, as depicted in Tab. 3.

In conclusion, our study offers an initial framework for evaluating and comparing ethical concepts within standards and regulations, specifically focusing on the notion of transparency in AI. Its application has highlighted that the definitions of transparency differ to a considerable degree among the considered documents. These differences can, to some extent, be attributed to the different purposes of these documents.

However, in the absence of a standardized framework for representing and comparing these definitions and their differences, the variety of notions arguably hinders the underlying goal of establishing a shared understanding across stakeholders.

Future research should expand upon this approach by assessing the criteria for transparency against a broader spectrum of references, which should include documents in the field of standardization on the one hand, e.g., regulations, standards, or guidelines; and on the other hand key academic publications as well as high-impact white papers.

In the European context, the implementation of the AI Act represents a particularly promising case for the application of our methodology as it necessitates the formulation of more specific and detailed standards for AI. By assessing how the various concepts and requirements for transparency adopted in different documents align with those found in the AI Act, our approach can help to identify key areas for creating a harmonized regulatory framework. These efforts should not only be consistent with the AI Act but also seek to elaborate more detailed standards for sector-specific applications.

Ultimately, our approach is designed to create clear and precise definitions for transparency dimensions and corresponding operationalizable criteria, representing the building blocks for agile AI governance frameworks. By doing so, we equip policymakers and industry stakeholders with fundamental tools to ensure AI trustworthiness and enhance their capacity to adapt to the ongoing developments in technology and the evolution of ethical standards in the AI field.