Transparency of AI Systems

The above definition is based on the presentation of the transparency concept in [5] and [2]. It is compliant with the transparency requirements in the EU AI Act (see full version of the paper for details) and represents the position of the BSI. In the following subsections, the individual elements of the definition are described in more detail.

The existence of already published definitions of the concept of transparency raises the question of the need for a further definition. The sheer breadth of the topic of transparency on the definition market ultimately reflects the different requirements for transparency depending on the stakeholders and the area of application. In order to provide a basis for further work of the BSI with a focus on broad stakeholder groups and generic areas of application, it was decided not to use existing definitions. In addition, the speed at which technologies in the AI sector are developing is enormous. This harbours the risk that definitions, once established, may lose their validity, especially if they are too specific. In order to keep pace with technical progress and to avoid constant renewal and adaptation of the definition to the current state of the art, the definition of transparency presented in this paper is as technology-neutral and future-proof as possible. On the one hand, it should be easy to understand, cover all relevant aspects of transparency and at the same time be open enough to allow individual interpretation depending on the respective stakeholder and the AI technology used. On the other hand, it should serve as a generic basis for future work of the BSI in this area. Furthermore, a holistic approach was taken to the definition: transparency includes both the provision of information about the AI system itself and about its ecosystem, such as the supply chain of the AI system or details about the provider. The weighting of the information provided is the responsibility of the respective stakeholder.

By promoting the transparency of AI systems, the aim is to strengthen the autonomy of stakeholders and enable them to decide for themselves whether the use, modification or provision of an AI system is appropriate and justifiable for them. It is not enough to simply describe the capabilities of the AI system. The limitations of the system must also be analysed and made transparent. Only in this way can a holistic assessment be made by the stakeholders, e.g. whether an AI system is suitable for a particular purpose or not. In the area of digital consumer protection, this should help to ensure that consumers can recognise and use safe and trustworthy AI systems despite increasing digitalisation. Companies and organisations should also be enabled to develop and operate their own AI systems transparently. Transparent information from the ecosystem of an AI system should also enable third parties affected by impacts to recognise how they can assert their rights in the event of damage. The transparency of AI systems thus serves to empower stakeholders.

The use of transparent AI systems can promote the traceability of decisions and the assessment of the appropriateness of systems. Transparency can also help to protect against misuse by enabling potential risks and undesirable effects to be recognised at an early stage. In order to react appropriately to problems, it is important to know, for example, whether the output of an AI system is free from discrimination or whether it violates licence conditions. In terms of consumer protection transparency can also act as a support tool. Transparency provides the basis for a correct assessment of the appropriateness of the system used. In order to be able to make such an assessment at all, information about the system must be accessible. A valid assessment of the adequacy of the AI system forms the basis for positive trust and acceptance processes. Initial publications show, for example, higher download numbers for transparent AI models, which could be an indication of better acceptance of these systems among developers [4]. Lack of transparency complicates the valid assessment of the adequacy of a system, and thus the assessment of its trustworthiness. The latter is a prerequisite for establishing and maintaining a positive relationship of trust with the system and the related output. In addition, transparency can enable users to exercise rights more easily by requiring transparency accompanied by clearer definitions of legal responsibilities and identification of those responsible for the use of AI systems. The listed aspects of traceability, abuse protection, acceptance and trustworthiness as well as legal responsibility show the relevance of transparency in the use of AI systems. This relevance is also reflected in regulatory and legal requirements (see full version of the paper). Transparency can, on the one hand, contribute to the security of AI systems and, on the other hand, promote the safe use of AI applications. In this way, transparency can enable the identification of possible problems and vulnerabilities, make undesirable system behaviour visible and contribute to problem identification and the prevention of misuse. In the context of IT security, transparency also provides the basis for the disclosure and assessment of risks associated with the use of the system. The identification of roles and responsibilities in the event of damage and adverse events as part of transparency requirements can also help stakeholders to detect faulty system behaviour, reduce response times and thus mitigate possible consequential damage. If transparency is practised in the early phases of the life cycle of an AI system, inconsistencies can be avoided within the development team from the outset, sources of error minimised and training phases shortened. AI systems are both developed and increasingly used as development tools – e.g. in AI-supported programming for the automatic generation of program code – for new systems. In the early development phases, it is also important from a developer’s point of view to know where the training/test/validation data come from, how they are obtained and whether they are free of bias – e.g. to avoid discrimination. This information is important in order to be able to prepare the data correctly before training/testing/validating (pre-processing). Current development trends also show that pre-existing models are often used, which makes the presence and accessibility of all safety-critical information on existing models particularly relevant. In the absence of this information, there is a risk of implementing the security risks of the basic models into your own products. Both systems are then interdependent, and any lack of transparency in the underlying system is transferred to the system built on top of it. The inheritance of the security risks described above from different areas leads to an increased overall risk in the examples mentioned, which underlines once again the explosiveness of transparency for security-relevant aspects when using AI systems. In addition, as discussed in the previous section, transparency can contribute to better user empowerment in assessing AI systems. A correct assessment of the application, suitable application scenarios and possible problems and security risks can promote safe use by users.

So far, the positive aspects of transparency have mainly been presented. Increasing/improving the transparency of AI systems can also have unintended negative effects. For example, the provision of information on the functionality or architecture of an AI system can reveal new attack vectors that attackers can exploit to misuse or compromise the system. Information about limitations or excluded areas of application of an AI system could also be deliberately exploited by attackers, e.g. to deliberately generate erroneous behaviour or destructive output. Conversely, attackers can also abuse the trust that transparency is supposed to create in order to deliberately provide incorrect information. For example, in reality, safety-critical applications can be presented as uncritical. In addition, non-transparent systems can also be marked as transparent. Such pseudo-transparency can be used for marketing purposes of the own product and lead to a wrong assessment of the system by consumers if the transparency label is not checked. Therefore, questions about the trustworthiness of the information disclosed/provided must also be answered in the future. An official transparency label and verifiable transparency criteria could provide a remedy here. The transparency of AI systems is therefore a double-edged sword and should hence be used with caution. The goals and problems are sometimes contradictory and cannot be solved simultaneously. Answering the key questions “What information does a stakeholder need to make a decision?” and “What information is not relevant?” can be helpful. Similar to the EU General Data Protection Regulation (GDPR) the principle of data minimisation is also recommended here, which is answered separately for each individual use case: as much information as necessary, but no more than strictly required, should be disclosed. This “need-to-know” principle applies especially to safety-critical information. The goal should be an appropriate level of transparency, which is sufficient, and at the same time aspects such as security should not be too disadvantaged.