Explainability in Focus: Advancing Evaluation through Reusable Experiment Design

In order to assess self-reported understanding, in line with other subjective assessments from the human factors literature [22], we suggest developing a suite of Likert scale-based questions, which probe users about their individual perceptions of how well they comprehend an AI system given any explanations that they have been provided. While Hoffman et al. have proposed explanation satisfaction and trust scales [10], scales that focus on self-reported human understanding have been underexplored to date.

Previous examples of such questions include [3, 24] (1) I understand how the model works to predict whether a defendant will reoffend [whether the primary tree species in an area is spruce/fir; “I understand the admission algorithm”]; (2) I can predict how the model will behave.

In addition to asking for Likert-based responses to questions from the categories above, it would be additionally useful to ask users to self-report their confidence for each item. We note that items may vary in terms of understanding complexity, and show different patterns in performance across a set of participants [25]. Therefore, we also recommend analysing performance on individual or specific questions rather than computing them on aggregate (e.g., sum of accurate responses).

The fundamental challenge in assessing user understanding in an objective manner is selecting assessments and metrics that faithfully reflect the user’s underlying mental model. Mental models can be shallow, covering only a functional understanding of a system (e.g. a driver knows how to operate a car), or deep, achieving a more structural understanding of how a system functions (e.g. a mechanic understands the inner-workings of a car and how to fix broken cars or extend their capabilities) [13]. Importantly, the appropriateness of such mental models depends on a user’s context, including their attributes and expertise, tasks, use cases, and goals. Previous work has proposed a mental model soundness scare which addresses these factors and incorporates domain-specific comprehension questions [13].

One structured approach to determine a user’s goal-informed informational needs within a given context is based on the situation awareness (SA) framework from the human factors literature. Endsley defines SA as the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status into the future [7]. SA requirements for a given context define a person’s informational needs which can be met in part through the application of XAI when a human is working with an AI system within their context. Sanneman and Shah apply the SA framework to XAI, and define the following three levels of XAI [20]:

• $\mathrm{■}$

  Level 1: XAI for Perception—explanations of what an AI system did or is doing and the decisions made by the system

• $\mathrm{■}$

  Level 2: XAI for Comprehension—explanations of why an AI system acted in a certain way or made a particular decision and what this means in terms of the system’s goals

• $\mathrm{■}$

  Level 3: XAI for Projection—explanations of what an AI system will do next, what it would do in a similar scenario, or what would be required for an alternate outcome

A process such as goal-directed task analysis (GDTA) can be applied to elicit a user’s informational needs at these three levels [8], applicable explanations techniques can be selected to meet these needs, and their efficacy at supporting a user’s mental model can be assessed by applying a technique such as the situation awareness global assessment technique (SAGAT), a validated SA assessment from the human factors literature [7, 19].

There are various other models. Speith et al. [21], for example, generalize Sannemann and Shah’s model by incorporating findings from various disciplines outside of human factors (e.g., cognitive psychology, philosophy). Their model comprises six levels of skills that can be tested in studies and are said to correlate with varying degrees of understanding. Their aim is to make studies more comparable.

We first address the definition of explanation fidelity (or faithfulness). The objective in this case is to evaluate whether an explanation accurately reflects the internal reasoning of the model. We consider this an objective property, since it concerns the alignment between the explanation and the model’s actual behavior, independently of any human interpretation. The definition of explanation fidelity is not straightforward. In the literature, we can find many terms, including faithfulness and correctness. However, faithfulness may be misleading due to its connotation of belief or trust, which does not align with the technical nature of the concept. In addition, its definition is highly task-dependent and varies significantly depending on the type of explanation considered. For instance, when considering feature importance-based explanations, faithfulness can be evaluated by masking the most important features at prediction time, but this only applies for this specific kind of explanations. Rather than relying on a single notion of fidelity, a useful perspective may come from related concepts such as comprehensiveness and sufficiency [6]. A comprehensive explanation contains all the critical components that influence the prediction, while a sufficient explanation identifies the minimal set of elements necessary to reach the same outcome. In particular, an explanation makes claims about the factors that cause or influence the model’s prediction. Therefore, the main objective of explanation fidelity is to verify whether these factors are truly influential. In practice, if a change in the explanation leads to a change in the prediction, this supports the causal relevance of the explanation’s components. As already mentioned, the technical implementation of this criteria can be challenging given the different ML tasks available, as well as the various kind of explanations. However, the evaluation can be approached by altering the components identified in the explanation, such as masking an important feature in a classifier or removing a rule in an expert system, and observing whether the model’s output changes accordingly.

Another important aspect of the explanation is its stability. Also in this case we can find different terms, such as consistency or robustness, and many definitions. After our discussion, we claim that stability refers to the consistency of a model’s predictions and explanations when provided with similar inputs. A stable explanation method should produce similar outputs and explanations for inputs that are close according to a similarity metric. To set the stage, we can consider two similar records, $x$ and $x^1$ that give the same output, $b(x)=b(x^1)$. In this case, we expect two similar explanations, $e$ and $e^1$. Therefore, the evaluation requires two components: a metric for measuring similarity between input instances $x$ and $x^1$, and a metric for comparing the corresponding explanations $e$ and $e^1$. This property can be assessed through objective criteria, but we prefer to consider it as an objective and subjective criteria. In fact, when defining what constitutes similar inputs or explanations, it may also involve human intervention to define or validate similarity measures. The specific approach to measuring stability can vary depending on the task at hand and the type of explanation being considered.

Although the performance of a Human-AI team is often the primary goal in most applications and heavily affected by users’ trust and reliance, the measurement of performance is left here vague on purpose, as it is very task specific.

You would use the same metric that you would use to evaluate the model in isolation (or the user in isolation). For instance, in model debugging performance means the quality of the model one obtains, whereas in a classification task it is rather the accuracy the joint performance of human and system.

Self-reported trust is the extent to which a user believes they trust in the AI model. There is a multitude of questionnaires to measure self-reported trust (see [12] for an overview). Some of the frequently used questionnaires may not be appropriate in every situation. For example, the questionnaire by Hoffman et al. [10] tends to include concepts different from trust.

Additionally, self-reported trust can be measured by means of betting markets, i.e., how much the participants are willing to bet on the model giving a correct output in situations without having personal stake.

Additional thoughts on self-reported trust in a specific instance:

• $\mathrm{■}$

  possibly only for experts, as laypeople might not distinguish between the system and its individual explanations

• $\mathrm{■}$

  maybe the very same metrics apply (e.g. trust in automation), with some customization

Methods to measure observed reliance include:

1. 1.

   Investment games in which the participant initially has several points that can be used to bet on the AI system. The number of points they are willing to spend indicates the reliance on the system. Two interesting resources for this kind of task are: [16] for general for general XAI models and [11] for explainable reinforcement learning.

2. 2.

   Stakes scenarios that investigate the behavior in different settings, e.g., using the system to recommend a movie vs. to diagnose a life-threatening illness

3. 3.

   Delegation tasks that examine in which cases the users entirely delegate a (sub)task to the system. These can be augmented with betting.

4. 4.

   Measurement of “switch rates” [26]: How often and when do users switch to the system’s recommendation in case it deviates from their own initial assumption or outcome? This can be designed as a multi-step approach where the user makes an initial assumption and then is presented the system’s recommendation. In case the user and system results are different, the participants in a final step have to decide whether they keep their own result or adopt the system’s recommendation.

5. 5.

   Willingness to follow advice: how much do people deviate towards the algorithmic estimate based on their own estimate. This approach is similar to the switch rate task but applied to numeric decisions.

How can we measure whether explanations help experts/designers improve models? We consider two types of measurements:

There is a delicate relationship between trust and model improvement. If a domain expert is involved, then exposing them to model errors may reduce their trust in the system. Conversely, if they have been involved in improving the model, then this may help increase their trust, and in fact involving domain experts may ultimately help them appropriately calibrate the right level of trust they should have in the system’s behavior. This is an important aspect to track and study in model improvement experiments, even though it is not the primary objective.

Training data misalignment Data misalignment between training data vs test (debugging) data Fundamentally missing from the training data (vs complete ground truth of all the possible data) Improving alignment between the AI model and expert user’s knowledge on the task Example based correction: by explaining the missing examples, one can point out which part of data is missing.

Why might a system be making mistakes in the first place? It is useful to consider two dimensions of misalignment (between the actual model and the ideal/perfect target model):

We distinguish between stakeholders and their roles in pre and post deployment scenarios. We identify the main roles from the perspective of model improvement task as following:

1. 1.

   Pre-deployment:

   1. (a)

      Model developers: the goal is to improve the model for deployment

   2. (b)

      Domain experts: might be consulted during pre-deployment to add expert knowledge to the model. Domain experts could spot model errors and gaps, identify corrections, provide labels, annotations, and data in general

2. 2.

   Post-deployment:

   1. (a)

      System user: with the goal of performing the end task

   2. (b)

      Model auditor:

In order to get a sense of what criteria and metrics would be useful for the specific use case, we consider different users that might be interested in assessing fairness of this AI, and specifically:

• $\mathrm{■}$

  The job applicant: they are trying to get a job and they presumably receive a pre-screening output from the AI. Naturally, it is in their interest to verify that the AI is indeed fair.

• $\mathrm{■}$

  The hiring manager of the company offering the job.

• $\mathrm{■}$

  The “watchdogs”, e.g., the ethics department of a company, the union, etc.

• $\mathrm{■}$

  (Optionally) The job centre personnel, who are responsible for facilitating job applications.

We assume the model is a machine learning classifier trained to discriminate between promising candidates and the rest on historical data. The training examples and the input instance would consist of text records (e.g. CV, education, prior positions) either pre-processed into or paired with tabular data (e.g., personal information). We do not focus on a specific classifier architecture, for two reasons. First, we assume the classifier has been trained – presumably by either the company or the job centre – for good performance out of the many option available. Second, many high-quality explainability techniques are model agnostic and can provide competitive explanations for a variety of classifier architectures. Of course, classification performance should be tracked (for instance, via model accuracy or $F_1$ score) for consistency with the primary goal of selecting promising candidates. It is a prerequisite that the classifier achieves non-trivial prediction accuracy.

We would expect four possible kinds of information would be of interest for assessing fairness for the four stakeholders we consider:

• $\mathrm{■}$

  Prediction confidence – useful across the board.

• $\mathrm{■}$

  Feature relevance – this is especially useful for hiring managers and watchdogs for understanding whether the model is, e.g., leveragin protected attributes for its decisions.

• $\mathrm{■}$

  Counterfactual explanations – these are especially useful for the applicant (and potentially the job centre facilitator) so as to gain actionable insights about the decision.

• $\mathrm{■}$

  Prototype-based explanation, e.g., distance from “ideal” applicants – these might be useful to get a sense what kind of profile(s) the classifier is expecting successful applicants to have, and whether these are in any way undesirable.

• $\mathrm{■}$

  Perceived prediction quality and fairness: this is the basic capability being assessed.

• $\mathrm{■}$

  Satisfaction with explanation: whether explanations are perceived as useful for assessing fairness.

• $\mathrm{■}$

  Actual understanding: whether explanations have been in fact understood by stakeholders (rather than merely perceived as useful).

• $\mathrm{■}$

  Self-reported trust: whether the stakeholder believes they can rely on the AI doing a good job at generating fair predictions.

• $\mathrm{■}$

  “Correctness” of explanations, and specifically:

  • –

    Fidelty: lack of fidelity means that it may be impossible to map poor justifications for the predictions (e.g., reliance on protected attributes) to the model’s actual behavior and capabilities.

  • –

    Stability Coverage: feature relevance explanations only provide a local, per-candidate view of the model’s reasoning; this does not necessarily generalize to other instances unless the explanations are somehow stable, i.e., do not vary enormously for similar candidates (and potentially decisions); coverage refers to the fact that in order to get a sense of the overall capabilities of the classifier as a whole, for all possible instances, it may be necessary to obtain local explanations for a sizeable number of individual cases, for statistical reasons.

    All these metrics can be computed mechanically without users studies.

The idea is to use a between-participant experimental design to assess the relative performance of different UI designs. We suggest to focus on a total of $6$ designs, one for each combination of explanation type ($3$ total) and ($2$ total).

An online study seems to be sufficient for evaluating the AI and its explanations given the chosen metrics and setup. The task could be briefly summarized as “look at UIs and then complete a questionnaire: how well do you think it does its job, given your specific role?”

The question is how to evaluate the UIs from different user perspectives, and specifically how to carry out the recruiting. Ideally, we would need a reasonably large sample of applicants, job centre personnel, hiring manager, and watchdogs. This immediately poses the issue of how to get access to such a sample. Naturally, power analysis can help to identify a sufficient sample size. For certain users – e.g., applicants – one could implement a role playing setup in which Prolific participants are asked to act as applicants and evaluate the AI system under the aforementioned metrics, obtaining feedback via questionnaires. This is more problematic for specialized users like hiring managers, who might be more difficult to simulate or role play properly.

Mental model /situational awareness extraction framework Questionnaire about perceived fairness / trust Satisfaction with explanations Open text response: any other info that would have helped you assess? Demographics

For the use-case, we decided on a medical diagnosis task. Specifically, a situation where a healthcare professional has to make a single decision. An example could be ICU triage – one single patient where the decision has to be made whether the person needs to wait or needs emergency treatment now. So, we can think of a single nurse that is working during a particular shift and one ICU bed has opened up, who of the top 5 patients at risk do they admit.

For our use-case we mainly need to things. Mecial Use Case/Data: We need a medical use case given by patient data. This data normally consists of both self-reported (interview) data and objective measurements (vitals). The modality of this data is often very multi-modal including text, images and tabular time series (EHR). Ideally, we would like to get this data from use case studies for medical training.

Model: A predictive AI Model that outputs urgency rankings for each patient. Based on this, the model gives a ranking where, e.g., Person A is more urgent than Person B.

To ground the discussion some more, we imagined two example explanations we want to compare. Because the task is comparative (which of the participants do you admit to the ICU), we chose two contrastive explanations. First, counterfactual explanations that change the input such that the model rates Person B as more urgent than Person A. Second, feature attribution that show how relevant each input was for the AI’s decision to assigning more urgency to Person A than Person B. In general, our evaluation example focuses on local explanations.

Because medical background knowledge is crucial to properly interpret the explanations we would aim to recruit healthcare professionals for our user study. In particular, for our use case, we would try to recruit nurses.

At first, the participants are given a description of the task and of the AI and Explanations methods that they will see during the study. [optional:] They will also get a pre-questionnaire about their demographics, medical expertise and previous experience with AI and XAI.

Afterwards, the participants will be given X decision tasks where they are told that one ICU bed opened up and they have to decide which of 10 patients they admit. First, they have to decide by themself, without AI support, which patient they admit. After that, they see an AI Recommendation for the urgency ranking of the particpants. Depending on the condition, they will also see an explanation here.

After X decisions, they will [optional: move to the understanding task and then] get a pos-questionnaire.

We envision a between subject design.

• $\mathrm{■}$

  Baseline Condition: These participants only see the recommendation of the AI.

• $\mathrm{■}$

  Explanation Condition 1: These participants see the AI recommendation together with the first explanation method – in our case counterfactual explanations.

• $\mathrm{■}$

  Explanation Condition 2: These participants see the AI recommendation together with the second explanation method – in our case contrastive feature attribution.

We will measure two tothree observed (or objective) metrics:

• $\mathrm{■}$

  Performance: Did nurses/doctors select the correct patient? And also interesting: how far is the distance from the 1st rank is the admitted person (relative performance). To incoporate the possibility of the AI making mistakes we will assume a somewhat realistic accuracy of 80%. That means that in 80% of the example decision, the AI will be correct. We do not choose a lower rate, since it might unrealistically bias the participants against the AI. However, in addition to the general performance, we will also reported individual performance for the group of correct and incorrect AI predictions.

• $\mathrm{■}$

  Appropriate AI Reliance: To what extent did people deviate towards the AI advice, given that they gave their initial estimate, then received the AI’s advice, and based on the latter, decided to comply (or not – or to what extent) with the AI’s estimate (“weight on advice”)

  Appropriate reliance = AI was recommending patient #1 ranking, and the user was following it (final user estimate = AI estimate = best) Underreliance $=$ AI was recommending patient #1 ranking but the user did not follow it (final user estimate $\ne$ AI estimate & AI estimate = best)

  Overreliance $=$ was NOT recommending patient #1 ranking and the user followed it (final user estimate AI estimate & AI estimate $\ne$ best)

  Special case = doctor’s initial estimate = best = AI estimate → “reinforced” appropriate reliance

• $\mathrm{■}$

  Observed Understanding [Optional, not as crucial for this task]: After the X decisions, we could add a prediction task as proposed by [10]. In this task, participants will see Y additional examples. Here, they will only see the input and the explanation (or no explanation for the baseline). Based on this they have to predict the urgency ranking of the AI. Depending on how good they are at predicting the AI will be used to judge their understanding of the AI models reasoning.

Additonally, we will measure several self-reported (subjective) measures in the post-questionnaire:

• $\mathrm{■}$

  Self-reported Trust: Trust in Automation (e.g., Perceived Competence, → Madsen & Gregor, 2000)

• $\mathrm{■}$

  Perceived Understanding

• $\mathrm{■}$

  Perceived Helpfulness

• $\mathrm{■}$

  Satisfaction with Explanation

• $\mathrm{■}$

  User experience questionnaire? (Maybe)

• $\mathrm{■}$

  Task Load / Perceived Accomplishment