DROPS

Document

Explainability in Focus: Advancing Evaluation through Reusable Experiment Design (Dagstuhl Seminar 25142)

Authors: Simone Stumpf, Stefano Teso, and Elizabeth M. Daly

Published in: Dagstuhl Reports, Volume 15, Issue 3 (2025)

Abstract

This report summarizes the outcomes of Dagstuhl Seminar 25142, which convened leading researchers and practitioners to address the pressing challenges in evaluating explainable artificial intelligence (XAI). The seminar focused on developing reusable experimental designs and robust evaluation frameworks that balance technical rigor with human-centered considerations. Key themes included the need for standardized metrics, the contextual relevance of evaluation criteria, and the integration of human understanding, trust, and reliance into assessment methodologies. Through a series of talks, collaborative discussions, and case studies across domains such as healthcare, hiring, and decision support, the seminar identified critical gaps in current XAI evaluation practices and proposed actionable strategies to bridge them. The report presents a refined taxonomy of evaluation criteria, practical guidance for experimental design, and a roadmap for future interdisciplinary collaboration in responsible and transparent AI development.

Cite as

Simone Stumpf, Stefano Teso, and Elizabeth M. Daly. Explainability in Focus: Advancing Evaluation through Reusable Experiment Design (Dagstuhl Seminar 25142). In Dagstuhl Reports, Volume 15, Issue 3, pp. 201-224, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025)

Copy BibTex To Clipboard

@Article{stumpf_et_al:DagRep.15.3.201,
  author =	{Stumpf, Simone and Teso, Stefano and Daly, Elizabeth M.},
  title =	{{Explainability in Focus: Advancing Evaluation through Reusable Experiment Design (Dagstuhl Seminar 25142)}},
  pages =	{201--224},
  journal =	{Dagstuhl Reports},
  ISSN =	{2192-5283},
  year =	{2025},
  volume =	{15},
  number =	{3},
  editor =	{Stumpf, Simone and Teso, Stefano and Daly, Elizabeth M.},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/DagRep.15.3.201},
  URN =		{urn:nbn:de:0030-drops-248935},
  doi =		{10.4230/DagRep.15.3.201},
  annote =	{Keywords: Explainability, Mental Models, interactive machine learning, Experiment Design, Human-centered AI Dagstuhl Seminar}
}

Document

DOI: 10.4230/DagMan.7.1.96

From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442)

Authors: Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, and Justin Zobel

Published in: Dagstuhl Manifestos, Volume 7, Issue 1 (2018)

Abstract

We describe the state-of-the-art in performance modeling and prediction for Information Retrieval (IR), Natural Language Processing (NLP) and Recommender Systems (RecSys) along with its shortcomings and strengths. We present a framework for further research, identifying five major problem areas: understanding measures, performance analysis, making underlying assumptions explicit, identifying application features determining performance, and the development of prediction models describing the relationship between assumptions, features and resulting performance.

Cite as

Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, and Justin Zobel. From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442). In Dagstuhl Manifestos, Volume 7, Issue 1, pp. 96-139, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2018)

Copy BibTex To Clipboard

@Article{ferro_et_al:DagMan.7.1.96,
  author =	{Ferro, Nicola and Fuhr, Norbert and Grefenstette, Gregory and Konstan, Joseph A. and Castells, Pablo and Daly, Elizabeth M. and Declerck, Thierry and Ekstrand, Michael D. and Geyer, Werner and Gonzalo, Julio and Kuflik, Tsvi and Lind\'{e}n, Krister and Magnini, Bernardo and Nie, Jian-Yun and Perego, Raffaele and Shapira, Bracha and Soboroff, Ian and Tintarev, Nava and Verspoor, Karin and Willemsen, Martijn C. and Zobel, Justin},
  title =	{{From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442)}},
  pages =	{96--139},
  journal =	{Dagstuhl Manifestos},
  ISSN =	{2193-2433},
  year =	{2018},
  volume =	{7},
  number =	{1},
  editor =	{Ferro, Nicola and Fuhr, Norbert and Grefenstette, Gregory and Konstan, Joseph A. and Castells, Pablo and Daly, Elizabeth M. and Declerck, Thierry and Ekstrand, Michael D. and Geyer, Werner and Gonzalo, Julio and Kuflik, Tsvi and Lind\'{e}n, Krister and Magnini, Bernardo and Nie, Jian-Yun and Perego, Raffaele and Shapira, Bracha and Soboroff, Ian and Tintarev, Nava and Verspoor, Karin and Willemsen, Martijn C. and Zobel, Justin},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/DagMan.7.1.96},
  URN =		{urn:nbn:de:0030-drops-98987},
  doi =		{10.4230/DagMan.7.1.96},
  annote =	{Keywords: Information Systems, Formal models, Evaluation, Simulation, User Interaction}
}

@Article{ferro_et_al:DagMan.7.1.96,
  author =	{Ferro, Nicola and Fuhr, Norbert and Grefenstette, Gregory and Konstan, Joseph A. and Castells, Pablo and Daly, Elizabeth M. and Declerck, Thierry and Ekstrand, Michael D. and Geyer, Werner and Gonzalo, Julio and Kuflik, Tsvi and Lind\'{e}n, Krister and Magnini, Bernardo and Nie, Jian-Yun and Perego, Raffaele and Shapira, Bracha and Soboroff, Ian and Tintarev, Nava and Verspoor, Karin and Willemsen, Martijn C. and Zobel, Justin},
  title =	{{From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442)}},
  pages =	{96--139},
  journal =	{Dagstuhl Manifestos},
  ISSN =	{2193-2433},
  year =	{2018},
  volume =	{7},
  number =	{1},
  editor =	{Ferro, Nicola and Fuhr, Norbert and Grefenstette, Gregory and Konstan, Joseph A. and Castells, Pablo and Daly, Elizabeth M. and Declerck, Thierry and Ekstrand, Michael D. and Geyer, Werner and Gonzalo, Julio and Kuflik, Tsvi and Lind\'{e}n, Krister and Magnini, Bernardo and Nie, Jian-Yun and Perego, Raffaele and Shapira, Bracha and Soboroff, Ian and Tintarev, Nava and Verspoor, Karin and Willemsen, Martijn C. and Zobel, Justin},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/entities/document/10.4230/DagMan.7.1.96},
  URN =		{urn:nbn:de:0030-drops-98987},
  doi =		{10.4230/DagMan.7.1.96},
  annote =	{Keywords: Information Systems, Formal models, Evaluation, Simulation, User Interaction}
}

Search Results

Documents authored by Daly, Elizabeth M.

Explainability in Focus: Advancing Evaluation through Reusable Experiment Design (Dagstuhl Seminar 25142)

Abstract

Cite as

From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442)

Abstract

Cite as

Thanks for your feedback!

Could not send message