2026-07-17T13:20:46Z https://drops.dagstuhl.de/oai

oai:drops-oai.dagstuhl.de:22994 2026-04-20T12:47:19Z ddc:004 open_access

Conversational Agents: A Framework for Evaluation (CAFE) (Dagstuhl Perspectives Workshop 24352) Bauer, Christine Chen, Li Ferro, Nicola Fuhr, Norbert Conversational Agents Evaluation Information Access Dagstuhl Perspectives Workshop This report documents the program and the outcomes of the Dagstuhl Perspectives Workshop 24352, "Conversational Agents: A Framework for Evaluation (CAFE)", which brought together 22 distinguished researchers and practitioners from 12 countries. In this workshop, a new framework for the evaluation of conversational information access systems was developed, consisting of six major components: 1) goals of the system’s stakeholders, 2) user tasks to be studied in the evaluation, 3) aspects of the users carrying out the tasks, 4) evaluation criteria to be considered, 5) evaluation methodology to be applied, and 6) measures for the quantitative criteria chosen. An evaluation design begins with identifying the stakeholders, whose goals determine the criteria. Tasks and evaluation methodology should be chosen according to these decisions. Schloss Dagstuhl – Leibniz-Zentrum für Informatik Christine Bauer and Li Chen and Nicola Ferro and Norbert Fuhr 2025 Is Part Of Dagstuhl Reports, Volume 14, Issue 8 (2025) Article Text doc-type:ResearchArticle publishedVersion application/pdf doi:10.4230/DagRep.14.8.53 urn:nbn:de:0030-drops-229940 https://drops.dagstuhl.de/entities/document/10.4230/DagRep.14.8.53 eng https://creativecommons.org/licenses/by/4.0/legalcode