DagRep.14.8.53.pdf
- Filesize: 2 MB
- 6 pages
Document
Conversational Agents: A Framework for Evaluation (CAFE) (Dagstuhl Perspectives Workshop 24352)
Authors and all authors of the abstracts in this report
-
Part of:
Issue:
Dagstuhl Reports, Volume 14, Issue 8
Volume: Dagstuhl Reports, Volume 14
Journal: Dagstuhl Reports (DagRep) - License:
Creative Commons Attribution 4.0 International license
- Publication Date: 2025-03-13
Cite As Get BibTex
Christine Bauer, Li Chen, Nicola Ferro, and Norbert Fuhr. Conversational Agents: A Framework for Evaluation (CAFE) (Dagstuhl Perspectives Workshop 24352). In Dagstuhl Reports, Volume 14, Issue 8, pp. 53-58, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025)
https://doi.org/10.4230/DagRep.14.8.53
Abstract
This report documents the program and the outcomes of the Dagstuhl Perspectives Workshop 24352, "Conversational Agents: A Framework for Evaluation (CAFE)", which brought together 22 distinguished researchers and practitioners from 12 countries. In this workshop, a new framework for the evaluation of conversational information access systems was developed, consisting of six major components: 1) goals of the system’s stakeholders, 2) user tasks to be studied in the evaluation, 3) aspects of the users carrying out the tasks, 4) evaluation criteria to be considered, 5) evaluation methodology to be applied, and 6) measures for the quantitative criteria chosen. An evaluation design begins with identifying the stakeholders, whose goals determine the criteria. Tasks and evaluation methodology should be chosen according to these decisions.
Subject Classification
ACM Subject Classification
- Information systems → Information retrieval
- Information systems → Recommender systems
- Computing methodologies → Natural language processing
Keywords
- Conversational Agents
- Evaluation
- Information Access
- Dagstuhl Perspectives Workshop
Metrics
- Access Statistics
-
Total Accesses (updated on a weekly basis)
0PDF Downloads