Conversational Agents: A Framework for Evaluation (CAFE)

Christine Bauer
Li Chen
Nicola Ferro
Norbert Fuhr

The framework starts from the assumption that a CONIAC system will be able to (i) interactwith users more naturally and seamlessly, (ii) guide a user through the process of refining and clarifying their needs, (iii) aid decision-makingby making personalized recommendations and information while being able to explain them, and (iv) generate, retrieve and summarize relevant information .

CAFE distinguishes six major elements of an evaluation design:

• $\mathrm{■}$

  Stakeholder goals. Stakeholders of a CONIAC system may have diverse goals that might or might not be directly accessible to system designers or evaluators and must often be implicitly inferred in evaluation. CONIAC systems might also have multiple goals ranging from end users having (in-)direct information needs, to platforms deploying CONIAC systems interested in content usage, user engagement, impression generation, and user retention, to name a few.

• $\mathrm{■}$

  Tasks. CONIAC involves tasks characterized by an information need (which may be specific or rather vague), human involvement, goal orientation, and mixed initiative between the user and the system. While some tasks and information needs may benefit from introducing a conversationally competent system, others may not, depending on the complexity of the task or need.

• $\mathrm{■}$

  User aspects. When developing an evaluation framework for CONIAC systems, it is crucial to consider user-specific aspects, such as preferences, specialized needs, expertise types, and background characteristics, which may make conversational systems more beneficial than non-conversational alternatives.

• $\mathrm{■}$

  Criteria. The scope of evaluation can range from single-turn interactions to entire conversations and long-term system usage, each requiring different criteria for assessment. Additionally, the temporal dimension, which examines how the system’s performance changes over time, is a critical factor that can intersect with both stationary and dynamic properties. Criteria may be system-centric, user-centric, or both. The former regard hardware and software aspects like e. g. efficiency, accuracy, comprehensiveness, and verifiability. For the latter, we can distinguish between conversation-oriented (like e. g. adaptability, coherence, fluency), content-oriented (like e. g. continuance, controllability, perceived accuracy, understandability), and consequences-oriented measures (like e. g. addiction, benevolence, decision quality, confidence, trust).

• $\mathrm{■}$

  Methodology. In addition to the standard distinction of user-focused and system-focused methodologies, our evaluation framework categorizes evaluation methodologies also according to the employed time model – a dimension especially relevant for CONIAC. This dimension ranges from stationary methodologies like single-interaction experiments to methodologies like controlled lab studies that allow for continuous measurements such as physiological ones.

• $\mathrm{■}$

  Measures. Finally, we allow for measures that typically focus on the system’s ability to provide accurate, relevant, and timely information during interactions. Measures include objective measures of effectiveness and subjective notions such as perceived effectiveness or user satisfaction (e. g., self-reported satisfaction). By incorporating both objective as well as subjective (self-reported) measures, evaluators can better understand the system’s strengths and areas for improvement.

When designing an evaluation, the first step is to identify the stakeholders and their goals that need to be addressed. Based on the goals, the user tasks to be studied in the evaluation have to be defined, as well as the user aspects to be considered. The central element of an evaluation are the criteria to be focused on, which can be determined by the stakeholder goals. The chosen criteria restrict the range of possible evaluation methods (e. g. any user-centric criterion requires the involvement of actual users in the evaluation procedure). Finally, an appropriate measure has to be defined for any quantitative criterion.

We thank Schloss Dagstuhl for hosting us.

• $\mathrm{■}$

  Avishek Anand – TU Delft – Delft, NL

• $\mathrm{■}$

  Christine Bauer – Paris Lodron University – Salzburg, AT

• $\mathrm{■}$

  Timo Breuer – TH Köln, DE

• $\mathrm{■}$

  Li Chen – Hong Kong Baptist University, HK

• $\mathrm{■}$

  Guglielmo Faggioli – University of Padua, IT

• $\mathrm{■}$

  Nicola Ferro – University of Padua – Padova, IT

• $\mathrm{■}$

  Ophir Frieder – Georgetwon University – Washington, DC, US

• $\mathrm{■}$

  Norbert Fuhr – Universität Duisburg-Essen – Duisburg, DE

• $\mathrm{■}$

  Hideo Joho – University of Tsukuba – Ibaraki, JP

• $\mathrm{■}$

  Jussi Karlgren – Silo AI – Helsinki, FI

• $\mathrm{■}$

  Johannes Kiesel – Bauhaus-Universität Weimar, DE

• $\mathrm{■}$

  Bart Knijnenburg – Clemson University, SC, US

• $\mathrm{■}$

  Lien Michiels – imec-SMIT, Vrije Universiteit Brussel, BE & University of Antwerp, BE

• $\mathrm{■}$

  Andrea Papenmeier – University of Twente – Enschede, NL

• $\mathrm{■}$

  Maria Soledad Pera – TU Delft, NL

• $\mathrm{■}$

  Aldo Lipani – University College London, UK

• $\mathrm{■}$

  Mark Sanderson – RMIT University – Melbourne, AU

• $\mathrm{■}$

  Scott Sanner – University of Toronto, CA

• $\mathrm{■}$

  Benno Stein – Bauhaus-Universität Weimar, DE

• $\mathrm{■}$

  Johanne Trippas – RMIT University – Melbourne, AU

• $\mathrm{■}$

  Karin Verspoor – RMIT University – Melbourne, AU

• $\mathrm{■}$

  Martijn C. Willemsen – TU Eindhoven, NL & Jheronimus Academy of Data Science – ’s-Hertogenbosch, NL