Evaluating Embodied Conversational Agents in Collaborative Virtual Environments

Abstract

There are currently no evaluation methods 
specific to ECAs in CVEs and traditional 
evaluation methods are limited in their 
applicability and consequently unlikely to 
address the full range of aspects now inherent in
such systems. We argue that a combination of 
controlled experimentation, quasi-experiments,
review-based evaluation and heuristic expert
reviews is needed. To operationalise these 
traditional evaluation methods the concept of 
presence was deployed, and it was argued that 
presence as a cognitive variable can be measured
and that such a measure can be a key indicator of 
the usability of ECAs in CVEs. Presence measures 
can be administered within controlled experiments 
and quasi-experiments to test certain aspects of 
the system. Such experiments might turn out 
particularly useful as a means of selecting 
between two or more design options and it is 
argued that issues concerning ECAs in CVEs can be
meaningfully evaluated by comparing subjects’
experience of presence. Further, although 
implementation issues were not the primary 
concern of this study, the strength and 
shortcomings of the prototype agent were 
evaluated as secondary variables within that 
experiment. A set of criteria developed for the 
evaluation of the strengths and shortcomings of 
the current prototype agent are partly based on 
Nielsen’s general usability guidelines and partly
on a set of heuristics proposed for non-embodied conversational systems.