License
When quoting this document, please refer to the following
URN: urn:nbn:de:0030-drops-4609
URL: http://drops.dagstuhl.de/opus/volltexte/2006/460/
Go to the corresponding Portal


Gerhard, Michael

Evaluating Embodied Conversational Agents in Collaborative Virtual Environments

pdf-format:
Document 1.pdf (57 KB)


Abstract

There are currently no evaluation methods specific to ECAs in CVEs and traditional evaluation methods are limited in their applicability and consequently unlikely to address the full range of aspects now inherent in such systems. We argue that a combination of controlled experimentation, quasi-experiments, review-based evaluation and heuristic expert reviews is needed. To operationalise these traditional evaluation methods the concept of presence was deployed, and it was argued that presence as a cognitive variable can be measured and that such a measure can be a key indicator of the usability of ECAs in CVEs. Presence measures can be administered within controlled experiments and quasi-experiments to test certain aspects of the system. Such experiments might turn out particularly useful as a means of selecting between two or more design options and it is argued that issues concerning ECAs in CVEs can be meaningfully evaluated by comparing subjects’ experience of presence. Further, although implementation issues were not the primary concern of this study, the strength and shortcomings of the prototype agent were evaluated as secondary variables within that experiment. A set of criteria developed for the evaluation of the strengths and shortcomings of the current prototype agent are partly based on Nielsen’s general usability guidelines and partly on a set of heuristics proposed for non-embodied conversational systems.

BibTeX - Entry

@InProceedings{gerhard:DSP:2006:460,
  author =	{Michael Gerhard},
  title =	{Evaluating Embodied Conversational Agents in Collaborative Virtual Environments},
  booktitle =	{Evaluating Embodied Conversational Agents},
  year =	{2006},
  editor =	{Zsofia Ruttkay and Elisabeth Andr{\'e} and W. Lewis Johnson and Catherine Pelachaud},
  number =	{04121},
  series =	{Dagstuhl Seminar Proceedings},
  ISSN =	{1862-4405},
  publisher =	{Internationales Begegnungs- und Forschungszentrum f{\"u}r Informatik (IBFI), Schloss Dagstuhl, Germany},
  address =	{Dagstuhl, Germany},
  URL =		{http://drops.dagstuhl.de/opus/volltexte/2006/460},
  annote =	{Keywords: Embodied Conversational Agents,Collaborative Virtual Environments, Presence}
}

Keywords: Embodied Conversational Agents,Collaborative Virtual Environments, Presence
Seminar: 04121 - Evaluating Embodied Conversational Agents
Issue Date: 2006
Date of publication: 14.03.2006


DROPS-Home | Fulltext Search | Imprint Published by LZI