Document Open Access Logo

Conversational Agents: A Framework for Evaluation (CAFE) (Dagstuhl Perspectives Workshop 24352)

Authors Christine Bauer, Li Chen, Nicola Ferro, Norbert Fuhr, Avishek Anand, Timo Breuer, Guglielmo Faggioli, Ophir Frieder, Hideo Joho, Jussi Karlgren, Johannes Kiesel, Bart P. Knijnenburg, Aldo Lipani, Lien Michiels, Andrea Papenmeier, Maria Soledad Pera, Mark Sanderson, Scott Sanner, Benno Stein, Johanne R. Trippas, Karin Verspoor, Martijn C. Willemsen

PDF

File

Thumbnail PDF
PDF
DagMan.11.1.19.pdf
  • Filesize: 17.29 MB
  • 49 pages

Document Identifiers

Subject Classification

ACM Subject Classification
  • Information systems → Information retrieval
  • Information systems → Recommender systems
  • Computing methodologies → Natural language processing
Keywords
  • Conversational Agents
  • Evaluation
  • Information Access

Metrics

Abstract

During the workshop, we deeply discussed what CONversational Information ACcess (CONIAC) is and its unique features, proposing a world model abstracting it, and defined the Conversational Agents Framework for Evaluation (CAFE) for the evaluation of CONIAC systems, consisting of six major components: 1) goals of the system’s stakeholders, 2) user tasks to be studied in the evaluation, 3) aspects of the users carrying out the tasks, 4) evaluation criteria to be considered, 5) evaluation methodology to be applied, and 6) measures for the quantitative criteria chosen.

Cite As Get BibTex

Christine Bauer, Li Chen, Nicola Ferro, Norbert Fuhr, Avishek Anand, Timo Breuer, Guglielmo Faggioli, Ophir Frieder, Hideo Joho, Jussi Karlgren, Johannes Kiesel, Bart P. Knijnenburg, Aldo Lipani, Lien Michiels, Andrea Papenmeier, Maria Soledad Pera, Mark Sanderson, Scott Sanner, Benno Stein, Johanne R. Trippas, Karin Verspoor, and Martijn C. Willemsen. Conversational Agents: A Framework for Evaluation (CAFE) (Dagstuhl Perspectives Workshop 24352). In Dagstuhl Manifestos, Volume 11, Issue 1, pp. 19-67, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2025) https://doi.org/10.4230/DagMan.11.1.19

Author Details

Christine Bauer
  • University of Salzburg, AT
Li Chen
  • Hong Kong Baptist University, HK, China
Nicola Ferro
  • University of Padua, IT
Norbert Fuhr
  • Universität Duisburg-Essen, DE
Avishek Anand
  • TU Delft, NL
Timo Breuer
  • TH Köln, DE
Guglielmo Faggioli
  • University of Padua, IT
Ophir Frieder
  • Georgetown University, USA
Hideo Joho
  • University of Tsukuba, JP
Jussi Karlgren
  • Silo AI, FI
Johannes Kiesel
  • Bauhaus-Universität Weimar, DE
Bart P. Knijnenburg
  • Clemson University, USA
Aldo Lipani
  • University College London, UK
Lien Michiels
  • imec-SMIT, Vrije Universiteit Brussel & University of Antwerp, BE
Andrea Papenmeier
  • University of Twente, NL
Maria Soledad Pera
  • TU Delft, NL
Mark Sanderson
  • RMIT University, AU
Scott Sanner
  • University of Toronto, CA
Benno Stein
  • Bauhaus-Universität Weimar, DE
Johanne R. Trippas
  • RMIT University, AU
Karin Verspoor
  • RMIT University, AU
Martijn C. Willemsen
  • TU Eindhoven & JADS, NL

Acknowledgements

We thank Schloß Dagstuhl for hosting us. Icons are taken from the Noun Project. This work was partially supported by CAMEO, PRIN 2022 n. 2022ZLL7MW, and by the Excellence in Digital Sciences and Interdisciplinary Technologies (EXDIGIT) project, funded by Land Salzburg under grant number 20204-WISS/263/6-6022.

References

  1. Jafar Afzali, Aleksander Mark Drzewiecki, Krisztian Balog, and Shuo Zhang. Usersimcrs: A user simulation toolkit for evaluating conversational recommender systems. In Tat-Seng Chua, Hady W. Lauw, Luo Si, Evimaria Terzi, and Panayiotis Tsaparas, editors, Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, pages 1160-1163. Association for Computing Machinery, 2023. URL: https://doi.org/10.1145/3539597.3573029.
  2. AI HLEG. Ethics guidelines for trustworthy AI, 2019. URL: https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai.
  3. Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffery Dalton, and Leif Azzopardi. TREC iKAT 2023: The interactive knowledge assistance track overview. CoRR, abs/2401.01330, 2024. URL: https://doi.org/10.48550/arXiv.2401.01330.
  4. Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffrey Dalton, and Leif Azzopardi. TREC iKAT 2023: A test collection for evaluating conversational and interactive knowledge assistants. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors, Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, pages 819-829. Association for Computing Machinery, 2024. URL: https://doi.org/10.1145/3626772.3657860.
  5. James Allan, Eunsol Choi, Daniel P. Lopresti, and Hamed Zamani. Future of Information Retrieval Research in the Age of Generative AI. Technical report, Computing Research Association (CRA), Washington, DC, USA, December 2024. CCC Workshop Report. URL: https://doi.org/10.48550/arXiv.2412.02043.
  6. Garrett Allen, Katherine Landau Wright, Jerry Alan Fails, Casey Kennington, and Maria Soledad Pera. Multi-perspective learning to rank to support children’s information seeking in the classroom. In 2023 IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2023, pages 311-317. IEEE, 2023. URL: https://doi.org/10.1109/WI-IAT59888.2023.00050.
  7. Avishek Anand, Lawrence Cavedon, Hideo Joho, Mark Sanderson, and Benno Stein. Conversational Search (Dagstuhl Seminar 19461). Dagstuhl Reports, 9(11):34-83, 2020. URL: https://doi.org/10.4230/DagRep.9.11.34.
  8. Avishek Anand, Venktesh V, Abhijit Anand, and Vinay Setty. Query understanding in the age of large language models, 2023. URL: https://doi.org/10.48550/arXiv.2306.16004.
  9. Ion Madrazo Azpiazu, Nevena Dragovic, Maria Soledad Pera, and Jerry Alan Fails. Online searching and learning: YUM and other search tools for children and teachers. Information Retrieval Journal, 20(5):524-545, 2017. URL: https://doi.org/10.1007/s10791-017-9310-1.
  10. Leif Azzopardi, Charles L. A. Clarke, Paul B. Kantor, Bhaskar Mitra, Johanne R. Trippas, Zhaochun Ren, Mohammad Aliannejadi, Negar Arabzadeh, Raman Chandrasekar, Maarten de Rijke, Panagiotis Eustratiadis, William R. Hersh, Jin Huang, Evangelos Kanoulas, Jasmin Kareem, Yongkang Li, Simon Lupart, Kidist Amde Mekonnen, Adam Roegiest, Ian Soboroff, Fabrizio Silvestri, Suzan Verberne, David Vos, Eugene Yang, and Yuyue Zhao. Report on the search futures workshop at ECIR 2024. SIGIR Forum, 58(1):1-41, 2024. URL: https://doi.org/10.1145/3687273.3687288.
  11. Krisztian Balog. Conversational ai from an information retrieval perspective: Remaining challenges and a case for user simulation. In Proceedings of the Second International Conference on Design of Experimental Search & Information REtrieval Systems. CEUR, 2021. URL: https://ceur-ws.org/Vol-2950/paper-03.pdf.
  12. Christine Bauer and Eva Zangerle. Leveraging multi-method evaluation for multi-stakeholder settings. In Oren Sar Shalom, Dietmar Jannach, and Ido Guy, editors, 1st Workshop on the Impact of Recommender Systems, co-located with 13th ACM Conference on Recommender Systems (ACM RecSys 2019), volume 2462 of ImpactRS ‘19, 2019. URL: https://ceur-ws.org/Vol-2462/short3.pdf.
  13. Thomas Beckers and Dennis Korbar. Using eye-tracking for the evaluation of interactive information retrieval. In Shlomo Geva, Jaap Kamps, Ralf Schenkel, and Andrew Trotman, editors, Comparative Evaluation of Focused Retrieval - 9th International Workshop of the Inititative for the Evaluation of XML Retrieval (INEX 2010), volume 6932 of Lecture Notes in Computer Science, pages 236-240. Springer, 2010. URL: https://doi.org/10.1007/978-3-642-23577-1_21.
  14. Nicholas J. Belkin, Colleen Cool, Diane Kelly, S-J Lin, SY Park, Jose Perez-Carballo, and Cynthia Sikora. Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing & Management, 37(3):403-434, 2001. URL: https://doi.org/10.1016/S0306-4573(00)00055-8.
  15. Nicolas J. Belkin. Helping people find what they don't know. Communications of the ACM, 43(8):58-61, August 2000. URL: https://doi.org/10.1145/345124.345143.
  16. James Bennett and Stan Lanning. The Netflix Prize. In In KDD Cup and Workshop in conjunction with KDD, 2007. Google Scholar
  17. Timothy W Bickmore, Dina Utami, Robin Matsuyama, and Michael K Paasche-Orlow. Improving access to online health information with conversational agents: A randomized controlled experiment. Journal of Medical Internet Research, 18(1):e1, January 2016. URL: https://doi.org/10.2196/jmir.5239.
  18. Craig Boutilier. A POMDP formulation of preference elicitation problems. In Eighteenth National Conference on Artificial Intelligence, pages 239-246. American Association for Artificial Intelligence, 2002. URL: https://doi.org/10.5555/777092.777132.
  19. Robin Burke, Gediminas Adomavicius, Toine Bogers, Tommaso Di Noia, Dominik Kowald, Julia Neidhardt, Özlem Özgöbek, Maria Soledad Pera, Nava Tintarev, and Jürgen Ziegler. De-centering the (traditional) user: Multistakeholder evaluation of recommender systems. Int. J. Hum. Comput. Stud., 203:103560, 2025. URL: https://doi.org/10.1016/J.IJHCS.2025.103560.
  20. Katriina Byström and Kalervo Järvelin. Task complexity affects information seeking and use. Information Processing & Management, 31(2):191-213, 1995. URL: https://doi.org/10.1016/0306-4573(95)80035-R.
  21. Wanling Cai, Yucheng Jin, and Li Chen. Impacts of personal characteristics on user trust in conversational recommender systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI '22, New York, NY, USA, 2022. Association for Computing Machinery. URL: https://doi.org/10.1145/3491102.3517471.
  22. Donald J. Campbell. Task complexity: A review and analysis. Academy of Management Review, 13(1):40-52, 1988. URL: https://doi.org/10.5465/amr.1988.4306775.
  23. Claudio Carpineto and Giovanni Romano. A survey of automatic query expansion in information retrieval. ACM Computing Surveys, 44(1), 2012. URL: https://doi.org/10.1145/2071389.2071390.
  24. Scott Carter and Jennifer Mankoff. When participants do the capturing: the role of media in diary studies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '05, pages 899-908, New York, NY, USA, 2005. Association for Computing Machinery. URL: https://doi.org/10.1145/1054972.1055098.
  25. John P Caughlin. A multiple goals theory of personal relationships: Conceptual integration and program overview. Journal of Social and Personal Relationships, 27(6):824-848, 2010. URL: https://doi.org/10.1177/0265407510373262.
  26. Yi Chang and Hongbo Deng. Query understanding for search engines. Springer, 2020. URL: https://doi.org/10.1007/978-3-030-58334-7.
  27. Kathy Charmaz. Grounded theory. In Jonathan A. Smith, editor, Qualitative psychology: A practical guide to research methods, volume 3, pages 53-84. SAGE, 2015. Google Scholar
  28. Nathan N. Cheek and Barry Schwartz. On the meaning and measurement of maximization. Judgment and Decision Making, 11(2):126-146, 2016. URL: https://doi.org/10.1017/S1930297500007257.
  29. Edgar F. Codd. Seven steps to rendezvous with the casual user. In J. W. Klimbie and K. L. Koffeman, editors, Proceedings of the IFIP Working Conference Data Base Management, pages 179-200. North-Holland, 1974. Google Scholar
  30. Allan Collins, John Seely Brown, and Susan E Newman. Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In Knowing, learning, and instruction, pages 453-494. Routledge, 2018. URL: https://doi.org/10.4324/9781315044408-14.
  31. Sebastian Cross, Ahmed Mourad, Guido Zuccon, and Bevan Koopman. Search engines vs. symptom checkers: A comparison of their effectiveness for online health advice. In Proceedings of the Web Conference 2021, WWW '21, pages 206-216, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3442381.3450140.
  32. Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. Wizard of Oz studies: why and how. In Proceedings of the 1st International Conference on Intelligent User Interfaces, IUI '93, pages 193-200, New York, NY, USA, 1993. Association for Computing Machinery. URL: https://doi.org/10.1145/169891.169968.
  33. J. Dalton, C. Xiong, and J. Callan. CAsT 2020: The Conversational Assistance Track Overview. In Ellen M. Voorhees and A. Ellis, editors, The Twenty-Ninth Text REtrieval Conference Proceedings (TREC 2020). National Institute of Standards and Technology (NIST), Special Publication 1266, Washington, USA, 2021. Google Scholar
  34. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. TREC CAsT 2019: The conversational assistance track overview, 2020. URL: https://arxiv.org/abs/2003.13624.
  35. Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. TREC CAsT 2021: The conversational assistance track overview. In Ian Soboroff and Angela Ellis, editors, Proceedings of the Thirtieth Text REtrieval Conference (TREC 2021), volume 500-335 of NIST Special Publication. National Institute of Standards and Technology (NIST), 2021. URL: https://trec.nist.gov/pubs/trec30/papers/Overview-CAsT.pdf.
  36. Brenda Dervin. From the mind’s eye of the user: The sense-making qualitative-quantitative methodology. In Jack D. Glazier and Ronald R. Powell, editors, Qualitative research in information management, pages 61-84. Libraries Unlimited, 1992. Google Scholar
  37. Robert F DeVellis. Scale development: Theory and applications. SAGE, Thousand Oaks, Calif., 2011. Google Scholar
  38. Tommaso Di Noia, Guglielmo Faggioli, Marco Ferrante, Nicola Ferro, Fedelucio Narducci, Raffaele Perego, and Giuseppe Santucci. CAMEO: Fostering joint conversational search and recommendation. In M. Atzori, P. Ciaccia, M. Ceci, F. Mandreoli, D. Malerba, M. Sanguinetti, A. Pellicani, and F. Motta, editors, Proc. 32nd Italian Symposium on Advanced Database Systems (SEBD 2024), pages 290-301. CEUR Workshop Proceedings (CEUR-WS.org), ISSN 1613-0073, https://ceur-ws.org/Vol-3741/, 2024. URL: https://ceur-ws.org/Vol-3741/paper33.pdf.
  39. Michael D. Ekstrand, Lex Beattie, Maria Soledad Pera, and Henriette Cramer. Not just algorithms: Strategically addressing consumer impacts in information retrieval. In European Conference on Information Retrieval, pages 314-335, Berlin, Heidelberg, 2024. Springer. URL: https://doi.org/10.1007/978-3-031-56066-8_25.
  40. Michael D. Ekstrand, Anubrata Das, Robin Burke, and Fernando Diaz. Fairness in information access systems. Foundations and Trends\textsuperscript® in Information Retrieval, 16(1-2):1-177, 2022. URL: https://doi.org/10.1561/1500000079.
  41. Norman E. Fenton and James Bieman. Software Metrics: A Rigorous & Practical Approach. Chapman and Hall/CRC, USA, 3rd edition, 2014. Google Scholar
  42. Nicola Ferro. What Happened in CLEF... For Another While? In L. Goeuriot, P. Mulhem, G. Quénot, D. Schwab, L. Soulier, G. M. Di Nunzio, P. Galuščáková, A. García Seco de Herrera, G. Faggioli, and Nicola Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024) - Part I, pages 3-57. Lecture Notes in Computer Science (LNCS) 14958, Springer, Heidelberg, Germany, 2024. URL: https://doi.org/10.1007/978-3-031-71736-9_1.
  43. Nicola Ferro and Carol Peters, editors. Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF, volume 41 of The Information Retrieval Series. Springer International Publishing, Germany, 2019. Google Scholar
  44. Anne P. J. Frederix. Investigating the effect of using personalized critiques in a decision-making tool. Master’s thesis, Eindhoven University of Technology, 2021. URL: https://pure.tue.nl/ws/files/174846002/Master_Thesis_A.P.J._Frederix_1396676.pdf.
  45. Norbert Fuhr. Some Common Mistakes In IR Evaluation, And How They Can Be Avoided. SIGIR Forum, 51(3):32-41, December 2017. URL: https://doi.org/10.1145/3190580.3190586.
  46. Jianfeng Gao, Michel Galley, and Lihong Li. Neural approaches to conversational AI. Foundations and Trends in Information Retrieval (FnTIR), 13(2-3):27-298, February 2019. URL: https://doi.org/10.1561/1500000074.
  47. Lukas Gienapp, Harrisen Scells, Niklas Deckers, Janek Bevendorff, Shuai Wang, Johannes Kiesel, Shahbaz Syed, Maik Fröbe, Guido Zuccon, Benno Stein, Matthias Hagen, and Martin Potthast. Evaluating Generative Ad Hoc Information Retrieval. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors, 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, pages 1916-1929. Association for Computing Machinery, July 2024. URL: https://doi.org/10.1145/3626772.3657849.
  48. Selwyn Goldsmith. Universal design. Routledge, 2007. Google Scholar
  49. Barbara J Grosz and Candace L Sidner. Attention, intentions, and the structure of discourse. Computational linguistics, 12(3):175-204, 1986. Google Scholar
  50. Asela Gunawardana, Guy Shani, and Sivan Yogev. Evaluating recommender systems. In Francesco Ricci, Lior Rokach, and Bracha Shapira, editors, Recommender Systems Handbook, pages 547-601. Springer US, New York, NY, 2022. URL: https://doi.org/10.1007/978-1-0716-2197-4_15.
  51. Donna K. Harman and Ellen M. Voorhees, editors. TREC. Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (MA), USA, 2005. Google Scholar
  52. David Hawking. Challenges in enterprise search. In Proceedings of the 15th Australasian Database Conference - Volume 27, ADC '04, pages 15-24, AUS, 2004. Australian Computer Society, Inc. URL: http://crpit.scem.westernsydney.edu.au/abstracts/CRPITV27Hawking1.html.
  53. Jerry R Hobbs. Coherence and coreference. Cognitive Science, 3(1):67-90, 1979. URL: https://doi.org/10.1207/s15516709cog0301_4.
  54. Jerry R Hobbs. Towards an understanding of coherence in discourse. In Strategies for natural language processing, pages 223-243. Psychology Press, 2014. Google Scholar
  55. Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 159-166. Association for Computing Machinery, 1999. URL: https://doi.org/10.1145/302979.303030.
  56. Chih-Wei Hsu, Martin Mladenov, Ofer Meshi, James Pine, Hubert Pham, Shane Li, Xujian Liang, Anton Polishko, Li Yang, Ben Scheetz, and Craig Boutilier. Minimizing live experiments in recommender systems: User simulation to evaluate preference elicitation policies. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors, Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 2925-2929. Association for Computing Machinery, 2024. URL: https://doi.org/10.1145/3626772.3661358.
  57. Andrea Iovine, Fedelucio Narducci, and Marco de Gemmis. A dataset of real dialogues for conversational recommender systems. In Raffaella Bernardi, Roberto Navigli, and Giovanni Semeraro, editors, Proceedings of the Sixth Italian Conference on Computational Linguistics, Bari, Italy, November 13-15, 2019, volume 2481 of CEUR Workshop Proceedings. CEUR-WS.org, 2019. URL: https://ceur-ws.org/Vol-2481/paper37.pdf.
  58. Chris Segrin James Price Dillard and Janie M. Harden. Primary and secondary goals in the production of interpersonal influence messages. Communication Monographs, 56(1):19-38, 1989. URL: https://doi.org/10.1080/03637758909390247.
  59. Dietmar Jannach. Evaluating conversational recommender systems: A landscape of research. Artificial Intelligence Review, 56(3):2365-2400, 2023. URL: https://doi.org/10.1007/s10462-022-10229-x.
  60. Dietmar Jannach and Christine Bauer. Escaping the McNamara Fallacy: toward more impactful recommender systems research. AI Magazine, 41(4):79-95, 2020. URL: https://doi.org/10.1609/aimag.v41i4.5312.
  61. Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. A survey on conversational recommender systems. ACM Computing Surveys, 54(5):105:1-36, 2022. URL: https://doi.org/10.1145/3453154.
  62. Yucheng Jin, Li Chen, Wanling Cai, and Xianglin Zhao. Crs-que: A user-centric evaluation framework for conversational recommender systems. ACM Transactions on Recommender Systems, 2(1), March 2024. URL: https://doi.org/10.1145/3631534.
  63. Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P. de Vries, Jeff Dalton, and Faegheh Hasibi. Doing personal LAPS: llm-augmented dialogue construction for personalized multi-session conversational search. In Grace Hui Yang, Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang, editors, Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 796-806. Association for Computing Machinery, 2024. URL: https://doi.org/10.1145/3626772.3657815.
  64. Patricia K. Kahr, Gerrit Rooks, Chris Snijders, and Martijn C. Willemsen. The trust recovery journey: The effect of timing of errors on the willingness to follow AI advice. In Proceedings of the 29th International Conference on Intelligent User Interfaces, IUI '24, pages 609-622, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3640543.3645167.
  65. Patricia K. Kahr, Gerrit Rooks, Martijn C. Willemsen, and Chris C. P. Snijders. Understanding trust and reliance development in ai advice: Assessing model accuracy, model explanations, and experiences from previous interactions. ACM Transactions on Interactive Intelligent Systems, 14(4), December 2024. URL: https://doi.org/10.1145/3686164.
  66. Diane Kelly. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends® in Information Retrieval, 3(1-2):1-224, 2009. URL: https://doi.org/10.1561/1500000012.
  67. Kimiya Keyvan and Jimmy Xiangji Huang. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys, 55(6):105:1-36, June 2022. URL: https://doi.org/10.1145/3534965.
  68. Johannes Kiesel, Arefeh Bahrami, Benno Stein, Avishek Anand, and Matthias Hagen. Clarifying false memories in voice-based search. In Leif Azzopardi, Martin Halvey, Ian Ruthven, Hideo Joho, Vanessa Murdock, and Pernilla Qvarfordt, editors, Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, CHIIR 2019, Glasgow, Scotland, UK, March 10-14, 2019, pages 331-335. ACM, 2019. URL: https://doi.org/10.1145/3295750.3298961.
  69. Johannes Kiesel, Marcel Gohsen, Nailia Mirzakhmedova, Matthias Hagen, and Benno Stein. Who Will Evaluate the Evaluators? Exploring the Gen-IR User Simulation Space. In Lorraine Goeuriot, Philippe Mulhem, Georges Quénot, Didier Schwab, Giorgio Maria Di Nunzio, Laure Soulier, Petra Galuscakova, Alba Garcia Seco Herrera, Guglielmo Faggioli, and Nicola Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction. 15th International Conference of the CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Berlin Heidelberg New York, September 2024. Springer. URL: https://doi.org/10.1007/978-3-031-71736-9_11.
  70. Johannes Kiesel, Lars Meyer, Martin Potthast, and Benno Stein. Meta-information in conversational search. ACM Transactions on Information Systems, 39(4), August 2021. URL: https://doi.org/10.1145/3468868.
  71. Min-Sun Kim. Cross-cultural comparisons of the perceived importance of conversational constraints. Human Communication Research, 21(1):128-151, 1994. URL: https://doi.org/10.1111/j.1468-2958.1994.tb00343.x.
  72. Bart P. Knijnenburg and Martijn C. Willemsen. Evaluating Recommender Systems with User Experiments. In Francesco Ricci, Lior Rokach, and Bracha Shapira, editors, Recommender Systems Handbook, pages 309-352. Springer US, 2015. URL: https://doi.org/10.1007/978-1-4899-7637-6_9.
  73. Bart P. Knijnenburg and Martijn C. Willemsen. Inferring capabilities of intelligent agents from their external traits. ACM Transactions on Interactive Intelligent Systems, 6(4), November 2016. URL: https://doi.org/10.1145/2963106.
  74. Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction, 22(4-5):441-504, 2012. URL: https://doi.org/10.1007/s11257-011-9118-4.
  75. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M Henne. Controlled experiments on the web: survey and practical guide. Data mining and knowledge discovery, 18:140-181, 2009. URL: https://doi.org/10.1007/s10618-008-0114-1.
  76. Carol C. Kuhlthau. Inside the search process: Information seeking from the user’s perspective. Journal of the American Society for Information Science, 42(5):361-371, 1991. URL: https://doi.org/10.1002/(SICI)1097-4571(199106)42:5<361::AID-ASI6>3.0.CO;2-%23.
  77. Monica Landoni, Mohammad Aliannejadi, Theo Huibers, Emiliana Murgia, and Maria Soledad Pera. Right way, right time: Towards a better comprehension of young students’ needs when looking for relevant search results. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, pages 256-261, New York, NY, USA, 2021. Association for Computing Machinery. URL: https://doi.org/10.1145/3450613.3456843.
  78. Monica Landoni, Davide Matteri, Emiliana Murgia, Theo Huibers, and Maria Soledad Pera. Sonny, cerca! evaluating the impact of using a vocal assistant to search at school. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9-12, 2019, Proceedings 10, pages 101-113. Springer, 2019. URL: https://doi.org/10.1007/978-3-030-28577-7_6.
  79. Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. Towards deep conversational recommendations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL: https://proceedings.neurips.cc/paper_files/paper/2018/file/800de15c79c8d840f4e78d3af937d4d4-Paper.pdf.
  80. Jinggui Liang, Yuxia Wu, Yuan Fang, Hao Fei, and Lizi Liao. A Survey of Ontology Expansion for Conversational Understanding. In H. Bouamor, J. Pino, and K. Bali, editors, Proc. of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 18111 - -18127. Association for Computational Linguistics, USA, 2023. URL: https://doi.org/10.18653/v1/2024.emnlp-main.1006.
  81. Tingting Liang, Chenxin Jin, Lingzhi Wang, Wenqi Fan, Congying Xia, Kai Chen, and Yuyu Yin. LLM-REDIAL: A large-scale dataset for conversational recommender systems created from user behaviors with LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 8926-8939, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/2024.findings-acl.529.
  82. Aldo Lipani, Ben Carterette, and Emine Yilmaz. How am I doing?: Evaluating conversational search systems offline. ACM Transactions on Information Systems, 39(4), August 2021. URL: https://doi.org/10.1145/3451160.
  83. Zeyang Liu, Ke Zhou, and Max L Wilson. Meta-evaluation of conversational search evaluation metrics. ACM Transactions on Information Systems, 39(4), September 2021. URL: https://doi.org/10.1145/3445029.
  84. William C. Mann and Sandra A. Thompson. Rhetorical structure theory: Toward a functional theory of text organization. Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243-281, 1988. URL: https://doi.org/10.1515/text.1.1988.8.3.243.
  85. Gary Marchionini. Exploratory Search: From Finding to Understanding. Commun. ACM, 49(4):41-46, 2006. URL: https://doi.org/10.1145/1121949.1121979.
  86. D. Harrison McKnight, Vivek Choudhury, and Charles Kacmar. Developing and Validating Trust Measures for e-Commerce: An Integrative Typology. Information Systems Research, 13(3):334-359, 2002. URL: https://doi.org/10.1287/isre.13.3.334.81.
  87. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: a human generated MAchine Reading COmprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d'Avila Garcez, and Greg Wayne, editors, Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), volume 1773 of CEUR Workshop Proceedings. CEUR-WS.org, 2016. URL: https://ceur-ws.org/Vol-1773/CoCoNIPS_2016_paper9.pdf.
  88. Elaine Ostroff. Universal design: an evolving paradigm. In Universal Design Handbook, chapter 1. McGraw-Hill, New York, NY, USA, 2011. Google Scholar
  89. Paul Owoicho, Jeff Dalton, Mohammad Aliannejadi, Leif Azzopardi, Johanne R. Trippas, and Svitlana Vakulenko. TREC CAsT 2022: Going beyond user ask and system retrieve with initiative and response generation. In Ian Soboroff and Angela Ellis, editors, Proceedings of the Thirty-First Text REtrieval Conference, (TREC 2022), volume 500-338 of NIST Special Publication. National Institute of Standards and Technology (NIST), 2022. URL: https://trec.nist.gov/pubs/trec31/papers/Overview_cast.pdf.
  90. Patrick Cheong-Iao Pang, Shanton Chang, Karin Verspoor, and Jon Pearce. Designing health websites based on users' web-based information-seeking behaviors: A mixed-method observational study. Journal Medical Internet Research, 18(6):e145, June 2016. URL: https://doi.org/10.2196/jmir.5661.
  91. Patrick Cheong-Iao Pang, Karin Verspoor, Shanton Chang, and Jon Pearce. Conceptualising health information seeking behaviours and exploratory search: result of a qualitative study. Health and Technology, 5:45-55, 2015. Google Scholar
  92. Veljko Pejovic, Neal Lathia, Cecilia Mascolo, and Mirco Musolesi. Mobile-Based Experience Sampling for Behaviour Research. In Marko Tkalčič, Berardina De Carolis, Marco de Gemmis, Ante Odić, and Andrej Košir, editors, Emotions and Personality in Personalized Services: Models, Evaluation and Applications, pages 141-161. Springer International Publishing, Cham, 2016. URL: https://doi.org/10.1007/978-3-319-31413-6_8.
  93. G Penha and C Hauff. Challenges in the evaluation of conversational search systems. In KDD 2020 Workshop on Conversational Systems Towards Mainstream Adoption, KDD-Converse 2020, volume 2666, 2020. Google Scholar
  94. Gustavo Penha, Alexandru Balan, and Claudia Hauff. Introducing MANtIS: a novel multi-domain information seeking dialogues dataset. CoRR, abs/1912.04639, 2019. URL: https://arxiv.org/abs/1912.04639.
  95. Maria Soledad Pera, Theo Huibers, Emiliana Murgia, and Monica Landoni. Some things never change: Overcoming persistent challenges in children IR. In Nicola Ferro, Maria Maistro, Gabriella Pasi, Omar Alonso, Andrew Trotman, and Suzan Verberne, editors, Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2025, Padua, Italy, July 13-18, 2025, pages 3107-3109. ACM, 2025. URL: https://doi.org/10.1145/3726302.3730270.
  96. Filip Radlinski and Nick Craswell. A theoretical framework for conversational search. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, CHIIR '17, pages 117-126, New York, NY, USA, 2017. Association for Computing Machinery. URL: https://doi.org/10.1145/3020165.3020183.
  97. Hossein A. Rahmani, Xi Wang, Yue Feng, Qiang Zhang, Emine Yilmaz, and Aldo Lipani. A Survey on Asking Clarification Questions Datasets in Conversational Systems. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proc. of the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), pages 2698-2716. Association for Computational Linguistics, USA, 2023. URL: https://doi.org/10.18653/v1/2020.acl-main.363.
  98. Robert Ramberg and Klas Karlgren. Fostering superficial learning. Journal of Computer Assisted Learning, 14(2):120-129, 1998. URL: https://doi.org/10.1046/j.1365-2729.1998.1420120.x.
  99. Tetsuya Sakai. SWAN: A generic framework for auditing textual conversational systems. CoRR, abs/2305.08290, 2023. URL: https://doi.org/10.48550/arXiv.2305.08290.
  100. Tetsuya Sakai, Douglas W. Oard, and Noriko Kando, editors. Evaluating Information Retrieval and Access Tasks - NTCIR’s Legacy of Research Impact, volume 43 of The Information Retrieval Series. Springer International Publishing, Germany, 2021. Google Scholar
  101. Mark Sanderson. Test collection based evaluation of information retrieval systems. Foundations and Trends\texttrademark in Information Retrieval, 4(4):247-375, 2010. URL: https://doi.org/10.1561/1500000009.
  102. Michael Schulte-Mecklenbeck, Joseph G. Johnson, Ulf Böckenholt, Daniel G. Goldstein, J. Edward Russo, Nicolette J. Sullivan, and Martijn C. Willemsen. Process-tracing methods in decision making: On growing up in the 70s. Current Directions in Psychological Science, 26(5):442-450, 2017. URL: https://doi.org/10.1177/0963721417708229.
  103. John R. Searle. Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, 1969. URL: https://doi.org/10.1017/CBO9781139173438.
  104. Ivan Sekulic, Silvia Terragni, Victor Guimarães, Nghia Khau, Bruna Guedes, Modestas Filipavicius, Andre Ferreira Manso, and Roland Mathis. Reliable LLM-based user simulator for task-oriented dialogue systems. In Yvette Graham, Qun Liu, Gerasimos Lampouras, Ignacio Iacobacci, Sinead Madden, Haider Khalid, and Rameez Qureshi, editors, Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat, SCI-CHAT 2024, pages 19-35, St. Julians, Malta, March 2024. Association for Computational Linguistics. URL: https://aclanthology.org/2024.scichat-1.3.
  105. C. Shah and E. M. Bender. Envisioning Information Access Systems: What Makes for Good Tools and a Healthy Web? ACM Transactions on the Web (TWEB), 18(3):33:1-33:24, August 2024. URL: https://doi.org/10.1145/3649468.
  106. Jack Sidnell and Tanya Stivers. The Handbook of Conversation Analysis. Blackwell, 2012. URL: https://doi.org/10.1002/9781118325001.
  107. Stefan Sitter and Adelheit Stein. Modeling the illocutionary aspects of information-seeking dialogues. Information Processing & Management, 28(2):165-180, 1992. URL: https://doi.org/10.1016/0306-4573(92)90044-Z.
  108. Neil R Smalheiser. Literature-based discovery: Beyond the ABCs. Journal of the American Society for Information Science and Technology, 63(2):218-224, 2012. URL: https://doi.org/10.1002/asi.21599.
  109. Neil R. Smalheiser. Rediscovering don swanson:the past, present and future of literature-based discovery. Journal of Data and Information Science, 2(4):43-64, 2017. URL: https://doi.org/10.1515/jdis-2017-0019.
  110. Damiano Spina, Johanne R. Trippas, Paul Thomas, Hideo Joho, Katriina Byström, Leigh Clark, Nick Craswell, Mary Czerwinski, David Elsweiler, Alexander Frummet, Souvick Ghosh, Johannes Kiesel, Irene Lopatovska, Daniel McDuff, Selina Meyer, Ahmed Mourad, Paul Owoicho, Sachin Pathiyan Cherumanal, Daniel Russell, and Laurianne Sitbon. Report on the future conversations workshop at CHIIR 2021. SIGIR Forum, 55(1), July 2021. URL: https://doi.org/10.1145/3476415.3476421.
  111. Amanda Spink, Minsoo Park, Bernard J. Jansen, and Jan Pedersen. Multitasking during Web search sessions. Information Processing & Management, 42(1):264-275, 2006. URL: https://doi.org/10.1016/j.ipm.2004.10.004.
  112. Manfred Stede and Jodi Schneider. Argumentation Mining. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers, 2018. URL: https://doi.org/10.2200/S00883ED1V01Y201811HLT040.
  113. Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 3645-3650. Association for Computational Linguistics, 2019. URL: https://doi.org/10.18653/V1/P19-1355.
  114. Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. Improving multi-turn dialogue modelling with utterance ReWriter. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 22-31, Florence, Italy, July 2019. Association for Computational Linguistics. URL: https://doi.org/10.18653/v1/P19-1003.
  115. Özge Sürer, Robin Burke, and Edward C Malthouse. Multistakeholder recommendation with provider constraints. In Proceedings of the 12th ACM Conference on Recommender Systems, RecSys '18, pages 54-62, New York, NY, USA, 2018. Association for Computing Machinery. URL: https://doi.org/10.1145/3240323.3240350.
  116. Don R Swanson. Two medical literatures that are logically but not bibliographically connected. Journal of the American Society for Information Science, 38(4):228-233, 1987. URL: https://doi.org/10.1002/(SICI)1097-4571(198707)38:4<228::AID-ASI2>3.0.CO;2-G.
  117. Robert S. Taylor. The process of asking questions. American Documentation, 13(4):391-396, 1962. URL: https://doi.org/10.1002/asi.5090130405.
  118. Jaime Teevan, Christine Alvarado, Mark S. Ackerman, and David R. Karger. The perfect search engine is not enough: a study of orienteering behavior in directed search. In Elizabeth Dykstra-Erickson and Manfred Tscheligi, editors, Proceedings of the 2004 Conference on Human Factors in Computing Systems (CHI 2004), pages 415-422. Association for Computing Machinery, 2004. URL: https://doi.org/10.1145/985692.985745.
  119. Helma Torkamaan, Mohammad Tahaei, Stefan Buijsman, Ziang Xiao, Daricia Wilkinson, and Bart P. Knijnenburg. The Role of Human-Centered AI in User Modeling, Adaptation, and Personalization—Models, Frameworks, and Paradigms. In Bruce Ferwerda, Mark Graus, Panagiotis Germanakos, and Marko Tkalčič, editors, A Human-Centered Perspective of Intelligent Personalized Environments and Systems, pages 43-84. Springer Nature Switzerland, Cham, 2024. URL: https://doi.org/10.1007/978-3-031-55109-3_2.
  120. Johanne R Trippas, Sara Fahad Dawood Al Lawati, Joel Mackenzie, and Luke Gallagher. What do users really ask large language models? an initial log analysis of Google Bard interactions in the wild. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3626772.3657914.
  121. Johanne R Trippas, Luke Gallagher, and Joel Mackenzie. Re-evaluating the command-and-control paradigm in conversational search interactions. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM '24, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3627673.3679588.
  122. Johanne R. Trippas, Damiano Spina, and Falk Scholer. Adapting generative information retrieval systems to users, tasks, and scenarios. In Ryen W. White and Chirag Shah, editors, Information Access in the Era of Generative AI, pages 73-109. Springer Nature Switzerland, Cham, Switzerland, 2025. URL: https://doi.org/10.1007/978-3-031-73147-1_4.
  123. Johanne R Trippas, Damiano Spina, Paul Thomas, Mark Sanderson, Hideo Joho, and Lawrence Cavedon. Towards a model for spoken conversational search. Information Processing & Management, 57(2), 2020. URL: https://doi.org/10.1016/J.IPM.2019.102162.
  124. Robin Ungruh and Maria Soledad Pera. Ah, that’s the great puzzle: On the quest of a holistic understanding of the harms of recommender systems on children. CoRR, abs/2405.02050, 2024. URL: https://doi.org/10.48550/arXiv.2405.02050.
  125. Svitlana Vakulenko, Kate Revoredo, Claudio Di Ciccio, and Maarten de Rijke. QRFA: A data-driven model of information-seeking dialogues. In Leif Azzopardi, Benno Stein, Norbert Fuhr, Philipp Mayr, Claudia Hauff, and Djoerd Hiemstra, editors, Advances in Information Retrieval - 41st European Conference on IR Research (ECIR 2019), volume 11437 of Lecture Notes in Computer Science, pages 541-557. Springer, 2019. URL: https://doi.org/10.1007/978-3-030-15712-8_35.
  126. R. W. White and C. Shah, editors. Information Access in the Era of Generative AI, volume 51 of The Information Retrieval Series. Springer International Publishing, Germany, 2025. Google Scholar
  127. Ryen W. White. Tasks, copilots, and the future of search: A keynote at sigir 2023. SIGIR Forum, 57(2), January 2024. URL: https://doi.org/10.1145/3642979.3642985.
  128. Ryen W White and Resa A Roth. Exploratory search: Beyond the query-response paradigm. Springer, Cham, Switzerland, 2009. URL: https://doi.org/10.1007/978-3-031-02260-9.
  129. John M Wiemann. Explication and test of a model of communicative competence. Human communication research, 3(3):195-213, 1977. URL: https://doi.org/10.1111/j.1468-2958.1977.tb00518.x.
  130. Martijn C Willemsen and Eric J Johnson. Visiting the decision factory: observing cognition with MouselabWEB and other information acquisition methods. In Michael Schulte-Mecklenbeck, Anton Kuehberger, and JohnsonJoseph G., editors, A handbook of process tracing methods for decision research: a critical review and user’s guide, pages 21-42. Psychology Press, 2011. URL: http://repository.tue.nl/699163.
  131. Robert L Winkler and Allan H Murphy. Experiments in the laboratory and the real world. Organizational Behavior and Human Performance, 10(2):252-270, 1973. URL: https://doi.org/10.1016/0030-5073(73)90017-2.
  132. A Wayne Wymore. Systems Engineering Methodology for Interdisciplinary Teams. Wiley, New York, NY, USA, 1976. Google Scholar
  133. Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. Conversational information seeking. Foundations and Trends® in Information Retrieval, 17(3-4):244-456, August 2023. URL: https://doi.org/10.1561/1500000081.
  134. Eva Zangerle and Christine Bauer. Evaluating recommender systems: Survey and framework. ACM Computing Surveys, 55(8), 2022. URL: https://doi.org/10.1145/3556536.
  135. Savvas Zannettou, Olivia Nemes Nemeth, Oshrat Ayalon, Angelica Goetzen, Krishna P. Gummadi, Elissa M. Redmiles, and Franziska Roesner. Analyzing user engagement with tiktok’s short format video recommendations using data donations. In Florian 'Floyd' Mueller, Penny Kyburz, Julie R. Williamson, Corina Sas, Max L. Wilson, Phoebe O. Toups Dugas, and Irina Shklovski, editors, Proceedings of the CHI Conference on Human Factors in Computing Systems, CHI 2024, pages 731:1-731:16. Association for Computing Machinery, 2024. URL: https://doi.org/10.1145/3613904.3642433.
  136. Saber Zerhoudi, Sebastian Günther, Kim Plassmeier, Timo Borst, Christin Seifert, Matthias Hagen, and Michael Granitzer. The SimIIR 2.0 framework: User types, markov model-based interaction simulation, and advanced query generation. In Mohammad Al Hasan and Li Xiong, editors, Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM 2022, pages 4661-4666. Association for Computing Machinery, 2022. URL: https://doi.org/10.1145/3511808.3557711.
  137. Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, and Jiaxin Mao. USimAgent: Large language models for simulating search users. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '24, pages 2687-2692, New York, NY, USA, 2024. Association for Computing Machinery. URL: https://doi.org/10.1145/3626772.3657963.
Any Issues?
X

Feedback on the Current Page

CAPTCHA

Thanks for your feedback!

Feedback submitted to Dagstuhl Publishing

Could not send message

Please try again later or send an E-mail
© 2023-2025 Schloss Dagstuhl – LZI GmbH About DROPS Imprint Privacy Contact