Interactive evaluation of conversational agents: reflections on the impact of search task design

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

2 Downloads (Pure)

Abstract

Undertaking an interactive evaluation of goal-oriented conversational agents (CAs) is challenging, it requires the search task to be realistic and relatable while accounting for the user‘s cognitive limitations. In the current paper we discuss findings of two Wizard of Oz studies and provide our reflections regarding the impact of different interactive search task designs on participants’ performance,
satisfaction and cognitive workload. In the first study, we tasked participants with finding a cheapest flight that met a certain departure time. In the second study we added an additional criterion: ‘travel time’ and asked participants to find a fight option that offered a good trade-off between price and travel time. We found that using search tasks where participants need to decide between several competing search criteria (price vs. time) led to a higher search involvement and lower variance in usability and cognitive workload ratings between different CAs. We hope that our results will provoke discussion on how to make the evaluation of voice-only goal-oriented CAs more reliable and ecologically valid.
Original languageEnglish
Title of host publicationICTIR '20 - Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval
Place of PublicationNew York, NY.
Pages85–88
Number of pages4
DOIs
Publication statusPublished - 14 Sep 2020
EventACM SIGIR International Conference on the Theory of Information Retrieval 2020 - Stavanger, Norway, Stavanger, Norway
Duration: 14 Sep 202018 Sep 2020
Conference number: 6
https://ictir2020.org

Conference

ConferenceACM SIGIR International Conference on the Theory of Information Retrieval 2020
Abbreviated titleICTIR 2020
CountryNorway
CityStavanger
Period14/09/2018/09/20
Internet address

Keywords

  • conversational search
  • performance evaluation
  • user study
  • search task design
  • information retrieval

Fingerprint Dive into the research topics of 'Interactive evaluation of conversational agents: reflections on the impact of search task design'. Together they form a unique fingerprint.

  • Cite this

    Dubiel, M., Halvey, M., Azzopardi, L., & Daronnat, S. (2020). Interactive evaluation of conversational agents: reflections on the impact of search task design. In ICTIR '20 - Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval (pp. 85–88). https://doi.org/10.1145/3409256.3409814