Abstract
Undertaking an interactive evaluation of goal-oriented conversational agents (CAs) is challenging, it requires the search task to be realistic and relatable while accounting for the user‘s cognitive limitations. In the current paper we discuss findings of two Wizard of Oz studies and provide our reflections regarding the impact of different interactive search task designs on participants’ performance,
satisfaction and cognitive workload. In the first study, we tasked participants with finding a cheapest flight that met a certain departure time. In the second study we added an additional criterion: ‘travel time’ and asked participants to find a fight option that offered a good trade-off between price and travel time. We found that using search tasks where participants need to decide between several competing search criteria (price vs. time) led to a higher search involvement and lower variance in usability and cognitive workload ratings between different CAs. We hope that our results will provoke discussion on how to make the evaluation of voice-only goal-oriented CAs more reliable and ecologically valid.
satisfaction and cognitive workload. In the first study, we tasked participants with finding a cheapest flight that met a certain departure time. In the second study we added an additional criterion: ‘travel time’ and asked participants to find a fight option that offered a good trade-off between price and travel time. We found that using search tasks where participants need to decide between several competing search criteria (price vs. time) led to a higher search involvement and lower variance in usability and cognitive workload ratings between different CAs. We hope that our results will provoke discussion on how to make the evaluation of voice-only goal-oriented CAs more reliable and ecologically valid.
| Original language | English |
|---|---|
| Title of host publication | ICTIR 2020 - Proceedings of the 2020 ACM SIGIR International Conference on Theory of Information Retrieval |
| Place of Publication | New York, NY. |
| Pages | 85–88 |
| Number of pages | 4 |
| DOIs | |
| Publication status | Published - 14 Sept 2020 |
| Event | ACM SIGIR International Conference on the Theory of Information Retrieval 2020 - Stavanger, Norway, Stavanger, Norway Duration: 14 Sept 2020 → 18 Sept 2020 Conference number: 6 https://ictir2020.org |
Conference
| Conference | ACM SIGIR International Conference on the Theory of Information Retrieval 2020 |
|---|---|
| Abbreviated title | ICTIR 2020 |
| Country/Territory | Norway |
| City | Stavanger |
| Period | 14/09/20 → 18/09/20 |
| Internet address |
Keywords
- conversational search
- performance evaluation
- user study
- search task design
- information retrieval