Agents, simulated users and humans: an analysis of performance and behaviour

David Maxwell, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

9 Citations (Scopus)
115 Downloads (Pure)

Abstract

Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naive configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably different from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.
Original languageEnglish
Title of host publicationCIKM '16 : Proceedings of the 25th ACM International on Conference on Information and Knowledge Management
Place of PublicationNew York, NY.
Pages731-740
Number of pages10
DOIs
Publication statusPublished - 24 Oct 2016

Keywords

  • agents
  • autonomous agents
  • user modeling
  • continuation strategies
  • interactive information retrieval
  • querying strategies
  • simulation
  • stopping strategies

Cite this