Agents, simulated users and humans

an analysis of performance and behaviour

David Maxwell, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

9 Citations (Scopus)
112 Downloads (Pure)

Abstract

Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naive configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably different from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.
Original languageEnglish
Title of host publicationProceedings of the 25th ACM International on Conference on Information and Knowledge Management
Place of PublicationNew York, NY
Pages731-740
Number of pages10
DOIs
Publication statusPublished - 24 Oct 2016

Fingerprint

performance
realism
simulation
recourse
information retrieval
search engine
pragmatics
innovation
lack
costs

Keywords

  • agents
  • autonomous agents
  • user modeling
  • continuation strategies
  • interactive information retrieval
  • querying strategies
  • simulation
  • stopping strategies

Cite this

Maxwell, D., & Azzopardi, L. (2016). Agents, simulated users and humans: an analysis of performance and behaviour. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (pp. 731-740). New York, NY. https://doi.org/10.1145/2983323.2983805
Maxwell, David ; Azzopardi, Leif. / Agents, simulated users and humans : an analysis of performance and behaviour. Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. New York, NY, 2016. pp. 731-740
@inproceedings{4be32169681a4650969b64a9de976150,
title = "Agents, simulated users and humans: an analysis of performance and behaviour",
abstract = "Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naive configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably different from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.",
keywords = "agents, autonomous agents, user modeling, continuation strategies, interactive information retrieval, querying strategies, simulation, stopping strategies",
author = "David Maxwell and Leif Azzopardi",
note = "{\circledC} 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management http://dx.doi.org/10.1145/2983323.2983805.",
year = "2016",
month = "10",
day = "24",
doi = "10.1145/2983323.2983805",
language = "English",
isbn = "9781450340731",
pages = "731--740",
booktitle = "Proceedings of the 25th ACM International on Conference on Information and Knowledge Management",

}

Maxwell, D & Azzopardi, L 2016, Agents, simulated users and humans: an analysis of performance and behaviour. in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. New York, NY, pp. 731-740. https://doi.org/10.1145/2983323.2983805

Agents, simulated users and humans : an analysis of performance and behaviour. / Maxwell, David; Azzopardi, Leif.

Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. New York, NY, 2016. p. 731-740.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Agents, simulated users and humans

T2 - an analysis of performance and behaviour

AU - Maxwell, David

AU - Azzopardi, Leif

N1 - © 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management http://dx.doi.org/10.1145/2983323.2983805.

PY - 2016/10/24

Y1 - 2016/10/24

N2 - Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naive configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably different from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.

AB - Most of the current models that are used to simulate users in Interactive Information Retrieval (IIR) lack realism and agency. Such models generally make decisions in a stochastic manner, without recourse to the actual information encountered or the underlying information need. In this paper, we develop a more sophisticated model of the user that includes their cognitive state within the simulation. The cognitive state maintains data about what the simulated user knows, has done and has seen, along with representations of what it considers attractive and relevant. Decisions to inspect or judge are then made based upon the simulated user's current state, rather than stochastically. In the context of ad-hoc topic retrieval, we evaluate the quality of the simulated users and agents by comparing their behaviour and performance against 48 human subjects under the same conditions, topics, time constraints, costs and search engine. Our findings show that while naive configurations of simulated users and agents substantially outperform our human subjects, their search behaviour is notably different from actual searchers. However, more sophisticated search agents can be tuned to act more like actual searchers providing greater realism. This innovation advances the state of the art in simulation, from simulated users towards autonomous agents. It provides a much needed step forward enabling the creation of more realistic simulations, while also motivating the development of more advanced cognitive agents and tools to help support and augment human searchers. Future work will focus not only on the pragmatics of tuning and training such agents for topic retrieval, but will also look at developing agents for other tasks and contexts such as collaborative search and slow search.

KW - agents

KW - autonomous agents

KW - user modeling

KW - continuation strategies

KW - interactive information retrieval

KW - querying strategies

KW - simulation

KW - stopping strategies

UR - http://www.cikm2016.org/

U2 - 10.1145/2983323.2983805

DO - 10.1145/2983323.2983805

M3 - Conference contribution book

SN - 9781450340731

SP - 731

EP - 740

BT - Proceedings of the 25th ACM International on Conference on Information and Knowledge Management

CY - New York, NY

ER -

Maxwell D, Azzopardi L. Agents, simulated users and humans: an analysis of performance and behaviour. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. New York, NY. 2016. p. 731-740 https://doi.org/10.1145/2983323.2983805