Searching and stopping: an analysis of stopping rules and strategies

David Maxwell, Leif Azzopardi, Kalervo Järvelin, Heikki Keskustalo

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

18 Citations (Scopus)

Abstract

Searching naturally involves stopping points, both at a query level (how far down the ranked list should I go?) and at a session level (how many queries should I issue?). Understanding when searchers stop has been of much interest to the community because it is fundamental to how we evaluate search behaviour and performance. Research has shown that searchers find it difficult to formalise stopping criteria, and typically resort to their intuition of what is "good enough". While various heuristics and stopping criteria have been proposed, little work has investigated how well they perform, and whether searchers actually conform to any of these rules. In this paper, we undertake the first large scale study of stopping rules, investigating how they influence overall session performance, and which rules best match actual stopping behaviour. Our work is focused on stopping at the query level in the context of ad-hoc topic retrieval, where searchers undertake search tasks within a fixed time period. We show that stopping strategies based upon the disgust or frustration point rules - both of which capture a searcher's tolerance to non-relevance - typically result in (i) the best overall performance, and (ii) provide the closest approximation to actual searcher behaviour, although a fixed depth approach also performs remarkably well. Findings from this study have implications regarding how we build measures, and how we conduct simulations of search behaviours.
LanguageEnglish
Title of host publicationCIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management
Place of PublicationNew York, NY, USA
Pages313-322
Number of pages10
DOIs
Publication statusPublished - 17 Oct 2015
Externally publishedYes

Fingerprint

performance
intuition
frustration
tolerance
heuristics
simulation
community
time

Keywords

  • evaluation
  • search behaviour
  • retrieval strategies

Cite this

Maxwell, D., Azzopardi, L., Järvelin, K., & Keskustalo, H. (2015). Searching and stopping: an analysis of stopping rules and strategies. In CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management (pp. 313-322). New York, NY, USA. https://doi.org/10.1145/2806416.2806476
Maxwell, David ; Azzopardi, Leif ; Järvelin, Kalervo ; Keskustalo, Heikki. / Searching and stopping : an analysis of stopping rules and strategies. CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management . New York, NY, USA, 2015. pp. 313-322
@inproceedings{69d0ba01cd844009821d5cfc19a00587,
title = "Searching and stopping: an analysis of stopping rules and strategies",
abstract = "Searching naturally involves stopping points, both at a query level (how far down the ranked list should I go?) and at a session level (how many queries should I issue?). Understanding when searchers stop has been of much interest to the community because it is fundamental to how we evaluate search behaviour and performance. Research has shown that searchers find it difficult to formalise stopping criteria, and typically resort to their intuition of what is {"}good enough{"}. While various heuristics and stopping criteria have been proposed, little work has investigated how well they perform, and whether searchers actually conform to any of these rules. In this paper, we undertake the first large scale study of stopping rules, investigating how they influence overall session performance, and which rules best match actual stopping behaviour. Our work is focused on stopping at the query level in the context of ad-hoc topic retrieval, where searchers undertake search tasks within a fixed time period. We show that stopping strategies based upon the disgust or frustration point rules - both of which capture a searcher's tolerance to non-relevance - typically result in (i) the best overall performance, and (ii) provide the closest approximation to actual searcher behaviour, although a fixed depth approach also performs remarkably well. Findings from this study have implications regarding how we build measures, and how we conduct simulations of search behaviours.",
keywords = "evaluation, search behaviour, retrieval strategies",
author = "David Maxwell and Leif Azzopardi and Kalervo J{\"a}rvelin and Heikki Keskustalo",
year = "2015",
month = "10",
day = "17",
doi = "10.1145/2806416.2806476",
language = "English",
isbn = "978-1-4503-3794-6",
pages = "313--322",
booktitle = "CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management",

}

Maxwell, D, Azzopardi, L, Järvelin, K & Keskustalo, H 2015, Searching and stopping: an analysis of stopping rules and strategies. in CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management . New York, NY, USA, pp. 313-322. https://doi.org/10.1145/2806416.2806476

Searching and stopping : an analysis of stopping rules and strategies. / Maxwell, David; Azzopardi, Leif; Järvelin, Kalervo; Keskustalo, Heikki.

CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management . New York, NY, USA, 2015. p. 313-322.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Searching and stopping

T2 - an analysis of stopping rules and strategies

AU - Maxwell, David

AU - Azzopardi, Leif

AU - Järvelin, Kalervo

AU - Keskustalo, Heikki

PY - 2015/10/17

Y1 - 2015/10/17

N2 - Searching naturally involves stopping points, both at a query level (how far down the ranked list should I go?) and at a session level (how many queries should I issue?). Understanding when searchers stop has been of much interest to the community because it is fundamental to how we evaluate search behaviour and performance. Research has shown that searchers find it difficult to formalise stopping criteria, and typically resort to their intuition of what is "good enough". While various heuristics and stopping criteria have been proposed, little work has investigated how well they perform, and whether searchers actually conform to any of these rules. In this paper, we undertake the first large scale study of stopping rules, investigating how they influence overall session performance, and which rules best match actual stopping behaviour. Our work is focused on stopping at the query level in the context of ad-hoc topic retrieval, where searchers undertake search tasks within a fixed time period. We show that stopping strategies based upon the disgust or frustration point rules - both of which capture a searcher's tolerance to non-relevance - typically result in (i) the best overall performance, and (ii) provide the closest approximation to actual searcher behaviour, although a fixed depth approach also performs remarkably well. Findings from this study have implications regarding how we build measures, and how we conduct simulations of search behaviours.

AB - Searching naturally involves stopping points, both at a query level (how far down the ranked list should I go?) and at a session level (how many queries should I issue?). Understanding when searchers stop has been of much interest to the community because it is fundamental to how we evaluate search behaviour and performance. Research has shown that searchers find it difficult to formalise stopping criteria, and typically resort to their intuition of what is "good enough". While various heuristics and stopping criteria have been proposed, little work has investigated how well they perform, and whether searchers actually conform to any of these rules. In this paper, we undertake the first large scale study of stopping rules, investigating how they influence overall session performance, and which rules best match actual stopping behaviour. Our work is focused on stopping at the query level in the context of ad-hoc topic retrieval, where searchers undertake search tasks within a fixed time period. We show that stopping strategies based upon the disgust or frustration point rules - both of which capture a searcher's tolerance to non-relevance - typically result in (i) the best overall performance, and (ii) provide the closest approximation to actual searcher behaviour, although a fixed depth approach also performs remarkably well. Findings from this study have implications regarding how we build measures, and how we conduct simulations of search behaviours.

KW - evaluation

KW - search behaviour

KW - retrieval strategies

U2 - 10.1145/2806416.2806476

DO - 10.1145/2806416.2806476

M3 - Conference contribution book

SN - 978-1-4503-3794-6

SP - 313

EP - 322

BT - CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

CY - New York, NY, USA

ER -

Maxwell D, Azzopardi L, Järvelin K, Keskustalo H. Searching and stopping: an analysis of stopping rules and strategies. In CIKM '15 Proceedings of the 24th ACM International on Conference on Information and Knowledge Management . New York, NY, USA. 2015. p. 313-322 https://doi.org/10.1145/2806416.2806476