Measuring the utility of search engine result pages: an information foraging based measure

Leif Azzopardi, Paul Thomas, Nick Craswell

Research output: Contribution to conferencePaper

19 Citations (Scopus)
25 Downloads (Pure)

Abstract

Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).
Original languageEnglish
Pages605-614
Number of pages10
DOIs
Publication statusPublished - 8 Jul 2018
Event41st International ACM SIGIR Conference on Research and Development in Information Retrieval - Ann Arbor, United States
Duration: 8 Jul 201812 Jul 2018

Conference

Conference41st International ACM SIGIR Conference on Research and Development in Information Retrieval
Abbreviated titleACM SIGIR 2018
CountryUnited States
CityAnn Arbor
Period8/07/1812/07/18

Keywords

  • information systems
  • retrieval effectiveness
  • human-centred computing

Fingerprint Dive into the research topics of 'Measuring the utility of search engine result pages: an information foraging based measure'. Together they form a unique fingerprint.

  • Cite this

    Azzopardi, L., Thomas, P., & Craswell, N. (2018). Measuring the utility of search engine result pages: an information foraging based measure. 605-614. Paper presented at 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, United States. https://doi.org/10.1145/3209978.3210027