Measuring the utility of search engine result pages: an information foraging based measure

Leif Azzopardi, Paul Thomas, Nick Craswell

Research output: Contribution to conferencePaper

Abstract

Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).

Conference

Conference41st International ACM SIGIR Conference on Research and Development in Information Retrieval
Abbreviated titleACM SIGIR 2018
CountryUnited States
CityAnn Arbor
Period8/07/1812/07/18

Fingerprint

Search engines
Costs
Units of measurement
Information theory

Keywords

  • information systems
  • retrieval effectiveness
  • human-centred computing

Cite this

Azzopardi, L., Thomas, P., & Craswell, N. (2018). Measuring the utility of search engine result pages: an information foraging based measure. Paper presented at 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, United States.DOI: 10.1145/3209978.3210027
Azzopardi, Leif ; Thomas, Paul ; Craswell, Nick. / Measuring the utility of search engine result pages : an information foraging based measure. Paper presented at 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, United States.10 p.
@conference{1be4c55089ee4570954b0d4c6e5b3704,
title = "Measuring the utility of search engine result pages: an information foraging based measure",
abstract = "Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).",
keywords = "information systems, retrieval effectiveness, human-centred computing",
author = "Leif Azzopardi and Paul Thomas and Nick Craswell",
year = "2018",
month = "7",
day = "8",
doi = "10.1145/3209978.3210027",
language = "English",
note = "41st International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM SIGIR 2018 ; Conference date: 08-07-2018 Through 12-07-2018",

}

Azzopardi, L, Thomas, P & Craswell, N 2018, 'Measuring the utility of search engine result pages: an information foraging based measure' Paper presented at 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, United States, 8/07/18 - 12/07/18, . DOI: 10.1145/3209978.3210027

Measuring the utility of search engine result pages : an information foraging based measure. / Azzopardi, Leif; Thomas, Paul; Craswell, Nick.

2018. Paper presented at 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, United States.

Research output: Contribution to conferencePaper

TY - CONF

T1 - Measuring the utility of search engine result pages

T2 - an information foraging based measure

AU - Azzopardi,Leif

AU - Thomas,Paul

AU - Craswell,Nick

PY - 2018/7/8

Y1 - 2018/7/8

N2 - Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).

AB - Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).

KW - information systems

KW - retrieval effectiveness

KW - human-centred computing

UR - http://sigir.org/sigir2018/

U2 - 10.1145/3209978.3210027

DO - 10.1145/3209978.3210027

M3 - Paper

ER -

Azzopardi L, Thomas P, Craswell N. Measuring the utility of search engine result pages: an information foraging based measure. 2018. Paper presented at 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, United States. Available from, DOI: 10.1145/3209978.3210027