Data-driven evaluation metrics for heterogeneous search engine result pages

Leif Azzopardi, Ryen W. White, Paul Thomas, Nick Craswell

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

4 Citations (Scopus)
10 Downloads (Pure)

Abstract

Evaluation metrics for search typically assume items are homoge- neous. However, in the context of web search, this assumption does not hold. Modern search engine result pages (SERPs) are composed of a variety of item types (e.g., news, web, entity, etc.), and their influence on browsing behavior is largely unknown.
In this paper, we perform a large-scale empirical analysis of pop- ular web search queries and investigate how different item types influence how people interact on SERPs. We then infer a user brows- ing model given people’s interactions with SERP items – creating a data-driven metric based on item type. We show that the proposed metric leads to more accurate estimates of: (1) total gain, (2) total time spent, and (3) stopping depth – without requiring extensive parameter tuning or a priori relevance information. These results suggest that item heterogeneity should be accounted for when de- veloping metrics for SERPs. While many open questions remain concerning the applicability and generalizability of data-driven metrics, they do serve as a formal mechanism to link observed user behaviors directly to how performance is measured. From this approach, we can draw new insights regarding the relationship be- tween behavior and performance – and design data-driven metrics based on real user behavior rather than using metrics reliant on some hypothesized model of user browsing behavior.
Original languageEnglish
Title of host publicationCHIIR 2020 - Proceedings of the 2020 Conference on Human Information Interaction and Retrieval
Place of PublicationNew York
Pages213–222
Number of pages10
ISBN (Electronic)9781450368926
DOIs
Publication statusPublished - 14 Mar 2020

Publication series

NameCHIIR 2020 - Proceedings of the 2020 Conference on Human Information Interaction and Retrieval

Keywords

  • information retrieval
  • search engine result pages
  • SERPs
  • web search
  • relevance

Fingerprint Dive into the research topics of 'Data-driven evaluation metrics for heterogeneous search engine result pages'. Together they form a unique fingerprint.

Cite this