Abstract
Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).
Original language | English |
---|---|
Pages | 605-614 |
Number of pages | 10 |
DOIs | |
Publication status | Published - 8 Jul 2018 |
Event | 41st International ACM SIGIR Conference on Research and Development in Information Retrieval - Ann Arbor, United States Duration: 8 Jul 2018 → 12 Jul 2018 |
Conference
Conference | 41st International ACM SIGIR Conference on Research and Development in Information Retrieval |
---|---|
Abbreviated title | ACM SIGIR 2018 |
Country/Territory | United States |
City | Ann Arbor |
Period | 8/07/18 → 12/07/18 |
Keywords
- information systems
- retrieval effectiveness
- human-centred computing