Abstract
| Language | English |
|---|---|
| Number of pages | 10 |
| DOIs | |
| Publication status | Published - 8 Jul 2018 |
| Event | 41st International ACM SIGIR Conference on Research and Development in Information Retrieval - Ann Arbor, United States Duration: 8 Jul 2018 → 12 Jul 2018 |
Conference
| Conference | 41st International ACM SIGIR Conference on Research and Development in Information Retrieval |
|---|---|
| Abbreviated title | ACM SIGIR 2018 |
| Country | United States |
| City | Ann Arbor |
| Period | 8/07/18 → 12/07/18 |
Fingerprint
Keywords
- information systems
- retrieval effectiveness
- human-centred computing
Cite this
}
Measuring the utility of search engine result pages : an information foraging based measure. / Azzopardi, Leif; Thomas, Paul; Craswell, Nick.
2018. Paper presented at 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, Ann Arbor, United States.Research output: Contribution to conference › Paper
TY - CONF
T1 - Measuring the utility of search engine result pages
T2 - an information foraging based measure
AU - Azzopardi, Leif
AU - Thomas, Paul
AU - Craswell, Nick
PY - 2018/7/8
Y1 - 2018/7/8
N2 - Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).
AB - Web Search Engine Result Pages (SERPs) are complex responses to queries, containing many heterogeneous result elements (web results, advertisements, and specialised “answers”) positioned in a variety of layouts. This poses numerous challenges when trying to measure the quality of a SERP because standard measures were designed for homogeneous ranked lists. In this paper, we aim to measure the utility and cost of SERPs. To ground this work we adopt the C/W/L framework which enables a direct comparison between different measures in the same units of measurement, i.e. expected (total) utility and cost. Within this framework, we propose a new measure based on information foraging theory, which can account for the heterogeneity of elements, through different costs, and which naturally motivates the development of a user stopping model that adapts behaviour depending on the rate of gain. This directly connects models of how people search with how we measure search, providing a number of new dimensions in which to investigate and evaluate user behaviour and performance. We perform an analysis over 1000 popular queries issued to a major search engine, and report the aggregate utility experienced by users over time. Then in an comparison against common measures, we show that the proposed foraging based measure provides a more accurate reflection of the utility and of observed behaviours (stopping rank and time spent).
KW - information systems
KW - retrieval effectiveness
KW - human-centred computing
UR - http://sigir.org/sigir2018/
U2 - 10.1145/3209978.3210027
DO - 10.1145/3209978.3210027
M3 - Paper
ER -