A comparison of user and system query performance predictions

Claudia Hauff, Diane Kelly, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

19 Citations (Scopus)

Abstract

Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.
Original languageEnglish
Title of host publicationCIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management
Place of PublicationNew York, NY, USA
Pages979-988
Number of pages10
DOIs
Publication statusPublished - 26 Oct 2010
Externally publishedYes

Keywords

  • query performance prediction
  • user ratings
  • query suggestions

Cite this

Hauff, C., Kelly, D., & Azzopardi, L. (2010). A comparison of user and system query performance predictions. In CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management (pp. 979-988). https://doi.org/10.1145/1871437.1871562