A comparison of user and system query performance predictions

Claudia Hauff, Diane Kelly, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

17 Citations (Scopus)

Abstract

Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.
LanguageEnglish
Title of host publicationCIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management
Place of PublicationNew York, NY, USA
Pages979-988
Number of pages10
DOIs
Publication statusPublished - 26 Oct 2010
Externally publishedYes

Fingerprint

performance
rating
intuition
evaluation

Keywords

  • query performance prediction
  • user ratings
  • query suggestions

Cite this

Hauff, C., Kelly, D., & Azzopardi, L. (2010). A comparison of user and system query performance predictions. In CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management (pp. 979-988). New York, NY, USA. https://doi.org/10.1145/1871437.1871562
Hauff, Claudia ; Kelly, Diane ; Azzopardi, Leif. / A comparison of user and system query performance predictions. CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management. New York, NY, USA, 2010. pp. 979-988
@inproceedings{73ba8014768a4048a6189d4b9f6e214b,
title = "A comparison of user and system query performance predictions",
abstract = "Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.",
keywords = "query performance prediction, user ratings, query suggestions",
author = "Claudia Hauff and Diane Kelly and Leif Azzopardi",
year = "2010",
month = "10",
day = "26",
doi = "10.1145/1871437.1871562",
language = "English",
isbn = "978-1-4503-0099-5",
pages = "979--988",
booktitle = "CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management",

}

Hauff, C, Kelly, D & Azzopardi, L 2010, A comparison of user and system query performance predictions. in CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management. New York, NY, USA, pp. 979-988. https://doi.org/10.1145/1871437.1871562

A comparison of user and system query performance predictions. / Hauff, Claudia; Kelly, Diane; Azzopardi, Leif.

CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management. New York, NY, USA, 2010. p. 979-988.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - A comparison of user and system query performance predictions

AU - Hauff, Claudia

AU - Kelly, Diane

AU - Azzopardi, Leif

PY - 2010/10/26

Y1 - 2010/10/26

N2 - Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.

AB - Query performance prediction methods are usually applied to estimate the retrieval effectiveness of queries, where the evaluation is largely system sided. However, little work has been conducted to understand query performance prediction from the user's perspective. The question we consider is, whether the predictions of query performance that systems make are in line with the predictions that users make. To this aim, we compare the performance ratings users assign to queries with the performance scores estimated by a range of pre-retrieval and post-retrieval query performance predictors. Two studies are presented that explore the relationship between user ratings and system predictions on two levels: (i) the topic level, and, (ii) the query suggestions level. It is shown that when predicting the performance of query suggestions, user ratings were mostly uncorrelated with system predictions. At the topic level though, where a single query is judged for each information need, we observed moderate correlations between user ratings and a subset of system predictions. As query performance prediction methods are often based on intuitions of how users might rate queries, these findings suggest that such methods are not representative of how users actually rate query suggestions and topics. This motivates further research into understanding the rating process engaged by users, and developing models of query performance prediction in order to bridge the divide between systems and users.

KW - query performance prediction

KW - user ratings

KW - query suggestions

U2 - 10.1145/1871437.1871562

DO - 10.1145/1871437.1871562

M3 - Conference contribution book

SN - 978-1-4503-0099-5

SP - 979

EP - 988

BT - CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management

CY - New York, NY, USA

ER -

Hauff C, Kelly D, Azzopardi L. A comparison of user and system query performance predictions. In CIKM '10 Proceedings of the 19th ACM International Conference on Information and Knowledge Management. New York, NY, USA. 2010. p. 979-988 https://doi.org/10.1145/1871437.1871562