A retrievability analysis: exploring the relationship between retrieval bias and retrieval performance

Colin Wilkie, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

8 Citations (Scopus)

Abstract

Retrievability provides an alternative way to assess an Information Retrieval (IR) system by measuring how easily documents can be retrieved. Retrievability can also be used to determine the level of retrieval bias a system exerts upon a collection of documents. It has been hypothesised that reducing the retrieval bias will lead to improved performance. To date, it has been shown that this hypothesis does not appear to hold on standard retrieval performance measures (MAP and P@10) when exploring the parameter space of a given retrieval model. However, the evidence is limited and confined to only a few models, collections and measures. In this paper, we perform a comprehensive empirical evaluation analysing the relationship between retrieval bias and retrieval performance using several well known retrieval models, five large TREC test collections and ten performance measures (including the recently proposed PRES, Time Biased Gain (TBG) and U-Measure). For traditional relevance based measures (MAP, P@10, MRR, Recall, etc) the correlation between retrieval bias and performance is moderate. However, for TBG and U-Measure, we find that there is strong and significant negative correlations between retrieval bias and performance (i.e as bias drops, performance increases). These findings suggest that for these more sophisticated, user oriented measures the retrievability bias hypothesis tends to hold. The implication is that for these measures, systems can then be tuned using retrieval bias, without recourse to relevance judgements.
LanguageEnglish
Title of host publicationCIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management
Place of PublicationNew York, NY, USA
Pages81-90
Number of pages10
DOIs
Publication statusPublished - 3 Nov 2014
Externally publishedYes

Fingerprint

trend
performance
recourse
information retrieval
evaluation
evidence
time

Keywords

  • effectiveness
  • user measures
  • retrievability

Cite this

Wilkie, C., & Azzopardi, L. (2014). A retrievability analysis: exploring the relationship between retrieval bias and retrieval performance. In CIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management (pp. 81-90). New York, NY, USA. https://doi.org/10.1145/2661829.2661948
Wilkie, Colin ; Azzopardi, Leif. / A retrievability analysis : exploring the relationship between retrieval bias and retrieval performance. CIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management . New York, NY, USA, 2014. pp. 81-90
@inproceedings{f57860f406744ed7836ce6e7fd4d9394,
title = "A retrievability analysis: exploring the relationship between retrieval bias and retrieval performance",
abstract = "Retrievability provides an alternative way to assess an Information Retrieval (IR) system by measuring how easily documents can be retrieved. Retrievability can also be used to determine the level of retrieval bias a system exerts upon a collection of documents. It has been hypothesised that reducing the retrieval bias will lead to improved performance. To date, it has been shown that this hypothesis does not appear to hold on standard retrieval performance measures (MAP and P@10) when exploring the parameter space of a given retrieval model. However, the evidence is limited and confined to only a few models, collections and measures. In this paper, we perform a comprehensive empirical evaluation analysing the relationship between retrieval bias and retrieval performance using several well known retrieval models, five large TREC test collections and ten performance measures (including the recently proposed PRES, Time Biased Gain (TBG) and U-Measure). For traditional relevance based measures (MAP, P@10, MRR, Recall, etc) the correlation between retrieval bias and performance is moderate. However, for TBG and U-Measure, we find that there is strong and significant negative correlations between retrieval bias and performance (i.e as bias drops, performance increases). These findings suggest that for these more sophisticated, user oriented measures the retrievability bias hypothesis tends to hold. The implication is that for these measures, systems can then be tuned using retrieval bias, without recourse to relevance judgements.",
keywords = "effectiveness, user measures, retrievability",
author = "Colin Wilkie and Leif Azzopardi",
year = "2014",
month = "11",
day = "3",
doi = "10.1145/2661829.2661948",
language = "English",
isbn = "978-1-4503-2598-1",
pages = "81--90",
booktitle = "CIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management",

}

Wilkie, C & Azzopardi, L 2014, A retrievability analysis: exploring the relationship between retrieval bias and retrieval performance. in CIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management . New York, NY, USA, pp. 81-90. https://doi.org/10.1145/2661829.2661948

A retrievability analysis : exploring the relationship between retrieval bias and retrieval performance. / Wilkie, Colin; Azzopardi, Leif.

CIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management . New York, NY, USA, 2014. p. 81-90.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - A retrievability analysis

T2 - exploring the relationship between retrieval bias and retrieval performance

AU - Wilkie, Colin

AU - Azzopardi, Leif

PY - 2014/11/3

Y1 - 2014/11/3

N2 - Retrievability provides an alternative way to assess an Information Retrieval (IR) system by measuring how easily documents can be retrieved. Retrievability can also be used to determine the level of retrieval bias a system exerts upon a collection of documents. It has been hypothesised that reducing the retrieval bias will lead to improved performance. To date, it has been shown that this hypothesis does not appear to hold on standard retrieval performance measures (MAP and P@10) when exploring the parameter space of a given retrieval model. However, the evidence is limited and confined to only a few models, collections and measures. In this paper, we perform a comprehensive empirical evaluation analysing the relationship between retrieval bias and retrieval performance using several well known retrieval models, five large TREC test collections and ten performance measures (including the recently proposed PRES, Time Biased Gain (TBG) and U-Measure). For traditional relevance based measures (MAP, P@10, MRR, Recall, etc) the correlation between retrieval bias and performance is moderate. However, for TBG and U-Measure, we find that there is strong and significant negative correlations between retrieval bias and performance (i.e as bias drops, performance increases). These findings suggest that for these more sophisticated, user oriented measures the retrievability bias hypothesis tends to hold. The implication is that for these measures, systems can then be tuned using retrieval bias, without recourse to relevance judgements.

AB - Retrievability provides an alternative way to assess an Information Retrieval (IR) system by measuring how easily documents can be retrieved. Retrievability can also be used to determine the level of retrieval bias a system exerts upon a collection of documents. It has been hypothesised that reducing the retrieval bias will lead to improved performance. To date, it has been shown that this hypothesis does not appear to hold on standard retrieval performance measures (MAP and P@10) when exploring the parameter space of a given retrieval model. However, the evidence is limited and confined to only a few models, collections and measures. In this paper, we perform a comprehensive empirical evaluation analysing the relationship between retrieval bias and retrieval performance using several well known retrieval models, five large TREC test collections and ten performance measures (including the recently proposed PRES, Time Biased Gain (TBG) and U-Measure). For traditional relevance based measures (MAP, P@10, MRR, Recall, etc) the correlation between retrieval bias and performance is moderate. However, for TBG and U-Measure, we find that there is strong and significant negative correlations between retrieval bias and performance (i.e as bias drops, performance increases). These findings suggest that for these more sophisticated, user oriented measures the retrievability bias hypothesis tends to hold. The implication is that for these measures, systems can then be tuned using retrieval bias, without recourse to relevance judgements.

KW - effectiveness

KW - user measures

KW - retrievability

U2 - 10.1145/2661829.2661948

DO - 10.1145/2661829.2661948

M3 - Conference contribution book

SN - 978-1-4503-2598-1

SP - 81

EP - 90

BT - CIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management

CY - New York, NY, USA

ER -

Wilkie C, Azzopardi L. A retrievability analysis: exploring the relationship between retrieval bias and retrieval performance. In CIKM '14 Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management . New York, NY, USA. 2014. p. 81-90 https://doi.org/10.1145/2661829.2661948