Retrievability and retrieval bias: a comparison of inequality measures

Colin Wilkie, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

5 Citations (Scopus)

Abstract

The disposition of a retrieval system to favour certain documents over others can be quantified using retrievability. Typically, the Gini Coefficient has been used to quantify the level of bias a system imposes across the collection with a single value. However, numerous inequality measures have been proposed that may provide different insights into retrievability bias. In this paper, we examine 8 inequality measures, and see the changes in the estimation of bias on 3 standard retrieval models across their respective parameter spaces. We find that most of the measures agree with each other, and that the parameter setting that minimise the inequality according to each measure is similar. This work suggests that the standard inequality measure, the Gini Coefficient, provides similar information regarding the bias. However, we find that Palma index and 20:20 Ratio show the greatest differences and may be useful to provide a different perspective when ranking systems according to bias.

LanguageEnglish
Title of host publicationAdvances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings
Place of PublicationSwitzerland
PublisherSpringer-Verlag
Pages209-214
Number of pages6
Volume9022
ISBN (Electronic)9783319163536
DOIs
Publication statusPublished - 2015
Event37th European Conference on Information Retrieval Research, ECIR 2015 - Vienna, Austria
Duration: 29 Mar 20152 Apr 2015

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9022
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference37th European Conference on Information Retrieval Research, ECIR 2015
CountryAustria
CityVienna
Period29/03/152/04/15

Fingerprint

Retrieval
Gini Coefficient
Parameter Space
Ranking
Quantify
Minimise
Standards
Model

Cite this

Wilkie, C., & Azzopardi, L. (2015). Retrievability and retrieval bias: a comparison of inequality measures. In Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings (Vol. 9022, pp. 209-214). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9022). Switzerland: Springer-Verlag. https://doi.org/10.1007/978-3-319-16354-3_22
Wilkie, Colin ; Azzopardi, Leif. / Retrievability and retrieval bias : a comparison of inequality measures. Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings. Vol. 9022 Switzerland : Springer-Verlag, 2015. pp. 209-214 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{862278312e7c4c20bbaf27aa06be7504,
title = "Retrievability and retrieval bias: a comparison of inequality measures",
abstract = "The disposition of a retrieval system to favour certain documents over others can be quantified using retrievability. Typically, the Gini Coefficient has been used to quantify the level of bias a system imposes across the collection with a single value. However, numerous inequality measures have been proposed that may provide different insights into retrievability bias. In this paper, we examine 8 inequality measures, and see the changes in the estimation of bias on 3 standard retrieval models across their respective parameter spaces. We find that most of the measures agree with each other, and that the parameter setting that minimise the inequality according to each measure is similar. This work suggests that the standard inequality measure, the Gini Coefficient, provides similar information regarding the bias. However, we find that Palma index and 20:20 Ratio show the greatest differences and may be useful to provide a different perspective when ranking systems according to bias.",
author = "Colin Wilkie and Leif Azzopardi",
year = "2015",
doi = "10.1007/978-3-319-16354-3_22",
language = "English",
volume = "9022",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer-Verlag",
pages = "209--214",
booktitle = "Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings",

}

Wilkie, C & Azzopardi, L 2015, Retrievability and retrieval bias: a comparison of inequality measures. in Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings. vol. 9022, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9022, Springer-Verlag, Switzerland, pp. 209-214, 37th European Conference on Information Retrieval Research, ECIR 2015, Vienna, Austria, 29/03/15. https://doi.org/10.1007/978-3-319-16354-3_22

Retrievability and retrieval bias : a comparison of inequality measures. / Wilkie, Colin; Azzopardi, Leif.

Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings. Vol. 9022 Switzerland : Springer-Verlag, 2015. p. 209-214 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9022).

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Retrievability and retrieval bias

T2 - a comparison of inequality measures

AU - Wilkie, Colin

AU - Azzopardi, Leif

PY - 2015

Y1 - 2015

N2 - The disposition of a retrieval system to favour certain documents over others can be quantified using retrievability. Typically, the Gini Coefficient has been used to quantify the level of bias a system imposes across the collection with a single value. However, numerous inequality measures have been proposed that may provide different insights into retrievability bias. In this paper, we examine 8 inequality measures, and see the changes in the estimation of bias on 3 standard retrieval models across their respective parameter spaces. We find that most of the measures agree with each other, and that the parameter setting that minimise the inequality according to each measure is similar. This work suggests that the standard inequality measure, the Gini Coefficient, provides similar information regarding the bias. However, we find that Palma index and 20:20 Ratio show the greatest differences and may be useful to provide a different perspective when ranking systems according to bias.

AB - The disposition of a retrieval system to favour certain documents over others can be quantified using retrievability. Typically, the Gini Coefficient has been used to quantify the level of bias a system imposes across the collection with a single value. However, numerous inequality measures have been proposed that may provide different insights into retrievability bias. In this paper, we examine 8 inequality measures, and see the changes in the estimation of bias on 3 standard retrieval models across their respective parameter spaces. We find that most of the measures agree with each other, and that the parameter setting that minimise the inequality according to each measure is similar. This work suggests that the standard inequality measure, the Gini Coefficient, provides similar information regarding the bias. However, we find that Palma index and 20:20 Ratio show the greatest differences and may be useful to provide a different perspective when ranking systems according to bias.

UR - http://www.scopus.com/inward/record.url?scp=84925433064&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-16354-3_22

DO - 10.1007/978-3-319-16354-3_22

M3 - Conference contribution book

VL - 9022

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 209

EP - 214

BT - Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings

PB - Springer-Verlag

CY - Switzerland

ER -

Wilkie C, Azzopardi L. Retrievability and retrieval bias: a comparison of inequality measures. In Advances in Information Retrieval - 37th European Conference on IR Research, ECIR 2015, Proceedings. Vol. 9022. Switzerland: Springer-Verlag. 2015. p. 209-214. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-16354-3_22