Algorithmic bias: do good systems make relevant documents more retrievable?

Colin Wilkie, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Algorithmic bias presents a dificult challenge within Information Retrieval. Long has it been known that certain algorithms favour particular documents due to attributes of these documents that are not directly related to relevance. The evaluation of bias has recently been made possible through the use of retrievability, a quanti .able measure of bias. While evaluating bias is relatively novel, the evaluation of performance has been common since the dawn of the Cran.eld approach and TREC. To evaluate performance, a pool of documents to be judged by human assessors is created from the collection.is pooling approach has faced accusations of bias due to the fact that the state of the art algorithms were used to create it, thus the inclusion of biases associated with these algorithms may be included in the pool.e introduction of retrievability has provided a mechanism to evaluate the bias of these pools. This work evaluates the varying degrees of bias present in the groups of relevant and non-relevant documents for topics. The differentiating power of a system is also evaluated by examining the documents from the pool that are retrieved for each topic. The analysis .nds that the systems that perform better, tend to have a higher chance of retrieving a relevant document rather than a non-relevant document for a topic prior to retrieval, indicating that retrieval systems which perform better at TREC are already predisposed to agree with the judgements regardless of the query posed.

LanguageEnglish
Title of host publicationCIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management
Place of PublicationNew York
Pages2375-2378
Number of pages4
DOIs
StatePublished - 6 Nov 2017
Event26th ACM International Conference on Information and Knowledge Management, CIKM 2017 - Singapore, Singapore
Duration: 6 Nov 201710 Nov 2017

Conference

Conference26th ACM International Conference on Information and Knowledge Management, CIKM 2017
CountrySingapore
CitySingapore
Period6/11/1710/11/17

Fingerprint

Information retrieval

Keywords

  • algorithmic bias
  • information retieval
  • retrieval systems
  • relevance assessment

Cite this

Wilkie, C., & Azzopardi, L. (2017). Algorithmic bias: do good systems make relevant documents more retrievable? In CIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management (pp. 2375-2378). New York. DOI: 10.1145/3132847.3133135
Wilkie, Colin ; Azzopardi, Leif. / Algorithmic bias : do good systems make relevant documents more retrievable?. CIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York, 2017. pp. 2375-2378
@inproceedings{ec32921110564e1ea77b0be750a37549,
title = "Algorithmic bias: do good systems make relevant documents more retrievable?",
abstract = "Algorithmic bias presents a dificult challenge within Information Retrieval. Long has it been known that certain algorithms favour particular documents due to attributes of these documents that are not directly related to relevance. The evaluation of bias has recently been made possible through the use of retrievability, a quanti .able measure of bias. While evaluating bias is relatively novel, the evaluation of performance has been common since the dawn of the Cran.eld approach and TREC. To evaluate performance, a pool of documents to be judged by human assessors is created from the collection.is pooling approach has faced accusations of bias due to the fact that the state of the art algorithms were used to create it, thus the inclusion of biases associated with these algorithms may be included in the pool.e introduction of retrievability has provided a mechanism to evaluate the bias of these pools. This work evaluates the varying degrees of bias present in the groups of relevant and non-relevant documents for topics. The differentiating power of a system is also evaluated by examining the documents from the pool that are retrieved for each topic. The analysis .nds that the systems that perform better, tend to have a higher chance of retrieving a relevant document rather than a non-relevant document for a topic prior to retrieval, indicating that retrieval systems which perform better at TREC are already predisposed to agree with the judgements regardless of the query posed.",
keywords = "algorithmic bias, information retieval, retrieval systems, relevance assessment",
author = "Colin Wilkie and Leif Azzopardi",
year = "2017",
month = "11",
day = "6",
doi = "10.1145/3132847.3133135",
language = "English",
isbn = "9781450349185",
pages = "2375--2378",
booktitle = "CIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management",

}

Wilkie, C & Azzopardi, L 2017, Algorithmic bias: do good systems make relevant documents more retrievable? in CIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York, pp. 2375-2378, 26th ACM International Conference on Information and Knowledge Management, CIKM 2017, Singapore, Singapore, 6/11/17. DOI: 10.1145/3132847.3133135

Algorithmic bias : do good systems make relevant documents more retrievable? / Wilkie, Colin; Azzopardi, Leif.

CIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York, 2017. p. 2375-2378.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Algorithmic bias

T2 - do good systems make relevant documents more retrievable?

AU - Wilkie,Colin

AU - Azzopardi,Leif

PY - 2017/11/6

Y1 - 2017/11/6

N2 - Algorithmic bias presents a dificult challenge within Information Retrieval. Long has it been known that certain algorithms favour particular documents due to attributes of these documents that are not directly related to relevance. The evaluation of bias has recently been made possible through the use of retrievability, a quanti .able measure of bias. While evaluating bias is relatively novel, the evaluation of performance has been common since the dawn of the Cran.eld approach and TREC. To evaluate performance, a pool of documents to be judged by human assessors is created from the collection.is pooling approach has faced accusations of bias due to the fact that the state of the art algorithms were used to create it, thus the inclusion of biases associated with these algorithms may be included in the pool.e introduction of retrievability has provided a mechanism to evaluate the bias of these pools. This work evaluates the varying degrees of bias present in the groups of relevant and non-relevant documents for topics. The differentiating power of a system is also evaluated by examining the documents from the pool that are retrieved for each topic. The analysis .nds that the systems that perform better, tend to have a higher chance of retrieving a relevant document rather than a non-relevant document for a topic prior to retrieval, indicating that retrieval systems which perform better at TREC are already predisposed to agree with the judgements regardless of the query posed.

AB - Algorithmic bias presents a dificult challenge within Information Retrieval. Long has it been known that certain algorithms favour particular documents due to attributes of these documents that are not directly related to relevance. The evaluation of bias has recently been made possible through the use of retrievability, a quanti .able measure of bias. While evaluating bias is relatively novel, the evaluation of performance has been common since the dawn of the Cran.eld approach and TREC. To evaluate performance, a pool of documents to be judged by human assessors is created from the collection.is pooling approach has faced accusations of bias due to the fact that the state of the art algorithms were used to create it, thus the inclusion of biases associated with these algorithms may be included in the pool.e introduction of retrievability has provided a mechanism to evaluate the bias of these pools. This work evaluates the varying degrees of bias present in the groups of relevant and non-relevant documents for topics. The differentiating power of a system is also evaluated by examining the documents from the pool that are retrieved for each topic. The analysis .nds that the systems that perform better, tend to have a higher chance of retrieving a relevant document rather than a non-relevant document for a topic prior to retrieval, indicating that retrieval systems which perform better at TREC are already predisposed to agree with the judgements regardless of the query posed.

KW - algorithmic bias

KW - information retieval

KW - retrieval systems

KW - relevance assessment

UR - http://www.scopus.com/inward/record.url?scp=85037378116&partnerID=8YFLogxK

UR - http://cikm2017.org/

U2 - 10.1145/3132847.3133135

DO - 10.1145/3132847.3133135

M3 - Conference contribution

SN - 9781450349185

SP - 2375

EP - 2378

BT - CIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management

CY - New York

ER -

Wilkie C, Azzopardi L. Algorithmic bias: do good systems make relevant documents more retrievable? In CIKM 2017 - Proceedings of the 2017 ACM Conference on Information and Knowledge Management. New York. 2017. p. 2375-2378. Available from, DOI: 10.1145/3132847.3133135