Retrievability

an evaluation measure for higher order information access tasks

Leif Azzopardi, Vishwa Vinay

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

56 Citations (Scopus)

Abstract

Evaluation in Information Retrieval (IR) has long focused on effectiveness and efficiency. However, new and emerging access tasks now demand alternative evaluation measures which go beyond this traditional view. A retrieval system provides a means of gaining access to documents, therefore intuitively, our view of the collection is shaped by the retrieval system. In this paper, we outline some emerging information access related scenarios that require knowledge about how the retrieval system affects the users' ability to access information. This provides the motivation for the proposed evaluation measures and methodology where the focus is on capturing the behavior of the system, in terms of how retrievable it makes individual documents within the collection. To demonstrate the utility of the proposed methods, we perform an extensive analysis on two TREC collections showing how the measures can be applied to evaluate different information access questions. For higher order information access tasks that are inherently dependent on retrievability, our novel evaluation methodology emphasizes that effectiveness is an insufficient characterization of a retrieval system. This paper provides the foundations for the evaluation of higher order access related tasks.
Original languageEnglish
Title of host publicationCIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management
Place of PublicationNew York, NY, USA
Pages561-570
Number of pages10
DOIs
Publication statusPublished - 26 Oct 2008
Externally publishedYes

Fingerprint

evaluation
methodology
information retrieval
scenario
efficiency
demand
ability

Keywords

  • accessibililty
  • retrievability

Cite this

Azzopardi, L., & Vinay, V. (2008). Retrievability: an evaluation measure for higher order information access tasks. In CIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management (pp. 561-570). New York, NY, USA. https://doi.org/10.1145/1458082.1458157
Azzopardi, Leif ; Vinay, Vishwa. / Retrievability : an evaluation measure for higher order information access tasks. CIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management. New York, NY, USA, 2008. pp. 561-570
@inproceedings{6f94f3dc54bb4e9d83c81a39417c5cd0,
title = "Retrievability: an evaluation measure for higher order information access tasks",
abstract = "Evaluation in Information Retrieval (IR) has long focused on effectiveness and efficiency. However, new and emerging access tasks now demand alternative evaluation measures which go beyond this traditional view. A retrieval system provides a means of gaining access to documents, therefore intuitively, our view of the collection is shaped by the retrieval system. In this paper, we outline some emerging information access related scenarios that require knowledge about how the retrieval system affects the users' ability to access information. This provides the motivation for the proposed evaluation measures and methodology where the focus is on capturing the behavior of the system, in terms of how retrievable it makes individual documents within the collection. To demonstrate the utility of the proposed methods, we perform an extensive analysis on two TREC collections showing how the measures can be applied to evaluate different information access questions. For higher order information access tasks that are inherently dependent on retrievability, our novel evaluation methodology emphasizes that effectiveness is an insufficient characterization of a retrieval system. This paper provides the foundations for the evaluation of higher order access related tasks.",
keywords = "accessibililty, retrievability",
author = "Leif Azzopardi and Vishwa Vinay",
year = "2008",
month = "10",
day = "26",
doi = "10.1145/1458082.1458157",
language = "English",
isbn = "978-1-59593-991-3",
pages = "561--570",
booktitle = "CIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management",

}

Azzopardi, L & Vinay, V 2008, Retrievability: an evaluation measure for higher order information access tasks. in CIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management. New York, NY, USA, pp. 561-570. https://doi.org/10.1145/1458082.1458157

Retrievability : an evaluation measure for higher order information access tasks. / Azzopardi, Leif; Vinay, Vishwa.

CIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management. New York, NY, USA, 2008. p. 561-570.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Retrievability

T2 - an evaluation measure for higher order information access tasks

AU - Azzopardi, Leif

AU - Vinay, Vishwa

PY - 2008/10/26

Y1 - 2008/10/26

N2 - Evaluation in Information Retrieval (IR) has long focused on effectiveness and efficiency. However, new and emerging access tasks now demand alternative evaluation measures which go beyond this traditional view. A retrieval system provides a means of gaining access to documents, therefore intuitively, our view of the collection is shaped by the retrieval system. In this paper, we outline some emerging information access related scenarios that require knowledge about how the retrieval system affects the users' ability to access information. This provides the motivation for the proposed evaluation measures and methodology where the focus is on capturing the behavior of the system, in terms of how retrievable it makes individual documents within the collection. To demonstrate the utility of the proposed methods, we perform an extensive analysis on two TREC collections showing how the measures can be applied to evaluate different information access questions. For higher order information access tasks that are inherently dependent on retrievability, our novel evaluation methodology emphasizes that effectiveness is an insufficient characterization of a retrieval system. This paper provides the foundations for the evaluation of higher order access related tasks.

AB - Evaluation in Information Retrieval (IR) has long focused on effectiveness and efficiency. However, new and emerging access tasks now demand alternative evaluation measures which go beyond this traditional view. A retrieval system provides a means of gaining access to documents, therefore intuitively, our view of the collection is shaped by the retrieval system. In this paper, we outline some emerging information access related scenarios that require knowledge about how the retrieval system affects the users' ability to access information. This provides the motivation for the proposed evaluation measures and methodology where the focus is on capturing the behavior of the system, in terms of how retrievable it makes individual documents within the collection. To demonstrate the utility of the proposed methods, we perform an extensive analysis on two TREC collections showing how the measures can be applied to evaluate different information access questions. For higher order information access tasks that are inherently dependent on retrievability, our novel evaluation methodology emphasizes that effectiveness is an insufficient characterization of a retrieval system. This paper provides the foundations for the evaluation of higher order access related tasks.

KW - accessibililty

KW - retrievability

U2 - 10.1145/1458082.1458157

DO - 10.1145/1458082.1458157

M3 - Conference contribution book

SN - 978-1-59593-991-3

SP - 561

EP - 570

BT - CIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management

CY - New York, NY, USA

ER -

Azzopardi L, Vinay V. Retrievability: an evaluation measure for higher order information access tasks. In CIKM '08 Proceedings of the 17th ACM Conference on Information and Knowledge Management. New York, NY, USA. 2008. p. 561-570 https://doi.org/10.1145/1458082.1458157