A comparison of primary and secondary relevance judgements for real-life topics

Simon Wakeling, Martin Halvey, Robert Villa, Laura Hasler

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

4 Citations (Scopus)
97 Downloads (Pure)

Abstract

The notion of relevance is fundamental to the field of Information Retrieval. Within the field a generally accepted conception of relevance as inherently subjective has emerged, with an individual’s assessment of relevance influenced by numerous contextual factors. In this paper we present a user study that examines in detail the differences between primary and secondary assessors on a set of “real-world” topics which were gathered specifically for the work. By gathering topics which are representative of the staff and students at a major university, at a particular point in time, we aim to explore differences between primary and secondary relevance judgements for real-life search tasks. Findings suggest that while secondary assessors may find the assessment task challenging in various ways (they generally possess less interest and knowledge in secondary topics and take longer to assess documents), agreement between primary and secondary assessors is high.
Original languageEnglish
Title of host publicationProceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR '16 )
Place of PublicationNew York
Pages173-182
Number of pages10
DOIs
Publication statusPublished - 16 Mar 2016
EventACM SIGIR Conference on Human Information Interaction and Retrieval - North Carolina, Chapel Hill, United States
Duration: 13 Mar 201617 Mar 2016

Conference

ConferenceACM SIGIR Conference on Human Information Interaction and Retrieval
CountryUnited States
CityChapel Hill
Period13/03/1617/03/16

Keywords

  • information retrieval
  • retrieval evaluation
  • relevance judgements
  • test collection

Fingerprint Dive into the research topics of 'A comparison of primary and secondary relevance judgements for real-life topics'. Together they form a unique fingerprint.

  • Cite this

    Wakeling, S., Halvey, M., Villa, R., & Hasler, L. (2016). A comparison of primary and secondary relevance judgements for real-life topics. In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (CHIIR '16 ) (pp. 173-182). New York. https://doi.org/10.1145/2854946.2854968