A Comparison of Primary and Secondary Relevance Judgements for Real-Life Topics

  • Simon Wakeling (Creator)
  • Martin Halvey (Creator)
  • Robert Villa (University of Sheffield) (Creator)
  • Laura Hasler (University of Sheffield) (Creator)



This dataset is from a user study that examines in detail the differences between primary and secondary assessors on a set of “real-world” topics which were gathered specifically for this study. By gathering topics which are representative of the staff and students at a major university, at a particular point in time, we aimed to explore differences between primary and secondary relevance judgements for real-life search tasks. Findings from our study suggest that while secondary assessors may find the assessment task challenging in various ways (they generally possess less interest and knowledge in secondary topics and take longer to assess documents), agreement between primary and secondary assessors is high.
Date made available6 Mar 2016
PublisherUniversity of Strathclyde
Date of data production1 Sep 2015

Cite this