A test collection for evaluating retrieval of studies for inclusion in systematic reviews

Harrisen Scells, Guido Zuccon, Bevan Koopman, Anthony Deacon, Leif Azzopardi, Shlomo Geva

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

12 Citations (Scopus)
34 Downloads (Pure)

Abstract

This paper introduces a test collection for evaluating the effectiveness of different methods used to retrieve research studies for inclusion in systematic reviews. Systematic reviews appraise and synthesise studies that meet specific inclusion criteria. Systematic reviews intended for a biomedical science audience use boolean queries with many, often complex, search clauses to retrieve studies; these are then manually screened to determine eligibility for inclusion in the review. This process is expensive and time consuming. The development of systems that improve retrieval effectiveness will have an immediate impact by reducing the complexity and resources required for this process. Our test collection consists of approximately 26 million research studies extracted from the freely available MEDLINE database, 94 review (query) topics extracted from Cochrane systematic reviews, and corresponding relevance assessments. Tasks for which the collection can be used for information retrieval system evaluation are described and the use of the collection to evaluate common baselines within one such task is demonstrated. The test collection is available at https://github.com/ielab/SIGIR2017-PICO-Collection.

Original languageEnglish
Title of host publicationSIGIR '17 Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval
Place of PublicationNew York, NY
Pages1237-1240
Number of pages4
ISBN (Electronic)9781450350228
DOIs
Publication statusPublished - 7 Aug 2017
Event40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017 - Tokyo, Shinjuku, Japan
Duration: 7 Aug 201711 Aug 2017

Conference

Conference40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2017
CountryJapan
CityTokyo, Shinjuku
Period7/08/1711/08/17

    Fingerprint

Keywords

  • evaluation
  • experimentation
  • systematic reviews
  • test collections

Cite this

Scells, H., Zuccon, G., Koopman, B., Deacon, A., Azzopardi, L., & Geva, S. (2017). A test collection for evaluating retrieval of studies for inclusion in systematic reviews. In SIGIR '17 Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 1237-1240). New York, NY. https://doi.org/10.1145/3077136.3080707