CLEF 2019 technology assisted reviews in empirical medicine overview

Evangelos Kanoulas, Dan Li, Leif Azzopardi, Rene Spijker

Research output: Contribution to journalConference articlepeer-review

26 Citations (Scopus)
33 Downloads (Pure)


Systematic reviews are a widely used method to provide an overview over the current scientific consensus, by bringing together multiple studies in a systematic, reliable, and transparent way. The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying all relevant studies in an unbiased way both complex and time consuming to the extent that jeopardizes the validity of their findings and the ability to inform policy and practice in a timely manner. The CLEF 2019 e-Health TAR Lab accommodated two tasks. Task 1 focused on retrieving relevant studies from PubMed without the use of a Boolean query, while Task 2 focused on the efficient and effective ranking of studies during the abstract and title screening phase of conducting a systematic review. In the 2019 lab we also expanded upon the type of systematics reviews considered. Hence, beyond Diagnostic Test Accuracy reviews, we also included Intervention, Prognosis, and Qualitative systematic reviews. We constructed a benchmark collection of 31 reviews published by Cochrane, and the corresponding relevant and irrelevant articles found by the original Boolean query. Three teams participated in Task 2, submitting automatic and semi-automatic runs, using information retrieval and machine learning algorithms over a variety of text representations, in a batch and iterative manner. This paper reports both the methodology used to construct the benchmark collection, and the results of the evaluation.

Original languageEnglish
Article number250
Number of pages31
JournalCEUR Workshop Proceedings
Publication statusPublished - 12 Sept 2019
Event20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF 2019 - Lugano, Switzerland
Duration: 9 Sept 201912 Sept 2019


  • active learning
  • evaluation
  • Information Retrieval
  • systematic reviews
  • TAR
  • text classification


Dive into the research topics of 'CLEF 2019 technology assisted reviews in empirical medicine overview'. Together they form a unique fingerprint.

Cite this