Query variation performance prediction for systematic reviews

Harrisen Scells, Leif Azzopardi, Guido Zuccon, Bevan Koopman

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

5 Citations (Scopus)

Abstract

When conducting systematic reviews, medical researchers heavily deliberate over the final query to pose to the information retrieval system. Given the possible query variations that they could construct, selecting the best performing query is difficult. This motivates a new type of query performance prediction (QPP) task where the challenge is to estimate the performance of a set of query variations given a particular topic. Query variations are the reductions, expansions and modifications of a given seed query under the hypothesis that there exists some variations (either generated from permutations or hand crafted) which will improve retrieval effectiveness over the original query. We use the CLEF 2017 TAR Collection, to evaluate sixteen pre and post retrieval predictors for the task of Query Variation Performance Prediction (QVPP). Our findings show the IDF based QPPs exhibits the strongest correlations with performance. However, when using QPPs to select the best query, little improvement over the original query can be obtained, despite the fact that there are query variations which perform significantly better. Our findings highlight the difficulty in identifying effective queries within the context of this new task, and motivates further research to develop more accurate methods to help systematic review researchers in the query selection process.

LanguageEnglish
Title of host publication41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018
Place of PublicationNew York, NY.
Pages1089-1092
Number of pages4
ISBN (Electronic)9781450356572
DOIs
Publication statusPublished - 27 Jun 2018
Event41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018 - Ann Arbor, United States
Duration: 8 Jul 201812 Jul 2018

Conference

Conference41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018
CountryUnited States
CityAnn Arbor
Period8/07/1812/07/18

Fingerprint

Information retrieval systems
Seed

Keywords

  • Information retrieval
  • systematic reviews
  • query formulation
  • information retrieval
  • HCI

Cite this

Scells, H., Azzopardi, L., Zuccon, G., & Koopman, B. (2018). Query variation performance prediction for systematic reviews. In 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018 (pp. 1089-1092). New York, NY.. https://doi.org/10.1145/3209978.3210078
Scells, Harrisen ; Azzopardi, Leif ; Zuccon, Guido ; Koopman, Bevan. / Query variation performance prediction for systematic reviews. 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018. New York, NY., 2018. pp. 1089-1092
@inproceedings{88b2c7cd06934080b3016683d9f54551,
title = "Query variation performance prediction for systematic reviews",
abstract = "When conducting systematic reviews, medical researchers heavily deliberate over the final query to pose to the information retrieval system. Given the possible query variations that they could construct, selecting the best performing query is difficult. This motivates a new type of query performance prediction (QPP) task where the challenge is to estimate the performance of a set of query variations given a particular topic. Query variations are the reductions, expansions and modifications of a given seed query under the hypothesis that there exists some variations (either generated from permutations or hand crafted) which will improve retrieval effectiveness over the original query. We use the CLEF 2017 TAR Collection, to evaluate sixteen pre and post retrieval predictors for the task of Query Variation Performance Prediction (QVPP). Our findings show the IDF based QPPs exhibits the strongest correlations with performance. However, when using QPPs to select the best query, little improvement over the original query can be obtained, despite the fact that there are query variations which perform significantly better. Our findings highlight the difficulty in identifying effective queries within the context of this new task, and motivates further research to develop more accurate methods to help systematic review researchers in the query selection process.",
keywords = "Information retrieval, systematic reviews, query formulation, information retrieval, HCI",
author = "Harrisen Scells and Leif Azzopardi and Guido Zuccon and Bevan Koopman",
year = "2018",
month = "6",
day = "27",
doi = "10.1145/3209978.3210078",
language = "English",
pages = "1089--1092",
booktitle = "41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018",

}

Scells, H, Azzopardi, L, Zuccon, G & Koopman, B 2018, Query variation performance prediction for systematic reviews. in 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018. New York, NY., pp. 1089-1092, 41st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, Ann Arbor, United States, 8/07/18. https://doi.org/10.1145/3209978.3210078

Query variation performance prediction for systematic reviews. / Scells, Harrisen; Azzopardi, Leif; Zuccon, Guido; Koopman, Bevan.

41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018. New York, NY., 2018. p. 1089-1092.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Query variation performance prediction for systematic reviews

AU - Scells, Harrisen

AU - Azzopardi, Leif

AU - Zuccon, Guido

AU - Koopman, Bevan

PY - 2018/6/27

Y1 - 2018/6/27

N2 - When conducting systematic reviews, medical researchers heavily deliberate over the final query to pose to the information retrieval system. Given the possible query variations that they could construct, selecting the best performing query is difficult. This motivates a new type of query performance prediction (QPP) task where the challenge is to estimate the performance of a set of query variations given a particular topic. Query variations are the reductions, expansions and modifications of a given seed query under the hypothesis that there exists some variations (either generated from permutations or hand crafted) which will improve retrieval effectiveness over the original query. We use the CLEF 2017 TAR Collection, to evaluate sixteen pre and post retrieval predictors for the task of Query Variation Performance Prediction (QVPP). Our findings show the IDF based QPPs exhibits the strongest correlations with performance. However, when using QPPs to select the best query, little improvement over the original query can be obtained, despite the fact that there are query variations which perform significantly better. Our findings highlight the difficulty in identifying effective queries within the context of this new task, and motivates further research to develop more accurate methods to help systematic review researchers in the query selection process.

AB - When conducting systematic reviews, medical researchers heavily deliberate over the final query to pose to the information retrieval system. Given the possible query variations that they could construct, selecting the best performing query is difficult. This motivates a new type of query performance prediction (QPP) task where the challenge is to estimate the performance of a set of query variations given a particular topic. Query variations are the reductions, expansions and modifications of a given seed query under the hypothesis that there exists some variations (either generated from permutations or hand crafted) which will improve retrieval effectiveness over the original query. We use the CLEF 2017 TAR Collection, to evaluate sixteen pre and post retrieval predictors for the task of Query Variation Performance Prediction (QVPP). Our findings show the IDF based QPPs exhibits the strongest correlations with performance. However, when using QPPs to select the best query, little improvement over the original query can be obtained, despite the fact that there are query variations which perform significantly better. Our findings highlight the difficulty in identifying effective queries within the context of this new task, and motivates further research to develop more accurate methods to help systematic review researchers in the query selection process.

KW - Information retrieval

KW - systematic reviews

KW - query formulation

KW - information retrieval

KW - HCI

U2 - 10.1145/3209978.3210078

DO - 10.1145/3209978.3210078

M3 - Conference contribution book

SP - 1089

EP - 1092

BT - 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018

CY - New York, NY.

ER -

Scells H, Azzopardi L, Zuccon G, Koopman B. Query variation performance prediction for systematic reviews. In 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018. New York, NY. 2018. p. 1089-1092 https://doi.org/10.1145/3209978.3210078