The combination and evaluation of query performance prediction methods

Claudia Hauff, Leif Azzopardi, Djoerd Hiemstra

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

3 Citations (Scopus)

Abstract

In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query’s performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method’s quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.
Original languageEnglish
Title of host publicationAdvances in Information Retrieval
Subtitle of host publication31th European Conference on IR Research, ECIR 2009, Toulouse, France, April 6-9, 2009. Proceedings
Place of PublicationBerlin, Heidelberg
PublisherSpringer-Verlag
Pages301-312
Number of pages12
ISBN (Print)978-3-642-00957-0
DOIs
Publication statusPublished - 27 Mar 2009
Externally publishedYes

Publication series

NameLecture Notes in Computer Science
PublisherSpringer Verlag
Volume5478

Keywords

  • data mining
  • knowledge discovery
  • prediction methods

Cite this