Relying on topic subsets for system ranking estimation

Claudia Hauff, Djoerd Hiemstra, Franciska de Jong, Leif Azzopardi

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

8 Citations (Scopus)

Abstract

Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.
LanguageEnglish
Title of host publicationCIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management
Place of PublicationNew York, NY, USA
Pages1859-1862
Number of pages4
DOIs
Publication statusPublished - 2 Nov 2009
Externally publishedYes

Fingerprint

ranking
performance

Keywords

  • evaluation
  • system ranking estimation

Cite this

Hauff, C., Hiemstra, D., de Jong, F., & Azzopardi, L. (2009). Relying on topic subsets for system ranking estimation. In CIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management (pp. 1859-1862). New York, NY, USA. https://doi.org/10.1145/1645953.1646249
Hauff, Claudia ; Hiemstra, Djoerd ; de Jong, Franciska ; Azzopardi, Leif. / Relying on topic subsets for system ranking estimation. CIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management. New York, NY, USA, 2009. pp. 1859-1862
@inproceedings{06ccf56d396b44b5a0f786d114a39a4e,
title = "Relying on topic subsets for system ranking estimation",
abstract = "Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the {"}right{"} subset of topics and show that using topic subsets improves the performance by 32{\%} on average, with a maximum improvement of up to 70{\%} in some cases.",
keywords = "evaluation, system ranking estimation",
author = "Claudia Hauff and Djoerd Hiemstra and {de Jong}, Franciska and Leif Azzopardi",
year = "2009",
month = "11",
day = "2",
doi = "10.1145/1645953.1646249",
language = "English",
isbn = "978-1-60558-512-3",
pages = "1859--1862",
booktitle = "CIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management",

}

Hauff, C, Hiemstra, D, de Jong, F & Azzopardi, L 2009, Relying on topic subsets for system ranking estimation. in CIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management. New York, NY, USA, pp. 1859-1862. https://doi.org/10.1145/1645953.1646249

Relying on topic subsets for system ranking estimation. / Hauff, Claudia; Hiemstra, Djoerd; de Jong, Franciska; Azzopardi, Leif.

CIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management. New York, NY, USA, 2009. p. 1859-1862.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Relying on topic subsets for system ranking estimation

AU - Hauff, Claudia

AU - Hiemstra, Djoerd

AU - de Jong, Franciska

AU - Azzopardi, Leif

PY - 2009/11/2

Y1 - 2009/11/2

N2 - Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.

AB - Ranking a number of retrieval systems according to their retrieval effectiveness without relying on costly relevance judgments was first explored by Soboroff et al [6]. Over the years, a number of alternative approaches have been proposed. We perform a comprehensive analysis of system ranking estimation approaches on a wide variety of TREC test collections and topics sets. Our analysis reveals that the performance of such approaches is highly dependent upon the topic or topic subset, used for estimation. We hypothesize that the performance of system ranking estimation approaches can be improved by selecting the "right" subset of topics and show that using topic subsets improves the performance by 32% on average, with a maximum improvement of up to 70% in some cases.

KW - evaluation

KW - system ranking estimation

U2 - 10.1145/1645953.1646249

DO - 10.1145/1645953.1646249

M3 - Conference contribution book

SN - 978-1-60558-512-3

SP - 1859

EP - 1862

BT - CIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management

CY - New York, NY, USA

ER -

Hauff C, Hiemstra D, de Jong F, Azzopardi L. Relying on topic subsets for system ranking estimation. In CIKM '09 Proceedings of the 18th ACM Conference on Information and Knowledge Management. New York, NY, USA. 2009. p. 1859-1862 https://doi.org/10.1145/1645953.1646249