Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

Guido Zuccon, Teerapong Leelanupab, Stewart Whiting, Emine Yilmaz, Joemon M. Jose, Leif Azzopardi

Research output: Contribution to journalArticle

23 Citations (Scopus)

Abstract

In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.
LanguageEnglish
Pages267-305
Number of pages39
JournalInformation Retrieval
Volume16
Issue number2
DOIs
Publication statusPublished - 1 Apr 2013
Externally publishedYes

Fingerprint

Information retrieval systems
information retrieval
Information retrieval
interaction
evaluation
Experiments
experiment
methodology
Costs
paradigm
costs
performance
lack
demand

Keywords

  • crowdsourcing evaluation
  • interactive IR evaluation

Cite this

Zuccon, Guido ; Leelanupab, Teerapong ; Whiting, Stewart ; Yilmaz, Emine ; Jose, Joemon M. ; Azzopardi, Leif. / Crowdsourcing interactions : using crowdsourcing for evaluating interactive information retrieval systems. In: Information Retrieval. 2013 ; Vol. 16, No. 2. pp. 267-305.
@article{dfa7f48e93924730b3635eb1b3cba4fb,
title = "Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems",
abstract = "In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.",
keywords = "crowdsourcing evaluation, interactive IR evaluation",
author = "Guido Zuccon and Teerapong Leelanupab and Stewart Whiting and Emine Yilmaz and Jose, {Joemon M.} and Leif Azzopardi",
year = "2013",
month = "4",
day = "1",
doi = "10.1007/s10791-012-9206-z",
language = "English",
volume = "16",
pages = "267--305",
journal = "Information Retrieval Journal",
issn = "1386-4564",
publisher = "Springer Netherlands",
number = "2",

}

Crowdsourcing interactions : using crowdsourcing for evaluating interactive information retrieval systems. / Zuccon, Guido; Leelanupab, Teerapong; Whiting, Stewart; Yilmaz, Emine; Jose, Joemon M.; Azzopardi, Leif.

In: Information Retrieval, Vol. 16, No. 2, 01.04.2013, p. 267-305.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Crowdsourcing interactions

T2 - Information Retrieval Journal

AU - Zuccon, Guido

AU - Leelanupab, Teerapong

AU - Whiting, Stewart

AU - Yilmaz, Emine

AU - Jose, Joemon M.

AU - Azzopardi, Leif

PY - 2013/4/1

Y1 - 2013/4/1

N2 - In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.

AB - In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.

KW - crowdsourcing evaluation

KW - interactive IR evaluation

U2 - 10.1007/s10791-012-9206-z

DO - 10.1007/s10791-012-9206-z

M3 - Article

VL - 16

SP - 267

EP - 305

JO - Information Retrieval Journal

JF - Information Retrieval Journal

SN - 1386-4564

IS - 2

ER -