A game-theory approach for effective crowdsource-based relevance assessment

Yashar Moshfeghi, Alvaro Francisco Huertas Rosero, Joemon M. Jose

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Despite the ever-increasing popularity of crowdsourcing (CS) in both industry and academia, procedures that ensure quality in its results are still elusive. We hypothesise that a CS design based on game theory can persuade workers to perform their tasks as quickly as possible with the highest quality. In order to do so, in this article we propose a CS framework inspired by the n-person Chicken game. Our aim is to address the problem of CS quality without compromising on CS benefits such as low monetary cost and high task completion speed. With that goal in mind, we study the effects of knowledge updates as well as incentives for good workers to continue playing. We define a general task with the characteristics of relevance assessment as a case study, because it has been widely explored in the past with CS due to its potential cost and complexity. In order to investigate our hypotheses, we conduct a simulation where we study the effect of the proposed framework on data accuracy, task completion time, and total monetary rewards. Based on a game-theoretical analysis, we study how different types of individuals would behave under a particular game scenario. In particular, we simulate a population comprised of different types of workers with varying ability to formulate optimal strategies and learn from their experiences. A simulation of the proposed framework produced results that support our hypothesis.
LanguageEnglish
Article number55
Number of pages25
Journal ACM Transactions on Intelligent Systems and Technology
Volume7
Issue number4
DOIs
Publication statusPublished - 31 Jul 2016

Fingerprint

Game theory
Game Theory
Game
Costs
Completion Time
Optimal Strategy
Incentives
Reward
Completion
Theoretical Analysis
Person
Simulation
Continue
Update
Industry
Scenarios
Relevance
Framework

Keywords

  • game theory
  • crowdsourcing
  • relevance assessment

Cite this

@article{a6641071bfca44a68820a13e5b3eb252,
title = "A game-theory approach for effective crowdsource-based relevance assessment",
abstract = "Despite the ever-increasing popularity of crowdsourcing (CS) in both industry and academia, procedures that ensure quality in its results are still elusive. We hypothesise that a CS design based on game theory can persuade workers to perform their tasks as quickly as possible with the highest quality. In order to do so, in this article we propose a CS framework inspired by the n-person Chicken game. Our aim is to address the problem of CS quality without compromising on CS benefits such as low monetary cost and high task completion speed. With that goal in mind, we study the effects of knowledge updates as well as incentives for good workers to continue playing. We define a general task with the characteristics of relevance assessment as a case study, because it has been widely explored in the past with CS due to its potential cost and complexity. In order to investigate our hypotheses, we conduct a simulation where we study the effect of the proposed framework on data accuracy, task completion time, and total monetary rewards. Based on a game-theoretical analysis, we study how different types of individuals would behave under a particular game scenario. In particular, we simulate a population comprised of different types of workers with varying ability to formulate optimal strategies and learn from their experiences. A simulation of the proposed framework produced results that support our hypothesis.",
keywords = "game theory, crowdsourcing, relevance assessment",
author = "Yashar Moshfeghi and Rosero, {Alvaro Francisco Huertas} and Jose, {Joemon M.}",
year = "2016",
month = "7",
day = "31",
doi = "10.1145/2873063",
language = "English",
volume = "7",
journal = "ACM Transactions on Intelligent Systems and Technology",
issn = "2157-6904",
number = "4",

}

A game-theory approach for effective crowdsource-based relevance assessment. / Moshfeghi, Yashar; Rosero, Alvaro Francisco Huertas; Jose, Joemon M.

In: ACM Transactions on Intelligent Systems and Technology, Vol. 7, No. 4, 55, 31.07.2016.

Research output: Contribution to journalArticle

TY - JOUR

T1 - A game-theory approach for effective crowdsource-based relevance assessment

AU - Moshfeghi, Yashar

AU - Rosero, Alvaro Francisco Huertas

AU - Jose, Joemon M.

PY - 2016/7/31

Y1 - 2016/7/31

N2 - Despite the ever-increasing popularity of crowdsourcing (CS) in both industry and academia, procedures that ensure quality in its results are still elusive. We hypothesise that a CS design based on game theory can persuade workers to perform their tasks as quickly as possible with the highest quality. In order to do so, in this article we propose a CS framework inspired by the n-person Chicken game. Our aim is to address the problem of CS quality without compromising on CS benefits such as low monetary cost and high task completion speed. With that goal in mind, we study the effects of knowledge updates as well as incentives for good workers to continue playing. We define a general task with the characteristics of relevance assessment as a case study, because it has been widely explored in the past with CS due to its potential cost and complexity. In order to investigate our hypotheses, we conduct a simulation where we study the effect of the proposed framework on data accuracy, task completion time, and total monetary rewards. Based on a game-theoretical analysis, we study how different types of individuals would behave under a particular game scenario. In particular, we simulate a population comprised of different types of workers with varying ability to formulate optimal strategies and learn from their experiences. A simulation of the proposed framework produced results that support our hypothesis.

AB - Despite the ever-increasing popularity of crowdsourcing (CS) in both industry and academia, procedures that ensure quality in its results are still elusive. We hypothesise that a CS design based on game theory can persuade workers to perform their tasks as quickly as possible with the highest quality. In order to do so, in this article we propose a CS framework inspired by the n-person Chicken game. Our aim is to address the problem of CS quality without compromising on CS benefits such as low monetary cost and high task completion speed. With that goal in mind, we study the effects of knowledge updates as well as incentives for good workers to continue playing. We define a general task with the characteristics of relevance assessment as a case study, because it has been widely explored in the past with CS due to its potential cost and complexity. In order to investigate our hypotheses, we conduct a simulation where we study the effect of the proposed framework on data accuracy, task completion time, and total monetary rewards. Based on a game-theoretical analysis, we study how different types of individuals would behave under a particular game scenario. In particular, we simulate a population comprised of different types of workers with varying ability to formulate optimal strategies and learn from their experiences. A simulation of the proposed framework produced results that support our hypothesis.

KW - game theory

KW - crowdsourcing

KW - relevance assessment

U2 - 10.1145/2873063

DO - 10.1145/2873063

M3 - Article

VL - 7

JO - ACM Transactions on Intelligent Systems and Technology

T2 - ACM Transactions on Intelligent Systems and Technology

JF - ACM Transactions on Intelligent Systems and Technology

SN - 2157-6904

IS - 4

M1 - 55

ER -