Identifying careless workers in crowdsourcing platforms

a game theory approach

Yashar Moshfeghi, Alvaro F. Huertas-Rosero, Joemon M. Jose

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

5 Citations (Scopus)

Abstract

In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.
Original languageEnglish
Title of host publicationProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval
Place of PublicationNew York, NY, USA
Pages857-860
Number of pages4
DOIs
Publication statusPublished - 2016

Publication series

NameSIGIR '16
PublisherACM

Fingerprint

Game theory
Quality assurance

Keywords

  • chicken game
  • crowdsourcing
  • game theory
  • relevant assessments

Cite this

Moshfeghi, Y., Huertas-Rosero, A. F., & Jose, J. M. (2016). Identifying careless workers in crowdsourcing platforms: a game theory approach. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 857-860). (SIGIR '16). New York, NY, USA. https://doi.org/10.1145/2911451.2914756
Moshfeghi, Yashar ; Huertas-Rosero, Alvaro F. ; Jose, Joemon M. / Identifying careless workers in crowdsourcing platforms : a game theory approach. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY, USA, 2016. pp. 857-860 (SIGIR '16).
@inproceedings{cb38d729555645eeba1aab9b0e66db0f,
title = "Identifying careless workers in crowdsourcing platforms: a game theory approach",
abstract = "In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.",
keywords = "chicken game, crowdsourcing, game theory, relevant assessments",
author = "Yashar Moshfeghi and Huertas-Rosero, {Alvaro F.} and Jose, {Joemon M.}",
year = "2016",
doi = "10.1145/2911451.2914756",
language = "English",
isbn = "978-1-4503-4069-4",
series = "SIGIR '16",
publisher = "ACM",
pages = "857--860",
booktitle = "Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval",

}

Moshfeghi, Y, Huertas-Rosero, AF & Jose, JM 2016, Identifying careless workers in crowdsourcing platforms: a game theory approach. in Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR '16, New York, NY, USA, pp. 857-860. https://doi.org/10.1145/2911451.2914756

Identifying careless workers in crowdsourcing platforms : a game theory approach. / Moshfeghi, Yashar; Huertas-Rosero, Alvaro F.; Jose, Joemon M.

Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY, USA, 2016. p. 857-860 (SIGIR '16).

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Identifying careless workers in crowdsourcing platforms

T2 - a game theory approach

AU - Moshfeghi, Yashar

AU - Huertas-Rosero, Alvaro F.

AU - Jose, Joemon M.

PY - 2016

Y1 - 2016

N2 - In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.

AB - In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.

KW - chicken game

KW - crowdsourcing

KW - game theory

KW - relevant assessments

U2 - 10.1145/2911451.2914756

DO - 10.1145/2911451.2914756

M3 - Conference contribution book

SN - 978-1-4503-4069-4

T3 - SIGIR '16

SP - 857

EP - 860

BT - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

CY - New York, NY, USA

ER -

Moshfeghi Y, Huertas-Rosero AF, Jose JM. Identifying careless workers in crowdsourcing platforms: a game theory approach. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY, USA. 2016. p. 857-860. (SIGIR '16). https://doi.org/10.1145/2911451.2914756