Identifying careless workers in crowdsourcing platforms: a game theory approach

Yashar Moshfeghi, Alvaro F. Huertas-Rosero, Joemon M. Jose

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

10 Citations (Scopus)


In this paper we introduce a game scenario for crowdsourcing (CS) using incentives as a bait for careless (gambler) workers, who respond to them in a characteristic way. We hypothesise that careless workers are risk-inclined and can be detected in the game scenario by their use of time, and test this hypothesis in two steps: first, we formulate and prove a theorem stating that a risk-inclined worker will react to competition with shorter Task Completion Time (TCT) than a risk-neutral or risk-averse worker. Second, we check if the game scenario introduces a link between TCT and performance, by performing a crowdsourced evaluation using 35 topics from the TREC-8 collection. Experimental evidence confirms our hypothesis, showing that TCT can be used as a powerful discrimination factor to detect careless workers. This is a valuable result in the quest for quality assurance in CS-based micro tasks such as relevance assessment.
Original languageEnglish
Title of host publicationProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval
Place of PublicationNew York, NY, USA
Number of pages4
Publication statusPublished - 2016

Publication series

NameSIGIR '16


  • chicken game
  • crowdsourcing
  • game theory
  • relevant assessments


Dive into the research topics of 'Identifying careless workers in crowdsourcing platforms: a game theory approach'. Together they form a unique fingerprint.

Cite this