Relationship between quality and payment in crowdsourced design

Hao Wu, Jonathan Corney, Michael Grant

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

19 Citations (Scopus)


In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to applied collaborative intelligence to areas that range from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. For example on crowdsourcing platforms, like Amazon’s Mechanical Turk, the relationship between remuneration and the final quality of designs has not been established, so it is unclear how much payment should be offered to ensure a particular standard of result. To investigate the relationship between crowd’s remuneration and the quality of their innovation, this paper reports how payment for a 2D interior design task (living room layout) was systematically varied and the quality of the output assessed by a ranking process designed that was also crowdsourced. Information about individual Mturk workers who participated in the study was also collected. The results suggest that while that average design quality only slowly increases, the quality of the “best” design generated by the crowd improved dramatically with payment levels. In other words, increasing monetary rewards does not improve the average creativity of design but rather it increases the chance of an excellent solution being generated by an individual in the crowd.
Original languageEnglish
Title of host publicationProceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD)
Subtitle of host publicationMay 21-23, 2014, National Tsing Hua University, Hsinchu, Taiwan
EditorsJiang-Liang Hou
Place of PublicationPiscataway, NJ.
Number of pages6
ISBN (Print)9781479937769
Publication statusPublished - 2014


  • Amazon Mechanical Turk (MTurk)
  • crowdscourcing
  • crowdsourcing demographics
  • design productivity
  • interior design
  • open innovation
  • architectural design
  • innovation
  • interactive computer systems
  • internet
  • product design


Dive into the research topics of 'Relationship between quality and payment in crowdsourced design'. Together they form a unique fingerprint.

Cite this