Relationship between quality and payment in crowdsourced design

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

10 Citations (Scopus)

Abstract

In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to applied collaborative intelligence to areas that range from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. For example on crowdsourcing platforms, like Amazon’s Mechanical Turk, the relationship between remuneration and the final quality of designs has not been established, so it is unclear how much payment should be offered to ensure a particular standard of result. To investigate the relationship between crowd’s remuneration and the quality of their innovation, this paper reports how payment for a 2D interior design task (living room layout) was systematically varied and the quality of the output assessed by a ranking process designed that was also crowdsourced. Information about individual Mturk workers who participated in the study was also collected. The results suggest that while that average design quality only slowly increases, the quality of the “best” design generated by the crowd improved dramatically with payment levels. In other words, increasing monetary rewards does not improve the average creativity of design but rather it increases the chance of an excellent solution being generated by an individual in the crowd.
LanguageEnglish
Title of host publicationProceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD)
Subtitle of host publicationMay 21-23, 2014, National Tsing Hua University, Hsinchu, Taiwan
EditorsJiang-Liang Hou
Place of PublicationPiscataway, NJ.
PublisherIEEE
Pages499-504
Number of pages6
ISBN (Print)9781479937769
DOIs
Publication statusPublished - 2014

Fingerprint

Innovation
Internet
Image analysis

Keywords

  • Amazon Mechanical Turk (MTurk)
  • crowdscourcing
  • crowdsourcing demographics
  • design productivity
  • interior design
  • open innovation
  • architectural design
  • innovation
  • interactive computer systems
  • internet
  • product design

Cite this

Wu, H., Corney, J., & Grant, M. (2014). Relationship between quality and payment in crowdsourced design. In J-L. Hou (Ed.), Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD): May 21-23, 2014, National Tsing Hua University, Hsinchu, Taiwan (pp. 499-504). [6839905] Piscataway, NJ.: IEEE. https://doi.org/10.1109/CSCWD.2014.6846895
Wu, Hao ; Corney, Jonathan ; Grant, Michael. / Relationship between quality and payment in crowdsourced design. Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD): May 21-23, 2014, National Tsing Hua University, Hsinchu, Taiwan. editor / Jiang-Liang Hou. Piscataway, NJ. : IEEE, 2014. pp. 499-504
@inproceedings{a3947c5070a24d89b3fcf70c8cd27833,
title = "Relationship between quality and payment in crowdsourced design",
abstract = "In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to applied collaborative intelligence to areas that range from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. For example on crowdsourcing platforms, like Amazon’s Mechanical Turk, the relationship between remuneration and the final quality of designs has not been established, so it is unclear how much payment should be offered to ensure a particular standard of result. To investigate the relationship between crowd’s remuneration and the quality of their innovation, this paper reports how payment for a 2D interior design task (living room layout) was systematically varied and the quality of the output assessed by a ranking process designed that was also crowdsourced. Information about individual Mturk workers who participated in the study was also collected. The results suggest that while that average design quality only slowly increases, the quality of the “best” design generated by the crowd improved dramatically with payment levels. In other words, increasing monetary rewards does not improve the average creativity of design but rather it increases the chance of an excellent solution being generated by an individual in the crowd.",
keywords = "Amazon Mechanical Turk (MTurk), crowdscourcing, crowdsourcing demographics, design productivity, interior design, open innovation, architectural design, innovation, interactive computer systems, internet, product design",
author = "Hao Wu and Jonathan Corney and Michael Grant",
year = "2014",
doi = "10.1109/CSCWD.2014.6846895",
language = "English",
isbn = "9781479937769",
pages = "499--504",
editor = "Jiang-Liang Hou",
booktitle = "Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD)",
publisher = "IEEE",

}

Wu, H, Corney, J & Grant, M 2014, Relationship between quality and payment in crowdsourced design. in J-L Hou (ed.), Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD): May 21-23, 2014, National Tsing Hua University, Hsinchu, Taiwan., 6839905, IEEE, Piscataway, NJ., pp. 499-504. https://doi.org/10.1109/CSCWD.2014.6846895

Relationship between quality and payment in crowdsourced design. / Wu, Hao; Corney, Jonathan; Grant, Michael.

Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD): May 21-23, 2014, National Tsing Hua University, Hsinchu, Taiwan. ed. / Jiang-Liang Hou. Piscataway, NJ. : IEEE, 2014. p. 499-504 6839905.

Research output: Chapter in Book/Report/Conference proceedingConference contribution book

TY - GEN

T1 - Relationship between quality and payment in crowdsourced design

AU - Wu, Hao

AU - Corney, Jonathan

AU - Grant, Michael

PY - 2014

Y1 - 2014

N2 - In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to applied collaborative intelligence to areas that range from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. For example on crowdsourcing platforms, like Amazon’s Mechanical Turk, the relationship between remuneration and the final quality of designs has not been established, so it is unclear how much payment should be offered to ensure a particular standard of result. To investigate the relationship between crowd’s remuneration and the quality of their innovation, this paper reports how payment for a 2D interior design task (living room layout) was systematically varied and the quality of the output assessed by a ranking process designed that was also crowdsourced. Information about individual Mturk workers who participated in the study was also collected. The results suggest that while that average design quality only slowly increases, the quality of the “best” design generated by the crowd improved dramatically with payment levels. In other words, increasing monetary rewards does not improve the average creativity of design but rather it increases the chance of an excellent solution being generated by an individual in the crowd.

AB - In recent years, the “power of the crowd” has been repeatedly demonstrated and various Internet platforms have been used to applied collaborative intelligence to areas that range from open innovation to image analysis. However, crowdsourcing applications in the fields of design research and creative innovation have been much slower to emerge. So, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. For example on crowdsourcing platforms, like Amazon’s Mechanical Turk, the relationship between remuneration and the final quality of designs has not been established, so it is unclear how much payment should be offered to ensure a particular standard of result. To investigate the relationship between crowd’s remuneration and the quality of their innovation, this paper reports how payment for a 2D interior design task (living room layout) was systematically varied and the quality of the output assessed by a ranking process designed that was also crowdsourced. Information about individual Mturk workers who participated in the study was also collected. The results suggest that while that average design quality only slowly increases, the quality of the “best” design generated by the crowd improved dramatically with payment levels. In other words, increasing monetary rewards does not improve the average creativity of design but rather it increases the chance of an excellent solution being generated by an individual in the crowd.

KW - Amazon Mechanical Turk (MTurk)

KW - crowdscourcing

KW - crowdsourcing demographics

KW - design productivity

KW - interior design

KW - open innovation

KW - architectural design

KW - innovation

KW - interactive computer systems

KW - internet

KW - product design

UR - http://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=6839905

UR - http://2014.cscwd.org/

U2 - 10.1109/CSCWD.2014.6846895

DO - 10.1109/CSCWD.2014.6846895

M3 - Conference contribution book

SN - 9781479937769

SP - 499

EP - 504

BT - Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD)

A2 - Hou, Jiang-Liang

PB - IEEE

CY - Piscataway, NJ.

ER -

Wu H, Corney J, Grant M. Relationship between quality and payment in crowdsourced design. In Hou J-L, editor, Proceedings of the 2014 IEEE 18th International Conference on Computer Supported Cooperative Work in Design (CSCWD): May 21-23, 2014, National Tsing Hua University, Hsinchu, Taiwan. Piscataway, NJ.: IEEE. 2014. p. 499-504. 6839905 https://doi.org/10.1109/CSCWD.2014.6846895