Crowdsourcing measures of design quality

Today there are numerous examples of collaborative online communities effectively creating innovative products (e.g., RepRap, Linux). But the potential of anonymous crowds to also engage in generative design, through the aggregation of many small contributions, is less clear. Although in recent years the “power of the crowd” has been repeatedly demonstrated in areas that range from image labelling to linguistic translation. The application of crowdsourcing in the fields of design research and creative innovation has been much slower to emerge. As a result, although there have been reports of systems and researchers using Internet crowdsourcing to carry out generative design, there are still many gaps in knowledge about the capability and limitations of the technology. For example on commercial crowdsourcing platforms, the relationship between remuneration and the final quality of designs has not been established, so it is unclear how much payment should be offered in order to ensure a particular standard of result. Key to investigating the relationship between the crowd’s remuneration and the value of their innovation is a robust method for quantifying the quality of the designs produced. This paper reports how payment for a design task (a 2D layout problem) was systematically varied and the quality of the output assessed through a separate crowdsourcing process. The work provides some interesting and valuable insight into how Crowdsourcing can be most effectively employed in design tasks.