No pain, no gain: revolutionary innovation comes at a cost

A team of researchers from the University of St Andrews has generated new insights into how human groups work to solve far-reaching problems together.

The new research, led by academics from the School of Biology, show humans generate collective solutions to technical challenges through incremental progress based on copying, interspersed with high-risk, high-gain innovations, to generate a unique form of collective intelligence.

The results, published in the journal Nature Communications (Wednesday 13 April), show how cultural transmission guides cumulative improvement when people solve realistic problems as a group. Collective improvement is produced by a combination of ‘tweaks’, copying the current best solution and adding a small improvement and ‘leaps’, radical new innovations that approach the problem in a new way.

The team of researchers analysed more than 21 million lines of computer code, entered in to online collaborative programming competitions organised over 14 years by the software company MathWorks®.

Our human ability to learn from each other and build a repertoire of knowledge, skills and traditions that increases and improves over generations, known as cumulative cultural evolution, has allowed our species to conquer every habitat of our planet and underpins our technological progress from moon-shots, medical advances and the progress of science itself.

Everyone understands how valuable this process is, but much less is known about how it works and how it evolved, apparently only in humans. This research shows that copying useful knowledge is key, but not sufficient for radical solutions that push the envelope and open up new domains for innovation.

‘Tweaks’ are a ‘safe’ strategy – producing only relatively modest improvements, but with lower risk of failing to perform at a level close to or exceeding the current best solution. In contrast ‘leaps’ are high risk – they most often result in a failure to improve on the current solution, and sometimes perform far worse, but on the flipside, when they do succeed, the improvements are typically much larger than those associated with ‘tweaks’.

Dr Elena Miu, who led the study and is now a postdoctoral researcher at Arizona State University, said: “Our study shows that merely sticking to the current paradigm and tweaking effective solutions is a safe strategy, that leads to steady, slow progress. Novel solutions, however, are tricky. Most fail, but occasionally a truly genius solution can really push the population towards remarkable accomplishments.

“This study also documents the emergence of a new form of collective intelligence arising from the recombination of existing ideas,” she continued. “The high level of copying in the contests allows populations to combine and synthesize good ideas from hundreds of individuals.”

The team, which also included Dr Luke Rendell and Professor Kevin Laland from the University of St Andrews, and Ned Gulley from MathWorks®, believe their results show that transformative innovation can only come at the price of accepting a risk of spectacular failure. The research findings can help guide the ways that we organise processes such as scientific funding to maximise progress.