New research reveals a ‘Hidden universe of uncertainty’

New research on the reliability – and credibility – of scientific research has been published by Dr Eike Mark Rinke, Lecturer in Politics and Media.

The research, ‘Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty’ has just been published in one of the most prestigious scientific journals in the world, Proceedings of the National Academy of Sciences of the United States of America (PNAS).

The research team coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions.

The research did not prove that therefore all science is inherently untrustworthy or unreliable, but that researchers should exercise humility  and awareness of the complexities of human societies and behaviour and strive to better account for the uncertainty in their work.

The results showed more than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation.

Dr Rinke said: “The main driver behind this research was a concern about the reliability – and credibility – of scientific research. So, we asked ourselves, first, “Will different researchers converge on similar findings when analysing the same data in a realistic social science setting?” and then, second, “What could be the sources of any variation in the results that differing researcher would come up with when trying to answer the same question with the same data?”

“The answer to the second question is the main contribution of this paper. We found that obvious potential explanations for why different researchers would arrive at different results (expertise, individual beliefs about what they will find, even major analytical decisions) do not actually hold much water: even when we account for these factors a hidden universe of uncertainty remains!

We do think that we show a very important source of unreliability in scientific research (idiosyncratic, often tiny, conscious and unconscious decisions in the analytical process) that has not yet been appreciated enough and cannot be easily explained away. 

Dr Rinke emphasised the research did not prove that therefore all science is inherently untrustworthy or unreliable, but that researchers should exercise humility and awareness of the complexities of human societies and behaviour and strive to better account for the uncertainty in their work.

“Rather, we argue that scientists, especially those working with the complexities of human societies and behaviour, should be a) humble when it comes to the “truth value” of their findings (they should always be treated as tentative, or the “best current” findings only, not absolute truths!) and b) striving to account as much as possible for the uncertainty in any work involving lots of small analytical decisions – the most important way of doing this is by being as open and transparent about one’s research pipeline as possible.”