Jonas Ranstam

Statistical consultant

Scientific evidence

In contrast to deductive sciences like mathematics, modern medical research is based on the fundamental principle that scientific findings must be evidence-based, have empirical support, i.e. that new knowledge is developed using observations and experiments. Randomised trials and epidemiological studies, in the way we now design and analyse them, did not take place until after world war II. Medical publications were earlier authority-based, mainly descriptive case reports or presentations of subjective hypotheses and expert opinions. An important problem with expert opinions, hypotheses and case reports is, however, that their relevance for future patients is not easily assessed.

Research reports must distinguish between assumptions and observations and describe the limitations of what has been observed. The uncertainty of the findings must be presented to the reader. It is not sufficient to just describe observations and outcomes that support the author’s conclusions. Circumstances that raise questions about the validity of the conclusions must also be disclosed. The author needs to be impartial vis-à-vis the findings and objectively quantify their uncertainty. Statistics, known as the science of uncertainty (1), provides a scientific methodology for this.

The trends of modern society have, however, a major tendency towards alarmism and exaggeration. This is a phenomenon that is related to the flow of information and disinformation on the internet and its effects on news media, which as a consequence, now are fighting for their survival. The current relationship between the internet and alarmism is similar to the relationship between the portable transistor radio and the development of the teenage culture during the 1960s.

The effects of the internet have many different consequences, and the publication of scientific findings is clearly affected. Scientists face a working environment that is characterised as “publish-or-perish”. An impartiality requirement for assessing the uncertainty of scientific findings is not always credible. Statistics risks being used as a tool for exaggerating empirical support in order to improve the chances of getting manuscripts published, the key to professional fame, financial support, and career advancement.

The trust in science is of course threatened (2), and a severe reproducibility crisis has also been widely discussed in medical science (3). Several important research findings have not been possible to reproduce, and this represents a serious waste of research resources with consequences for us all (4).

To improve the situation we both need to develop a more effective system for peer review and to find a way to help readers become more critical in their interpretations of published results.

References

1. Evans MJ, Rosenthal JS. Probability and Statistics: The Science of Uncertainty. W.H.Freeman Co Ltd, United States, 2010.

2. The Guardian 20 June 2017. Exaggerations threaten public trust in science says leading statistician.Retrieved from https://www.theguardian.com/science/2017/jun/28/ exaggerations-threaten-public-trust-in-science-leading-statistician-david-spiegelhalter.

3. Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature 2012;483, 531.

4. Harris R. Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hopes, and Wastes Billions. Basic Books, United States, 2017.