The illusion of knowing

"How do we know that cigarettes cause lung cancer?" the professor asked, going on to say, "It has never been tested in a clinical trial. The implication was that a clinical trial was needed to know for sure, and that observational studies could at best provide suggestions for further research. There was no discussion of the ethical and practical problems of conducting clinical trials to assess harmful effects on participants.

Clinical trials, unlike observational studies, can be designed to reduce the uncertainty of research results by eliminating selection bias and confounding through randomisation, allocation concealment and treatment masking, but they cannot provide completely certain results. The uncertainty introduced by sampling from a population is related to sample size and is impossible to eliminate when the population being studied is infinite (i.e. today's medical research is usually done for tomorrow's patients). However, a well-designed clinical trial can, at least in principle, provide information that suffers only from aleatory uncertainty (i.e. the inherent randomness of sampling). Systematic reviews, which combine the results of several independent clinical trials, are generally considered to provide the most reliable evidence.

The same is not true for the results of observational studies. These rely on statistical adjustments made by the investigators according to their assumptions, and whether or not these assumptions are met is usually unknown. Another problem is that the data needed to make the adjustments are not always available, and for practical reasons simplifications in the calculations are usually necessary. Consequently, the results suffer from both aleatoric and epistemic uncertainty (i.e. lack of knowledge about something that could, in principle, be known).

All empirical research results, whether experimental or observational, are uncertain to some degree. Yet many of them form the basis of what we take to be known. The willingness of society or groups to accept some uncertain evidence as indicating truth and other uncertain evidence as reflecting error is not easy to explain. It doesn't seem to be directly related to the degree of uncertainty itself, and as evidence accumulates over time, opinions can change. However, economic, social, political or other factors are likely to be important in determining what is considered to be true. A critical interpretation of proclaimed "truths" and a reminder of the American statistician Carroll D. Wright's 1889 statement that "figures do not lie, but liars figure" is crucial to avoid being deceived.

Phrases such as "we have demonstrated" and "we have shown" are often used in research reports, too often without empirical support. The primary purpose of scientific journals is to document the studies that have been done, their results, and the uncertainty of those results. My advice to authors is to accept uncertainty and focus on objective evidence rather than trying to convince the reader with subjective ideas.