The uncertainty described by a confidence interval is related to sampling. The sampling uncertainty of a study is known as the study’s statistical precision. The magnitude of the uncertainty depends partly on the sample size. Other forms of uncertainty (non-sampling uncertainty) are related to validity rather than precision, and these also need to be considered when designing or analysing a study. For example, various selection mechanisms may affect the representativity of sampled subjects, imbalance in the distributions of background factors may cause confounding issues, and systematic errors in the information collection can severely affect the validity of the findings in a study.
Uncertainty about the validity of a study’s findings can be reduced in two ways: by designing the study to eliminate the problems or by adjusting the results in the statistical analysis. The first alternative requires an experimental study, for example, a trial with randomisation, concealed treatment allocation, and treatment masking. The second alternative can be used in the statistical analysis of an observational study. However, while the design of a randomised trial eliminates all known and unknown confounding factors, the confounding adjustments in an observational study can only be performed for suspected confounding factors with available measurements in the used dataset. Furthermore, analysis adjustments are always based on several assumptions, and the fulfilment of such assumptions is usually unknown. Observational studies, therefore, yield more uncertain results than randomised trials. On the other hand, experimental studies are more expensive, require more work, and can, for ethical reasons, only be used to study effects that are beneficial for the participants.
Many laboratory experiments are designed without considering the uncertainty of the findings, which is especially unfortunate as evaluation issues typically are addressed using statistical significance as a (misconstrued) token of practical relevance. In very few cases, a standard sample size of n = 3 will give sufficient statistical precision. Not surprisingly, findings from laboratory experiments play a leading role in the current methodological crisis, in which published research findings cannot be reproduced.
