Many medical publications present the results of more or less random statistical tests of data available in datasets collected for other reasons. The statistical test results are then interpreted according to the investigators’ prejudices, suitable hypotheses are formulated, and reports are written pretending that the studies from beginning to end were performed to test the specific (post-hoc) hypotheses. This way of conducting an observational study can be described as a “fishing expedition.” It does not represent a reasonable scientific approach as it purposely misleads reviewers and readers about the uncertainty of the findings.
To find more reliable empirical support for a medical hypothesis, the hypothesis itself should be the starting point. As experiments on humans require ethics approval, and as the data collection and statistical analysis depend on how a medical hypothesis has been operationalised, it is necessary first to consider essential tasks, such as finding a suitable estimand and estimator, to decide necessary adjustments of the estimator, to decide how missing values should be handled, to develop a strategy for addressing multiplicity issues, and to calculate sample size. If these are involved, ethics committees and regulatory authorities require pre-specification of these considerations in a study protocol. Several medical journals contribute to this process by reviewing and publishing study protocols.
Some issues addressed in trial analysis plans, such as the strategy for addressing multiplicity issues, may be irrelevant in observational studies. However, several other issues are crucial and should be included in the presentation when a study is reported. For example, without knowledge about estimand in an investigation (the quantity to be estimated in accordance with the studied hypothesis, independent of the operational assumptions that are necessary for the estimator, e.g. vis-à-vis missing values and multivariable adjustments), it is impossible to know if the performed statistical procedures and the presented estimate are appropriate.
