Undue emphasis on p-values

As indicated in another post, statistical significance and p-values are often misunderstood, and this is not a new problem. Frank Yates, one of the pioneers of 20th century statistics, stated as early as 1951 that the most common weakness is the failure to recognise that estimates of treatment effects, together with estimates of the errors to which they are subject, are the quantities of primary interest in clinical research, not p-values (Yates F. The influence of Statistical Methods for Research Workers on the development of the science of statistics. J Am Stat Assoc 1951;46:19-34).

What Yates was referring to is that a standard p-value can be used to test whether a treatment effect exists, regardless of the size of the effect, but this is often done, especially in observational studies, without knowing the risk of a false positive or false negative test result. Furthermore, p-values depend on the sample size. In a large sample, such as in a registry study, negligible effects may be statistically significant, and in a small sample, even major effects may be statistically nonsignificant. In addition, because of sampling variability, the effect size observed in a sample does not tell us much about the true size of a statistically significant or nonsignificant effect. Finally, describing a statistically nonsignificant test result as "no effect", which is an annoying habit in medical research reports, completes the mistake.

A better approach is to try to distinguish between clinically relevant effects and negligible or nonexistent effects using a confidence interval. Confidence intervals describe the inferential uncertainty of an estimate and are presented with lower and upper limits. A clinically important effect is indicated by a confidence interval that excludes the minimum clinically important size, regardless of whether or not the effect is statistically significant. Similar approaches are used in the evaluation of equivalence trials and non-inferiority trials.