Normality assessments

The statistical section of research reports often includes information on how the distributional properties of the data have been assessed. Summaries of non-normally distributed variables are then presented using median and min-max instead of mean and standard deviation, and hypothesis testing is done using distribution-free tests instead of asymptotic tests. Several tests have been developed to test hypotheses about the distributional properties of a variable, such as the Kolmogorov-Smirnov test and the Shapiro-Wilk test. However, it is unclear what benefit the authors expect.

The hypothesis tested in these distributional tests is not about the distribution of the observed data, but about the distribution of the variable in the population from which the sample was drawn. It needs to be explained why the distribution of a variable in a fictitious population is relevant for how observed data is described.

Some investigators may claim that many tests, such as Student's t-test and ANOVA, are based on an assumption of a normal distribution, which is true but only relevant for small samples. The Central Limit Theorem implies that for large sample sizes, say 30 or more, mean values can be reliably tested regardless of the distribution of the tested variables.

Distribution-free (a.k.a. exact or non-parametric) tests may be useful for performing hypothesis tests with small samples, but while these tests can yield statistically significant results, they are not useful for estimating effect size, which is necessary when evaluating the clinical significance of an effect or difference.

Furthermore, simulation studies show that relying on the testing of assumptions before applying a Student's t-test can even be counterproductive, as these distribution tests often lead to wrong decisions. The use of Satterthwaite's or Welch's t-tests should be considered instead (see Rasch et al. The two-sample t test: pre-testing its assumptions does not pay off. Stat Papers 2011 Feb 1;52(1):219-231).