Meta analyses

This note addresses some important aspects of meta-analysis that are often overlooked.

Observational studies differ from randomised trials in the respect that an observational study cannot be designed to prevent validity problems by randomisation, concealed allocation, and masking. The statistical analysis needs to be based on special considerations regarding internal validity and include adjustments to reduce bias. How well these issues have been addressed needs to be considered in detail and taken into account when conducting a meta-analysis (see also Faber et al. Meta-analyses including non-randomised studies of therapeutic interventions: a methodological review. BMC Medical Research Methodology 2016:35).

The same goes for multiplicity issues in meta-analyses of confirmatory randomised trials (see Bender et al. Attention should be given to multiplicity issues in systematic review. Journal of Clinical Epidemiology 2008;61(9):857-865).

Unlike fixed-effect models, which estimate a common effect, random-effect models estimate an average effect. The variability of the effects represented by their average may have consequences for the clinical interpretation of the findings. It can therefore be recommended to include a prediction interval in the forest plots to describe the variability (see also Hout et al. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open 2016;6:e010247).

The choice between fixed and random effect models is often based on I2, i.e. the percentage of variability due to heterogeneity across studies rather than sampling error. However, I2 depends on sample size, which can be misleading. A clinically relevant definition of the degree of between-study variability measured by tau2 would be more appropriate for this purpose (see Rücker et al. Undue reliance on I2 in assessing heterogeneity may mislead. BMC Med Res Methodol 2008 Nov 27;8:79).