When performing a meta-analysis, the investigator often experiences heterogeneity problems; the effects planned to be pooled in a meta-analysis are more heterogeneous than could be expected just from sampling variation. One solution to the problem is to try to estimate an average effect instead of a common one. Technically, this means fitting a random-effects model instead of a fixed-effects model. However, the problem may be more complicated than what is expected.
First, many investigators let the choice between fitting a fixed- or random-effect model be based on I2, i.e. the percentage of variability due to heterogeneity across studies, rather than statistical precision, i.e. sampling uncertainty. However, the statistical precision varies with sample size. Therefore, I2 also depends on the sample size, which can be misleading. A clinically more relevant definition of the degree of between-study variability, measured by τ2, is more appropriate. See Rücker et al. “Undue reliance on I2 in assessing heterogeneity may mislead”. BMC Med Res Methodol 2008 Nov 27;8:79.
Second, observational studies differ from randomised trials in that they cannot be designed to prevent validity problems by randomisation, concealed allocation, masking, etc. The statistical analysis needs to be based on considerations regarding internal validity, and it usually includes various, not always well defined, adjustments to reduce bias. The consequential multiplicity of effect estimates may affect the estimates’ variability.
Third, modelling effect estimates sometimes includes transforming the estimates to stabilise the variance. For example, arcsine‐based transformations are popular when pooling proportions. However, this transformation may violate the assumptions of the random-effects model and yield unreliable results; see Lin L, Xu C. Arcsine-based transformations for meta-analysis of proportions: Pros, cons, and alternatives. Health Science Reports. 2020;3(3):e178. doi:10.1002/hsr2.178.
Moreover, the variability of effects represented by their average may have important consequences for the clinical interpretation of the analysis results. Therefore, it is recommended to include a prediction interval to describe the variability. See also Hout et al. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open 2016;6:e010247.
Another form of heterogeneity may be reflected when performing a meta-analysis of multiple outcomes, some of which are perhaps not defined as primary. It should then be carefully considered whether a meta-analysis of these outcomes is appropriate. If it is, multiplicity issues may need to be addressed. See Bender, et al. Attention should be given to multiplicity issues in systematic reviews. Journal of Clinical Epidemiology [Internet]. 2008 Sep 1 [cited 2024 May 9];61(9):857–65. Available from: https://www.jclinepi.com/article/S0895-4356(08)00093-0/abstract.
