BACKGROUND: Pre-post studies report a measure before and after a change in treatment where there is no parallel control group. Meta-analyses of pre-post studies often utilize standardized mean difference (SMD), which incorrectly assumes independence of the pre-change and post-change results. However, effect sizes such as standardized mean change (SMC) can account for the nonindependence of the pre-change and post-change results. OBJECTIVES: We sought to explore the conditions under which SMC and SMD provide equivalent or disparate results through the simulation of meta-analyses and the re-analysis of a published meta-analysis.
METHODS: We simulated 1,470,000 datasets across 147 combinations of pre-post change and the correlation between the pre and post measures. Each dataset was meta-analyzed using SMC and SMD. The statistical significance of the pooled estimate was recorded for each meta-analysis. For each combination of variables, the frequency of statistically significant pooled estimates when using SMC and SMD were compared. Additionally, a recent meta-analysis of moderate-to-vigorous-intensity physical activity in children was re-analyzed using both SMC and SMD.
RESULTS: Simulations of meta-analyses revealed that a positive correlation between the pre-change and post-change measure resulted in pooled effect sizes being more likely to be statistically significant for SMC than SMD. However, for negative correlations, the reverse was observed. When performing a meta-analysis of moderate-to-vigorous-intensity physical activity using SMC, the reported pooled estimate was statistically significant (SMC = −0.19; 95% confidence interval, −0.35 to −0.04). When the same data were meta-analyzed using SMD, the reported pooled estimate was nonsignificant (SMD = −0.18; 95% confidence interval, −0.40 to 0.04).
CONCLUSIONS: Simulations and a re-analysis of a published meta-analysis have shown that the use of SMD and SMC may lead to contrasting results. Importantly, the use of SMD may lead to erroneous conclusions. As such, SMC should be used when meta-analyzing pre-post studies.
