Is it time to rethink pre-post randomised controlled trials in strength and conditioning?

A review of statistical approaches with derivations and simulations.

Abstract

Randomised controlled trials (RCTs) featuring a single measurement at baseline and post-intervention is the most common study type used to build an evidence base in strength and conditioning. The purpose of this simulation study with mathematical derivations was to explore the accuracy of inferences made with this design and the factors that may increase the proportion of Type I and Type II errors.

Realistic pre-post RCT data were simulated for strength and conditioning interventions based on parameters obtained from recent large meta-analyses. A total of nine outcomes from three domains (strength, power, and speed) were simulated whilst adjusting for a range of factors including sample size (N=10,15,25, and 50), the average treatment effect (ATE), the relationship between baseline and change scores, the amount of baseline imbalance, and measurement error. Four categories of ATE were used including zero, to investigate proportion of Type I errors, and small, medium and large ATEs to investigate proportion of Type II errors. Monte-Carlo simulation with 10,000 iterations were performed for each scenario using three different statistical tests including ANOVA, T-test on post-intervention values, and ANCOVA.

Proportion of Type I errors were close to 5% when testing a single outcome and increased to ~10-13% when testing three outcomes, and ~20-30% when testing nine outcomes. ANCOVA was shown to be the most precise statistical test with increased precision obtained with baseline imbalance and relationships between baseline and change scores. Sample size, ATE and measurement error were shown to be the most relevant factors controlling Type II errors. In the worst-case scenario (e.g., N=10, small ATE and large measurement error) statistical power was likely to be ~0.1. Even with sample sizes of 50, statistical power was unlikely to exceed 0.4 when combined with small ATE and large measurement error.

The results of this study show that sample sizes and ATE commonly investigated in strength and conditioning are likely to lead to a high proportion of inferential errors. More novel study designs and analysis approaches are required to account for these statistical challenges whilst adhering to the resource constraints that typically exist within the discipline.