The effects of variance-covariance misspecification in randomised controlled trials with high frequency data collection.

A simulation study in sport science.

Abstract

Due to problems of low statistical power in sport science with pre-post randomised controlled trials (RCTs), future studies may seek to conduct high frequency data collection to achieve adequate statistical power with feasible sample sizes. Increasing frequency of data collection requires more sophisticated statistical analyses that may have to account for increased complexity in variance-covariance structures. The purpose of this simulation study was to investigate the effects of misspecifying variance-covariance structures when fitting linear mixed models (LMMs) to realistic data generated across a range of intervention durations and measurement frequencies.

Simulated data were created to compare two interventions across either eight or sixteen weeks, with measurements made either once, twice or three times per week. Data were generated according to four models with different variance-covariance structures and fixed effects held constant. The data generating models included: 1) random intercept only; 2) correlated random intercepts and slopes; 3)&4) correlated random intercepts and slopes with autoregressive level one residuals (=0.3, or =0.6). The effects of both underspecifying and overspecifying variance-covariance structures with fitted LMMs were investigated focusing on type I errors of the average treatment effect when set to zero, and statistical power of the average treatment effect when set to a small but non-zero value. Statistical power of likelihood ratio tests to distinguish variance-covariance structures was also estimated. Monte Carlo simulations were performed across the different conditions with 5000 iterations included in each analysis.  

The results showed that underspecifying the variance-covariance structure by fitting a random intercept LMM when more complex structures existed created a high percentage of type I errors that ranged from ~7 to 40%. Percentages of type I errors increased with greater amounts of data from both a longer intervention and increased number of measurements per week. Results also demonstrated that statistical power decreased when the true data generating mechanism included autoregressive level one residuals compared to just correlated random intercepts and slopes. The decline in statistical power was influenced by both the amount of data available and the degree of autocorrelation. Statistical power dropped to its lowest (~0.55) when combining the least amount of data (e.g. eight weeks with one measurement per week) and highest autocorrelation (=0.6). Statistical power of likelihood ratio tests to correctly identify the existence of random slopes was influenced by intervention duration, with low statistical power (0.22 to 0.44) for a short intervention, and high statistical power (>0.99) for a long intervention.

In conclusion, correct specification of the variance-covariance structure is likely to be important for future RCTs in sport science conducted with high frequency measurements. Complex variance-covariance structures particularly with correlated level one residuals may require substantive increases in the number of participants recruited to obtain high statistical power. Researchers should be cautious of omitting random slopes where required given high percentages of type I errors this may cause. Caution is also required when testing for random slopes particularly with relatively small amounts of data that may be achieved with short interventions.