Preprint / Version 1

On the reproducibility of power analyses in motor behavior research




Metascience, Power analysis, Motor behavior, Open science


Recent metascience suggests that motor behavior research may be underpowered, on average. Researchers can perform a priori power analyses to ensure adequately powered studies. However, there are common pitfalls that can result in underestimating the required sample size for a given design and effect size of interest. Critical evaluation of power analyses requires successful analysis reproduction, which is conditional on the reporting of sufficient information. Here we attempted to reproduce every power analysis reported in articles (k = 84/635) in three motor behavior journals between January 2019 and June 2021. We reproduced 7% of analyses using the reported information, which increased to 43% when we assumed plausible values for missing parameters. Among studies that reported sufficient information to evaluate, 63% reported using the same statistical test in the power analysis as in the study itself, and in 77% the test addressed at least one of the identified hypotheses. Overall, power analyses were not commonly reported with sufficient information to ensure reproducibility. A non-trivial number of power analyses were also affected by common pitfalls. There is substantial opportunity to address the issue of underpowered research in motor behavior by increasing adoption of power analyses and ensuring reproducible reporting practices.


Metrics Loading ...


Abt, G., Boreham, C., Davison, G., Jackson, R., Nevill, A., Wallace, E., & Williams, M. (2020). Power, precision, and sample size estimation in sport and exercise science research. Journal of Sports Sciences, 38 (17), 1933–1935. 1776002

Aust, F., & Barth, M. (2020). papaja: Prepare reproducible APA journal articles with R Markdown.

Bacelar, M. F. B., Parma, J. O., Murrah, W. M., & Miller, M. W. (2022). Meta-analyzing enhanced expectancies on motor learning: Positive effects but methodological concerns. International Review of Sport and Exercise Psychology, 0 (0), 1–30.

Borg, D. N., Barnett, A., Caldwell, A. R., White, N., & Stewart, I. (2022). The bias for statistical significance in sport and exercise medicine.

Carnegie, E., Marchant, D., Towers, S., & Ellison, P. (2020). Beyond visual fixations and gaze behaviour. Using pupillometry to examine the mechanisms in the planning and motor performance of a golf putt. Human Movement Science, 71, 102622.

Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144 (4), 796–815.

Chang, W. (2022). Extrafont: Tools for using fonts.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.

Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349 (6251), aac4716.

Daou, M., Rhoads, J. A., Jacobs, T., Lohse, K. R., & Miller, M. W. (2019). Does limiting pre-movement time during practice eliminate the benefit of practicing while expecting to teach? Human Movement Science, 64, 153–163.

Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using g*power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41 (4), 1149–1160.

Fitzpatrick, P., de Jonge, E., & Warnes, G. R. (2019). Daff: Diff, patch and merge for data.frames.

Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing type s (sign) and type m (magnitude) errors. Perspectives on Psychological Science, 9 (6), 641–651.

Harry, J. R., Lanier, R., Nunley, B., & Blinch, J. (2019). Focus of attention effects on lower extremity biomechanics during vertical jump landings. Human Movement Science, 68, 102521.

Lakens, D. (2021). Sample size justification.

Lakens, D., & Caldwell, A. R. (2021). Simulation-based power analysis for factorial analysis of variance designs. Advances in Methods and Practices in Psychological Science, 4 (1), 2515245920951503.

Lohse, K., Buchanan, T., & Miller, M. (2016). Underpowered and overworked: Problems with data analysis in motor learning studies. Journal of Motor Learning and Development, 4 (1), 37–58.

Maier, M., Bartoš, F., Stanley, T. D., Shanks, D. R., Harris, A. J. L., & Wagenmakers, E.-J. (2022). No evidence for nudging after adjusting for publication bias. Proceedings of the National Academy of Sciences, 119 (31), e2200300119. 0300119

McCrum, C., Beek, J. van, Schumacher, C., Janssen, S., & Van Hooren, B. (2022). Sample size justifications in gait & posture. Gait & Posture, 92, 333–337. 6/j.gaitpost.2021.12.010

McKay, B., Bacelar, M., Parma, J. O., Miller, M. W., & Carter, M. J. (2022). The combination of reporting bias and underpowered study designs have substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: A meta-analysis. PsyArXiv.

McKay, B., Corson, A., Vinh, M.-A., Jeyarajan, G., Tandon, C., Brooks, H., Hubley, J., & Carter, M. J. (2022). Low prevalence of a priori power analyses in motor behavior research. SportRxiv.

McKay, B., Hussien, J., Vinh, M.-A., Mir-Orefice, A., Brooks, H., & Ste-Marie, D. M. (2022). Meta-analysis of the reduced relative feedback frequency effect on motor learning and performance. Psychology of Sport and Exercise, 102165. hsport.2022.102165

McKay, B., Yantha, Z. D., Hussien, J., Carter, M. J., & Ste-Marie, D. M. (in-press). Meta-analytic findings in the self-controlled motor learning literature: Underpowered, biased, and lacking evidential value. Meta-Psychology.

Mesquida, C., Murphy, J., Lakens, D., & Warne, J. (2022). Replication concerns in sports science: A narrative review of selected methodological issues in the field. SportRxiv.

Mumby, P. J. (2002). Statistical power of non-parametric tests: A quick guide for designing sampling strategies. Marine Pollution Bulletin, 44 (1), 85–87. S0025-326X(01)00097-2 R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing.

Rhoads, J. A., Daou, M., Lohse, K. R., & Miller, M. W. (2019). The effects of expecting to teach and actually teaching on motor learning. Journal of Motor Learning & Development, 7 (1), 84–105.

Robinson, M. A., Vanrenterghem, J., & Pataky, T. C. (2021). Sample size estimation for biomechanical waveforms: Current practice, recommendations and a comparison to discrete power analysis. Journal of Biomechanics, 122, 110451. j.jbiomech.2021.110451

Rudis, B., & Gandy, D. (2019). Waffle: Create waffle chart visualizations. https://gitlab.c om/hrbrmstr/waffle

Twomey, R., Yingling, V., Warne, J., Schneider, C., McCrum, C., Atkins, W., Murphy, J., Medina, C. R., Harlley, S., & Caldwell, A. (2021). The nature of our literature: A registered report on the positive result rate and reporting practices in kinesiology. Communications in Kinesiology, 1 (3).

Uiga, L., Poolton, J. M., Capio, C. M., Wilson, M. R., Ryu, D., & Masters, R. S. W. (2020). The role of conscious processing of movements during balance by young and older adults. Human Movement Science, 70, 102566.

Ushey, K. (2022). Renv: Project environments.

Vohs, K., Schmeichel, B., Lohmann, S., Gronau, Q. F., Finley, A. J., Wagenmakers, E.-J., & Albarracín, D. (2021). A multi-site preregistered paradigmatic test of the ego depletion effect.

Westfall, J. (2015). PANGEA: Power analysis for general ANOVA designs. Unpublished Manuscript. Available at Http://Jakewestfall. Org/Publications/Pangea. Pdf, 4.

Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., . . . Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4 (43), 1686.

Wulf, G., & Lewthwaite, R. (2016). Optimizing performance through intrinsic motivation and attention for learning: The OPTIMAL theory of motor learning. Psychonomic Bulletin & Review, 23 (5), 1382–1414.