On the reproducibility of power analyses in motor behavior research
DOI:
https://doi.org/10.51224/SRXIV.184Keywords:
Metascience, Power analysis, Motor behavior, Open scienceAbstract
Recent metascience suggests that motor behavior research may be underpowered, on average. Researchers can perform a priori power analyses to ensure adequately powered studies. However, there are common pitfalls that can result in underestimating the required sample size for a given design and effect size of interest. Critical evaluation of power analyses requires successful analysis reproduction, which is conditional on the reporting of sufficient information. Here we attempted to reproduce every power analysis reported in articles (k = 84/635) in three motor behavior journals between January 2019 and June 2021. We reproduced 7% of analyses using the reported information, which increased to 43% when we assumed plausible values for missing parameters. Among studies that reported sufficient information to evaluate, 63% reported using the same statistical test in the power analysis as in the study itself, and in 77% the test addressed at least one of the identified hypotheses. Overall, power analyses were not commonly reported with sufficient information to ensure reproducibility. A non-trivial number of power analyses were also affected by common pitfalls. There is substantial opportunity to address the issue of underpowered research in motor behavior by increasing adoption of power analyses and ensuring reproducible reporting practices.
Metrics
References
Abt, G., Boreham, C., Davison, G., Jackson, R., Nevill, A., Wallace, E., & Williams, M. (2020). Power, precision, and sample size estimation in sport and exercise science research. Journal of Sports Sciences, 38 (17), 1933–1935. https://doi.org/10.1080/02640414.2020. 1776002
Aust, F., & Barth, M. (2020). papaja: Prepare reproducible APA journal articles with R Markdown. https://github.com/crsh/papaja
Bacelar, M. F. B., Parma, J. O., Murrah, W. M., & Miller, M. W. (2022). Meta-analyzing enhanced expectancies on motor learning: Positive effects but methodological concerns. International Review of Sport and Exercise Psychology, 0 (0), 1–30. https://doi.org/10.1080/1750984X.2022.2042839
Borg, D. N., Barnett, A., Caldwell, A. R., White, N., & Stewart, I. (2022). The bias for statistical significance in sport and exercise medicine. https://doi.org/10.31219/osf.io/t7yfc
Carnegie, E., Marchant, D., Towers, S., & Ellison, P. (2020). Beyond visual fixations and gaze behaviour. Using pupillometry to examine the mechanisms in the planning and motor performance of a golf putt. Human Movement Science, 71, 102622. https://doi.org/10.1016/j.humov.2020.102622
Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144 (4), 796–815. https://doi.org/10.1037/xge0000083
Chang, W. (2022). Extrafont: Tools for using fonts. https://CRAN.R-project.org/package=extrafont
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.
Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349 (6251), aac4716. https://doi.org/10.1126/science.aac4716
Daou, M., Rhoads, J. A., Jacobs, T., Lohse, K. R., & Miller, M. W. (2019). Does limiting pre-movement time during practice eliminate the benefit of practicing while expecting to teach? Human Movement Science, 64, 153–163. https://doi.org/10.1016/j.humov.2018.11.017
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using g*power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41 (4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149
Fitzpatrick, P., de Jonge, E., & Warnes, G. R. (2019). Daff: Diff, patch and merge for data.frames. https://CRAN.R-project.org/package=daff
Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing type s (sign) and type m (magnitude) errors. Perspectives on Psychological Science, 9 (6), 641–651.
Harry, J. R., Lanier, R., Nunley, B., & Blinch, J. (2019). Focus of attention effects on lower extremity biomechanics during vertical jump landings. Human Movement Science, 68, 102521. https://doi.org/10.1016/j.humov.2019.102521
Lakens, D. (2021). Sample size justification.
Lakens, D., & Caldwell, A. R. (2021). Simulation-based power analysis for factorial analysis of variance designs. Advances in Methods and Practices in Psychological Science, 4 (1), 2515245920951503. https://doi.org/10.1177/2515245920951503
Lohse, K., Buchanan, T., & Miller, M. (2016). Underpowered and overworked: Problems with data analysis in motor learning studies. Journal of Motor Learning and Development, 4 (1), 37–58. https://doi.org/10.1123/jmld.2015-0010
Maier, M., Bartoš, F., Stanley, T. D., Shanks, D. R., Harris, A. J. L., & Wagenmakers, E.-J. (2022). No evidence for nudging after adjusting for publication bias. Proceedings of the National Academy of Sciences, 119 (31), e2200300119. https://doi.org/10.1073/pnas.220 0300119
McCrum, C., Beek, J. van, Schumacher, C., Janssen, S., & Van Hooren, B. (2022). Sample size justifications in gait & posture. Gait & Posture, 92, 333–337. https://doi.org/10.101 6/j.gaitpost.2021.12.010
McKay, B., Bacelar, M., Parma, J. O., Miller, M. W., & Carter, M. J. (2022). The combination of reporting bias and underpowered study designs have substantially exaggerated the motor learning benefits of self-controlled practice and enhanced expectancies: A meta-analysis. PsyArXiv. https://doi.org/10.31234/osf.io/3nhtc
McKay, B., Corson, A., Vinh, M.-A., Jeyarajan, G., Tandon, C., Brooks, H., Hubley, J., & Carter, M. J. (2022). Low prevalence of a priori power analyses in motor behavior research. SportRxiv. https://sportrxiv.org/index.php/server/preprint/view/175
McKay, B., Hussien, J., Vinh, M.-A., Mir-Orefice, A., Brooks, H., & Ste-Marie, D. M. (2022). Meta-analysis of the reduced relative feedback frequency effect on motor learning and performance. Psychology of Sport and Exercise, 102165. https://doi.org/10.1016/j.psyc hsport.2022.102165
McKay, B., Yantha, Z. D., Hussien, J., Carter, M. J., & Ste-Marie, D. M. (in-press). Meta-analytic findings in the self-controlled motor learning literature: Underpowered, biased, and lacking evidential value. Meta-Psychology. https://doi.org/10.31234/osf.io/8d3nb
Mesquida, C., Murphy, J., Lakens, D., & Warne, J. (2022). Replication concerns in sports science: A narrative review of selected methodological issues in the field. SportRxiv. https://sportrxiv.org/index.php/server/preprint/view/127
Mumby, P. J. (2002). Statistical power of non-parametric tests: A quick guide for designing sampling strategies. Marine Pollution Bulletin, 44 (1), 85–87. https://doi.org/10.1016/ S0025-326X(01)00097-2 R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/
Rhoads, J. A., Daou, M., Lohse, K. R., & Miller, M. W. (2019). The effects of expecting to teach and actually teaching on motor learning. Journal of Motor Learning & Development, 7 (1), 84–105.
Robinson, M. A., Vanrenterghem, J., & Pataky, T. C. (2021). Sample size estimation for biomechanical waveforms: Current practice, recommendations and a comparison to discrete power analysis. Journal of Biomechanics, 122, 110451. https://doi.org/10.1016/ j.jbiomech.2021.110451
Rudis, B., & Gandy, D. (2019). Waffle: Create waffle chart visualizations. https://gitlab.c om/hrbrmstr/waffle
Twomey, R., Yingling, V., Warne, J., Schneider, C., McCrum, C., Atkins, W., Murphy, J., Medina, C. R., Harlley, S., & Caldwell, A. (2021). The nature of our literature: A registered report on the positive result rate and reporting practices in kinesiology. Communications in Kinesiology, 1 (3). https://doi.org/10.51224/cik.v1i3.43
Uiga, L., Poolton, J. M., Capio, C. M., Wilson, M. R., Ryu, D., & Masters, R. S. W. (2020). The role of conscious processing of movements during balance by young and older adults. Human Movement Science, 70, 102566. https://doi.org/10.1016/j.humov.2019.102566
Ushey, K. (2022). Renv: Project environments. https://CRAN.R-project.org/package=renv
Vohs, K., Schmeichel, B., Lohmann, S., Gronau, Q. F., Finley, A. J., Wagenmakers, E.-J., & Albarracín, D. (2021). A multi-site preregistered paradigmatic test of the ego depletion effect.
Westfall, J. (2015). PANGEA: Power analysis for general ANOVA designs. Unpublished Manuscript. Available at Http://Jakewestfall. Org/Publications/Pangea. Pdf, 4.
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M., Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., . . . Yutani, H. (2019). Welcome to the tidyverse. Journal of Open Source Software, 4 (43), 1686. https://doi.org/10.21105/joss.01686
Wulf, G., & Lewthwaite, R. (2016). Optimizing performance through intrinsic motivation and attention for learning: The OPTIMAL theory of motor learning. Psychonomic Bulletin & Review, 23 (5), 1382–1414.
Downloads
Posted
Categories
License
Copyright (c) 2022 Brad McKay, Mariane F. B. Bacelar, Michael J. Carter
This work is licensed under a Creative Commons Attribution 4.0 International License.
