DOI of the published article https://doi.org/10.51224/cik.2023.53
A Survey on the Attitudes Towards and Perception of Reproducibility and Replicability in Sports and Exercise Science
DOI:
https://doi.org/10.51224/SRXIV.234Keywords:
replication, reproducibility, statistics, research methods, open scienceAbstract
There are formal calls for increased reproducibility and replicability in sports and exercise science, yet there is minimal information on the overall knowledge of these concepts at a field-wide level. Therefore, we conducted a survey on the attitudes and perceptions of sports and exercise science researchers towards reproducibility and replicability. Descriptive statistics (e.g., proportion of responses), and thematic analysis, were utilized to characterize the responses. Of the 511 respondents, 42% (n = 217) believe there is a significant crisis of reproducibility or replicability in sports and exercise science while 36% (n = 182) believe there is a slight crisis. 3% (n = 15) of respondents believe there is no crisis while 19% (n = 95) did not know. Four themes were generated in the thematic analysis: the research and publishing culture, educational barriers to research integrity, research responsibility to ensure reproducibility and replicability, and current practices facilitating reproducibility and replicability. Researchers believe that engaging in open science can be detrimental to career opportunities due to lack of incentives. They also feel journals are a barrier to reproducible and replicable research due to high publication charges and a focus on novelty. Statistical expertise was identified as a key factor for improving reproducibility and replicability in the future, particularly, a better understanding of study design and different statistical techniques. Statistical education should be prioritised for early career researchers which could positively affect publication and peer review. Researchers must accept responsibility for reproducibility and replicability with thorough project design, appropriate planning of analyses, and transparent reporting practices.
Metrics
References
Abt, G., Boreham, C., Davison, G., Jackson, R., Wallace, E. and Williams, A. M. (2021) ‘Registered Reports in the Journal of Sports Sciences’, Journal of Sports Sciences, pp. 1789–1790. doi: 10.1080/02640414.2021.1950974.
Allen, C. and Mehler, D. M. A. (2019) ‘Open science challenges, benefits and tips in early career and beyond’, PLoS Biology, 17(5), p. e3000246. doi: 10.1371/journal.pbio.3000246.
Aschwanden, C. and Nguyen, M. (2018) How Shoddy Statistics Found A Home In Sports Research | FiveThirtyEight, FiveThirtyEight. Available at: https://fivethirtyeight.com/features/how-shoddy-statistics-found-a-home-in-sports-research/ (Accessed: 24 May 2022).
Atkinson, G., Batterham, A. and Drust, B. (2008) ‘Is it time for sports performance researchers to adopt a clinical-type research framework?’, International Journal of Sports Medicine, 29(9), pp. 703–705. doi: 10.1055/S-2008-1038545/ID/15.
Baker, M. and Penny, D. (2016) ‘Is there a reproducibility crisis?’, Nature, 533(7604), pp. 452–454. doi: 10.1038/533452A.
Bernards, J., Sato, K., Haff, G. and Bazyler, C. (2017) ‘Current research and statistical practices in sport science and a need for change’, Sports, 5(87). doi: 10.3390/sports5040087.
Blincoe, S. and Buchert, S. (2020) ‘Research preregistration as a teaching and learning tool in undergraduate psychology courses’, Psychology Learning and Teaching, 19(1), pp. 107–115. doi: 10.1177/1475725719875844.
Borg, D. N., Bon, J. J., Sainani, K. L., Baguley, B. J., Tierney, N. J. and Drovandi, C. (2020) ‘Comment on: “Moving sport and exercise science forward: a call for the adoption of more transparent research practices”’, Sports Medicine, 50(8), pp. 1551–1553. doi: 10.1007/s40279-020-01298-5.
Borg, D. N., Lohse, K. R. and Sainani, K. L. (2020) ‘Ten common statistical errors from all phases of research, and their fixes’, PM&R, 12(6), pp. 610–614. doi: 10.1002/pmrj.12395.
Boulbes, D. R., Costello, T., Baggerly, K., Fan, F., Wang, R., Bhattacharya, R., Ye, X. and Ellis, L. M. (2018) ‘A survey on data reproducibility and the effect of publication process on the ethical reporting of laboratory research’, Clinical Cancer Research, 24(14). doi: 10.1158/1078-0432.CCR-18-0227.
Braun, V. and Clarke, V. (2019) ‘Reflecting on reflexive thematic analysis’, Qualitative Research in Sport, Exercise and Health, 8 August, pp. 589–597. doi: 10.1080/2159676X.2019.1628806.
Buttliere, B. and Wicherts, J. (2018) ‘Opinions on the value of direct replication: A survey of 2,000 psychologists’. doi: 10.31234/osf.io/z9kx6.
Büttner, F., Toomey, E., McClean, S., Roe, M. and Delahunt, E. (2020) ‘Are questionable research practices facilitating new discoveries in sport and exercise medicine? The proportion of supported hypotheses is implausibly high’, British Journal of Sports Medicine, pp. 1–7. doi: 10.1136/bjsports-2019-101863.
Camerer, C. F., Dreber, A., Forsell, E., Ho, T. H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M. and Wu, H. (2016) ‘Evaluating replicability of laboratory experiments in economics’, Science, 351(6280), pp. 1433–1436. doi: 10.1126/science.aaf0918.
Chalmers, S., Debenedictis, T. A., Zacharia, A., Townsley, S., Gleeson, C., Lynagh, M., Townsley, A. and Fuller, J. T. (2018) ‘Asymmetry during Functional Movement Screening and injury risk in junior football players: A replication study’, Scandinavian Journal of Medicine and Science in Sports, 28(3), pp. 1281–1287. doi: 10.1111/sms.13021.
Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D. and Etchells, P. J. (2014) ‘Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond’, AIMS Neuroscience, 1(1), pp. 4–17. doi: 10.3934/Neuroscience.2014.1.4.
Derksen, M. and Field, S. (2022) ‘The tone debate: Knowledge, self, and social order’, Review of General Psychology, 26(2), pp. 172–183. doi: 10.1177/10892680211015636.
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E. and Nosek, B. A. (2021) ‘Investigating the replicability of preclinical cancer biology.’, eLife, 10, pp. 1–30. doi: 10.7554/eLife.71601.
Fanelli, D., Costas, R. and Ioannidis, J. P. A. (2017) ‘Meta-assessment of bias in science’, Proceedings of the National Academy of Sciences of the United States of America, 114(14), pp. 3714–3719. doi: 10.1073/pnas.1618569114.
Gabelica, M., Bojčić, R. and Puljak, L. (2022) ‘Many researchers were not compliant with their published data sharing statement: a mixed-methods study’, Journal of Clinical Epidemiology, 150, pp. 33–41. doi: 10.1016/j.jclinepi.2022.05.019.
Guttinger, S. (2020) ‘The limits of replicability’, European Journal for Philosophy of Science, 10(2), pp. 1–17. doi: 10.1007/s13194-019-0269-1.
Halperin, I., Pyne, D. B. and Martin, D. T. (2015) ‘Threats to internal validity in exercise science: A review of overlooked confounding variables’, International Journal of Sports Physiology and Performance, 10(7), pp. 823–829. doi: 10.1123/IJSPP.2014-0566.
Hansford, H. J., Cashin, A. G., Wewege, M. A., Ferraro, M. C., McAuley, J. H. and Jones, M. D. (2022) ‘Open and transparent sports science research: the role of journals to move the field forward’, Knee Surgery, Sports Traumatology, Arthroscopy, pp. 1–3. doi: 10.1007/s00167-022-06893-9.
Heathers, J. A. J. and Grimes, D. R. (2022) ‘The Mechanics Behind A Precipitious Rise in Impact Factor: A Case Study From the British Journal of Sports Medicine’. doi: 10.17605/OSF.IO/4C6XA.
Heneghan, C., Perera, R., Nunan, D., Mahtani, K. and Gill, P. (2012) ‘Forty years of sports performance research and little insight gained.’, BMJ, 345. doi: 10.1136/bmj.e4797.
Impellizzeri, F. M., McCall, A. and Meyer, T. (2019) ‘Registered reports coming soon: our contribution to better science in football research’, Science and Medicine in Football, 3(2), pp. 87–88. doi: 10.1080/24733938.2019.1603659.
Janz, N. and Freese, J. (2020) ‘Replicate others as you would like to be replicated yourself’, PS: Political Science and Politics, 54(2), pp. 1–4. doi: 10.1017/S1049096520000943.
Jekel, M., Fiedler, S., Allstadt Torras, R., Mischkowski, D., Dorrough, A. R. and Glöckner, A. (2020) ‘How to teach open science principles in the undergraduate curriculum—The Hagen cumulative science project’, Psychology Learning and Teaching, 19(1), pp. 91–106. doi: 10.1177/1475725719868149.
John, L. K., Loewenstein, G. and Prelec, D. (2012) ‘Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling’, Psychological Science, 23(5), pp. 524–532. doi: 10.1177/0956797611430953.
Kathawalla, U.-K., Silverstein, P. and Syed, M. (2020) ‘Easing into open science: A guide for graduate students and their advisors’, Collabra: Psychology, 7(1), pp. 1–38. doi: 10.1525/collabra.18684.
Knudson, D. (2005) ‘Statistical and reporting errors in applied biomechanics research’, in International Society of Biomechanics in Sport Conference. Beijing, China, pp. 811–814.
Knudson, D. (2017) ‘Confidence crisis of results in biomechanics research’, Sports Biomechanics, 16(4), pp. 425–433. doi: 10.1080/14763141.2016.1246603.
Makel, M. C. and Plucker, J. A. (2014) ‘Facts are more important than Nnvelty: Replication in the education sciences’, Educational Researcher, 43(6), pp. 304–316. doi: 10.3102/0013189X14545513.
Maxwell, S. E., Lau, M. Y. and Howard, G. S. (2015) ‘Is psychology suffering from a replication crisis?: What does “failure to replicate” really mean?’, American Psychologist, 70(6), pp. 487–498. doi: 10.1037/a0039400.
Mellor, D. T. (2021) ‘Improving norms in research culture to incentivize transparency and rigor’, Educational Psychologist, 56(2), pp. 122–131. doi: 10.1080/00461520.2021.1902329.
Mesquida, C., Murphy, J., Lakens, D. and Warne, J. (2022) ‘Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field’, Royal Society Open Science, 9(220946). doi: https://doi.org/10.1098/rsos.220946.
Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., Lewandowsky, S., Morey, C. C., Newman, D. P., Schönbrodt, F. D., Vanpaemel, W., Wagenmakers, E. J. and Zwaan, R. A. (2015) ‘The Peer Reviewers’ Openness Initiative: incentivizing open research practices through peer review’, Royal Society Open Science, 3(1). doi: 10.1098/RSOS.150547.
Morin, J. B., Samozino, P., Murata, M., Cross, M. R. and Nagahara, R. (2019) ‘A simple method for computing sprint acceleration kinetics from running velocity data: Replication study with improved design’, Journal of Biomechanics, 94, pp. 82–87. doi: 10.1016/j.jbiomech.2019.07.020.
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie Du Sert, N., Simonsohn, U., Wagenmakers, E. J., Ware, J. J. and Ioannidis, J. P. A. (2017) ‘A manifesto for reproducible science’, Nature Human Behaviour. Nature Publishing Group. doi: 10.1038/s41562-016-0021.
Murphy, J., Mesquida, C., Caldwell, A. R., Earp, B. D. and Warne, J. (2022) Selection Protocol for Replication in Sports and Exercise Science. Available at: https://osf.io/v3wz4/.
Nielsen, R. O., Chapman, C. M., Louis, W. R., Stovitz, S. D., Mansournia, M. A., Windt, J., Møller, M., Parner, E. T., Hulme, A., Bertelsen, M. L., Finch, C. F., Casals, M. and Verhagen, E. (2018) ‘Seven sins when interpreting statistics in sports injury science’, British Journal of Sports Medicine, 52(22), pp. 1410–1412. doi: 10.1136/bjsports-2017-098524.
Nosek, B. A. and Errington, T. M. (2020) ‘What is replication?’, PLoS Biology, 18(3), pp. 1–8. doi: 10.1371/journal.pbio.3000691.
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. grave le B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D. and Vazire, S. (2022) ‘Replicability, robustness, and reproducibility in psychological science’, Annual Review of Psychology, pp. 719–748. doi: 10.1146/annurev-psych-020821-114157.
Nosek, B. A., Spies, J. R. and Motyl, M. (2012) ‘Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability’, Perspectives on Psychological Science, 7(6), pp. 615–631. doi: 10.1177/1745691612459058.
Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S. and Wicherts, J. M. (2016) ‘The prevalence of statistical reporting errors in psychology (1985–2013)’, Behavior Research Methods, 48(4), pp. 1205–1226. doi: 10.3758/s13428-015-0664-2.
Open Science Collaboration (2015) ‘Estimating the reproducibility of psychological science’, Science, 349(6251). doi: 10.1126/science.aac4716.
Ostaszewski, M. (2014) ‘Analysis of the attitude within academic and research communities towards open science – a quantitative survey’, pp. 1–16. doi: depot.ceon.pl/handle/123456789/3719.
Pashler, H. and Wagenmakers, E. J. (2012) ‘Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?’, Perspectives on Psychological Science, 7(6), pp. 528–530. doi: 10.1177/1745691612465253.
Pitsch, W. and Emrich, E. (2012) ‘The frequency of doping in elite sport: Results of a replication study’, International Review for the Sociology of Sport, 47(5), pp. 559–580. doi: 10.1177/1012690211413969.
Piwowar, H. A. and Vision, T. J. (2013) ‘Data reuse and the open data citation advantage’, PeerJ, 2013(1), pp. 1–25. doi: 10.7717/peerj.175.
Powell, K. (2015) ‘The future of the postdoc’, Nature, pp. 144–147. doi: 10.4324/9781315674827-14.
Prieto, D. (2017) ‘Make databases language-proof’, Nature, 550(332), p. 29. doi: 10.1038/d41586-018-05847-x.
R Core Team (2022) ‘A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/’. Vienna, Austria: R Foundation for Statistical Computing.
Rosenthal, R. (1979) ‘The “file drawer problem” and tolerance for null results’, Psychological Bulletin, 86(3), pp. 638–641.
Sainani, K. and Chamari, K. (2022) ‘Wish list for improving the quality of statistics in sport science’, International Journal of Sports Physiology and Performance, 17(5), pp. 673–674. doi: 10.1123/ijspp.2020-0934.
Sainani, K. L., Borg, D. N., Caldwell, A. R., Butson, M. L., Tenan, M. S., Vickers, A. J., Vigotsky, A. D., Warmenhoven, J., Nguyen, R., Lohse, K. R., Knight, E. J. and Bargary, N. (2021) ‘Call to increase statistical collaboration in sports science, sport and exercise medicine and sports physiotherapy’, British Journal of Sports Medicine, 55(2), pp. 118–122. doi: 10.1136/bjsports-2020-102607.
Sainani, K. L., Lohse, K. R., Jones, P. R. and Vickers, A. (2019) ‘Magnitude-based Inference is not Bayesian and is not a valid method of inference’, Scandinavian Journal of Medicine and Science in Sports, 29(9), pp. 1428–1436. doi: 10.1111/sms.13491.
Scheel, A. M., Tiokhin, L., Isager, P. M. and Lakens, D. (2020) ‘Why hypothesis testers should spend less time testing hypotheses’, Perspectives on Psychological Science, 16(4), pp. 744–755. doi: 10.1177/1745691620966795.
Schmidt, S. (2009) ‘Shall we really do it again? The powerful concept of replication is neglected in the social sciences’, Review of General Psychology, 13(2), pp. 90–100. doi: 10.1037/a0015108.
Smaldino, P. E. and McElreath, R. (2016) ‘The natural selection of bad science’, Royal Society Open Science, 3(9). doi: 10.1098/rsos.160384.
Smith, L. M., Yu, F. and Schmid, K. K. (2021) ‘Role of replication research in biostatistics graduate education’, 29(1), pp. 95–104. doi: 10.1080/10691898.2020.1844105.
Stojmenovska, D., Bol, T. and Leopold, T. (2019) ‘Teaching Replication to Graduate Students’, Teaching Sociology, 47(4), pp. 303–313. doi: 10.1177/0092055X19867996/ASSET/IMAGES/LARGE/10.1177_0092055X19867996-FIG1.JPEG.
Stroebe, W. and Strack, F. (2014) ‘The Alleged Crisis and the Illusion of Exact Replication’, Perspectives on Psychological Science, 9(1), pp. 59–71. doi: 10.1177/1745691613514450.
Turner, L., Shamseer, L., Altman, D. G., Schulz, K. F. and Moher, D. (2012) ‘Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane reviewa’, Systematic Reviews, 1(1), pp. 1–7. doi: 10.1186/2046-4053-1-60.
Twomey, R., Yingling, V. R., Warne, J. P., Scheider, C., McCrum, C., Atkins, W. C., Murphy, J., Medina, C. R., Harlley, S. and Caldwell, A. R. (2021) ‘The nature of our literature: A registered report on the positive result rate and reporting practices in kinesiology’, Communications in Kinesiology, 1(3), pp. 1–14. doi: 10.51224/cik.v1i3.43.
Vines, T. H., Albert, A. Y. K., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., Gilbert, K. J., Moore, J. S., Renaut, S. and Rennison, D. J. (2014) ‘The availability of research data declines rapidly with article age’, Current Biology, 24(1), pp. 94–97. doi: 10.1016/j.cub.2013.11.014.
Vinkers, C. H., Tijdink, J. K. and Otte, W. M. (2015) ‘Use of positive and negative words in scientific PubMed abstracts between 1974 and 2014: retrospective analysis’, BMJ, 351. doi: 10.1136/BMJ.H6467.
de Vrieze, J. (2021) ‘Large survey finds questionable research practices are common Dutch study finds 8% of scientists have committed fraud’, Science, 373(6552), p. 265. doi: 10.1126/science.373.6552.265.
Wagge, J. R., Brandt, M. J., Lazarevic, L. B., Legate, N., Christopherson, C., Wiggins, B. and Grahe, J. E. (2019) ‘Publishing research with undergraduate students via replication work: The collaborative replications and education project’, Frontiers in Psychology, 10(FEB), pp. 1–4. doi: 10.3389/fpsyg.2019.00247.
Klein, R. A. et al. (2014) ‘Investigating variation in replicability: A “many labs” replication project’, Social Psychology, 45(3), pp. 142–152. doi: 10.1027/1864-9335/a000178.
Nosek, B. A. et al. (2015) ‘Promoting an open research culture’, Science, 348(6242), pp. 1422–1425. doi: 10.1126/science.aab2374.
Camerer, C. F. et al. (2018) ‘Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015’, Nature Human Behaviour, 2(9), pp. 637–644. doi: 10.1038/s41562-018-0399-z.
Klein, R. A. et al. (2018) ‘Many Labs 2: Investigating Variation in Replicability Across Samples and Settings’, Advances in Methods and Practices in Psychological Science, 1(4), pp. 443–490. doi: 10.1177/2515245918810225.
Caldwell, A. R. et al. (2020) ‘Moving sport and exercise science forward: A call for the adoption of more transparent research practices’, Sports Medicine, 50(3), pp. 449–459. doi: 10.1007/s40279-019-01227-1.
Kent, B. A. et al. (2022) ‘Recommendations for empowering early career researchers to improve research culture and practice’, PLOS Biology, 20(7), p. e3001680. doi: 10.1371/journal.pbio.3001680.
Pownall, M. et al. (2022) ‘The impact of open and reproducible scholarship on students’ scientific literacy, engagement, and attitudes towards science: A review and synthesis of the evidence’, MetaArXiv Preprints, pp. 1–56. doi: 10.31222/OSF.IO/9E526.
Downloads
Posted
License
Copyright (c) 2022 Jennifer Murphy, Cristian Mesquida, Joe P. Warne
This work is licensed under a Creative Commons Attribution 4.0 International License.
