The perils of misinterpreting and misusing “publication bias” in meta-analyses
An education review
DOI:
https://doi.org/10.51224/SRXIV.296Keywords:
publication bias, reporting bias, meta-research, meta-analysis, sports sciencesAbstract
Publication bias refers to a systematic deviation from the truth in the results of a meta-analysis due to the higher likelihood for published studies to be included in meta-analyses than unpublished studies. Publication bias can lead to misleading recommendations for decision- and policy-making. In this education review, we introduce, explain, and provide solutions to the pervasive misuses and misinterpretations of publication bias that afflict evidence syntheses in sport and exercise medicine. Publication bias is more routinely assessed by visually inspecting funnel plot asymmetry, although it has been consistently deemed unreliable, leading to the development of statistical tests to assess publication bias. However, statistical tests of publication bias (i) cannot rule out alternative explanations for funnel plot asymmetry (e.g., between-study heterogeneity, choice of metric, chance), and (ii) are grossly underpowered, even when using an arbitrary minimum threshold of ≥10 studies. We performed a cross-sectional, meta-research investigation of how publication bias was assessed in systematic reviews with meta-analysis published in the top two sport and exercise medicine journals throughout 2021. This analysis highlights that publication bias is frequently misused and misinterpreted, even in top tier journals. Due to conceptual and methodological problems when assessing and interpreting publication bias, preventive strategies (e.g., pre-registration, disclosing protocol deviations, and reporting all study findings regardless of direction or magnitude) offer the best and most efficient solution to mitigate the misuse and misinterpretation of publication bias. Because true publication bias is very difficult to determine, we recommend that future publications use the term “risk of publication bias”.
Metrics
References
Tang JL, Liu JL. Misleading funnel plot for detection of bias in meta-analysis. J Clin Epidemiol. 2000;53(5):477-84. doi: https://doi.org/10.1016/s0895-4356(99)00204-8.
Higgins JP, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions (2nd Ed.). Chichester (UK): John Wiley & Sons; 2019.
Copas J, Shi JQ. Meta-analysis, funnel plots and sensitivity analysis. Biostatistics. 2000;1(3):247-62. doi: https://doi.org/10.1093/biostatistics/1.3.247.
Sinha MK, Montori VM. Reporting bias and other biases affecting systematic reviews and meta-analyses: a methodological commentary. Expert Review of Pharmacoeconomics & Outcomes Research. 2006;6(5):603-11. doi: https://doi.org/10.1586/14737167.6.5.603.
Godavitarne C, Robertson A, Ricketts DM, Rogers BA. Understanding and interpreting funnel plots for the clinician. Br J Hosp Med. 2018;79(10):578-83. doi: https://doi.org/10.12968/hmed.2018.79.10.578.
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi: https://doi.org/10.1136/bmj.n71.
Tod D. Conducting systematic reviews in sport, exercise, and physical activity. Palgrave Macmillan, Springer Nature Switzerland AG; 2019.
Zwetsloot PP, Van Der Naald M, Sena ES, Howells DW, IntHout J, De Groot JA, et al. Standardized mean differences cause funnel plot distortion in publication bias assessments. Elife. 2017;6:e24260. doi: https://doi.org/10.7554/eLife.24260.
Biljana M, Jelena M, Branislav J, Milorad R. Bias in meta-analysis and funnel plot asymmetry. Stud Health Technol Inform. 1999;68:323-8.
Sterne JA, Egger M. Funnel plots for detecting bias in meta-analysis: guidelines on choice of axis. J Clin Epidemiol. 2001;54(10):1046-55. doi: https://doi.org/10.1016/s0895-4356(01)00377-8.
Dwan K, Gamble C, Kolamunnage-Dona R, Mohammed S, Powell C, Williamson PR. Assessing the potential for outcome reporting bias in a review: a tutorial. Trials. 2010;11(1):52. doi: https://doi.org/10.1186/1745-6215-11-52.
Liu JL. The role of the funnel plot in detecting publication and related biases in meta-analysis. Evid Based Dent. 2011;12(4):121-2. doi: https://doi.org/10.1038/sj.ebd.6400831.
Howland RH. What you see depends on where you're looking and how you look at it: publication bias and outcome reporting bias. J Psychosoc Nurs Ment Health Serv. 2011;49(8):13-5. doi: https://doi.org/10.3928/02793695-20110705-06.
Chiocchia V, Nikolakopoulou A, Higgins JPT, Page MJ, Papakonstantinou T, Cipriani A, et al. ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis. BMC Med. 2021;19(1):Article 304. doi: https://doi.org/10.1186/s12916-021-02166-3.
Dal-Ré R, Caplan AL. Journal editors impasse with outcome reporting bias. European Journal of Clinical Investigation. 2015;45(9):895-8. doi: https://doi.org/10.1111/eci.12484.
Schmid CH. Outcome Reporting Bias: A Pervasive Problem in Published Meta-analyses. Am J Kidney Dis. 2017;69(2):172-4. doi: https://doi.org/10.1053/j.ajkd.2016.11.003.
Page MJ, Sterne JAC, Higgins JPT, Egger M. Investigating and dealing with publication bias and other reporting biases in meta-analyses of health research: A review. Res Synth Methods. 2021;12(2):248-59. doi: https://doi.org/10.1002/jrsm.1468.
Pustejovsky JE, Rodgers MA. Testing for funnel plot asymmetry of standardized mean differences. Res Synth Methods. 2019;10(1):57-71. doi: https://doi.org/10.1002/jrsm.1332.
Ioannidis JP, Caplan AL, Dal-Ré R. Outcome reporting bias in clinical trials: why monitoring matters. BMJ. 2017;356:j408. doi: https://doi.org/10.1136/bmj.j408.
Kirkham JJ, Altman DG, Chan A-W, Gamble C, Dwan KM, Williamson PR. Outcome reporting bias in trials: a methodological approach for assessment and adjustment in systematic reviews. BMJ. 2018;362:k3802. doi: https://doi.org/10.1136/bmj.k3802.
Pirracchio R, Resche-Rigon M, Chevret S, Journois D. Do simple screening statistical tools help to detect reporting bias? Annals of Intensive Care. 2013;3(1):29. doi: https://doi.org/10.1186/2110-5820-3-29.
Stanley TD, Doucouliagos H. Meta-regression approximations to reduce publication selection bias. Res Synth Methods. 2014;5(1):60-78. doi: https://doi.org/10.1002/jrsm.1095.
Stanley TD. Limitations of PET-PEESE and Other Meta-Analysis Methods. Social Psychological and Personality Science. 2017;8(5):581-91. doi: https://doi.org/10.1177/1948550617693062.
Sedgwick P, Marston L. How to read a funnel plot in a meta-analysis. BMJ. 2015;351:h4718. doi: https://doi.org/10.1136/bmj.h4718.
Sedgwick P. Meta-analysis: testing for reporting bias. BMJ. 2015;350:g7857. doi: https://doi.org/10.1136/bmj.g7857.
Spineli LM, Pandis N. Reporting bias: Notion, many faces and implications. Am J Orthod Dentofacial Orthop. 2021;159(1):136-8. doi: https://doi.org/10.1016/j.ajodo.2020.10.010.
Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ. 2011;343:d4002. doi: https://doi.org/10.1136/bmj.d4002.
McGauran N, Wieseler B, Kreis J, Schüler Y-B, Kölsch H, Kaiser T. Reporting bias in medical research - a narrative review. Trials. 2010;11(1):37. doi: https://doi.org/10.1186/1745-6215-11-37.
Egger M, Smith GD. Bias in location and selection of studies. BMJ. 1998;316(7124):61-6. doi: https://doi.org/10.1136/bmj.316.7124.61.
Sterne JA, Gavaghan D, Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. J Clin Epidemiol. 2000;53(11):1119-29. doi: https://doi.org/10.1016/s0895-4356(00)00242-0.
Macaskill P, Walter SD, Irwig L. A comparison of methods to detect publication bias in meta-analysis. Stat Med. 2001;20(4):641-54. doi: https://doi.org/10.1002/sim.698.
Terrin N, Schmid CH, Lau J. In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. J Clin Epidemiol. 2005;58(9):894-901. doi: https://doi.org/10.1016/j.jclinepi.2005.01.006.
Doleman B, Freeman SC, Lund JN, Williams JP, Sutton AJ. Funnel plots may show asymmetry in the absence of publication bias with continuous outcomes dependent on baseline risk: presentation of a new publication bias test. Res Synth Methods. 2020;11(4):522-34. doi: https://doi.org/10.1002/jrsm.1414.
Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol. 2011;64(12):1277-82. doi: https://doi.org/10.1016/j.jclinepi.2011.01.011.
Mavridis D, Salanti G. How to assess publication bias: funnel plot, trim-and-fill method and selection models. Evid Based Ment Health. 2014;17(1):30. doi: https://doi.org/10.1136/eb-2013-101699.
Hunter JP, Saratzis A, Sutton AJ, Boucher RH, Sayers RD, Bown MJ. In meta-analyses of proportion studies, funnel plots were found to be an inaccurate method of assessing publication bias. J Clin Epidemiol. 2014;67(8):897-903. doi: https://doi.org/10.1016/j.jclinepi.2014.03.003.
Rosenthal R. The file drawer problem and tolerance for null results. Psychological Bulletin. 1979;86(3):638-41. doi: https://doi.org/10.1037/0033-2909.86.3.638.
Bartoš F, Maier M, Quintana DS, Wagenmakers E. Adjusting for Publication Bias in JASP and R: Selection Models, PET-PEESE, and Robust Bayesian Meta-Analysis. Advances in Methods and Practices in Psychological Science. Ahead of Print. doi: https://doi.org/10.1177/25152459221109259.
Duval S, Tweedie R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics. 2000;56(2):455-63. doi: https://doi.org/10.1111/j.0006-341x.2000.00455.x.
Lin L. Graphical augmentations to sample-size-based funnel plot in meta-analysis. Res Synth Methods. 2019;10(3):376-88. doi: https://doi.org/10.1002/jrsm.1340.
Ioannidis JP, Munafò MR, Fusar-Poli P, Nosek BA, David SP. Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends Cogn Sci. 2014;18(5):235-41. doi: https://doi.org/10.1016/j.tics.2014.02.010.
Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315(7109):629-34. doi: https://doi.org/10.1136/bmj.315.7109.629.
van der Steen J, ter Riet G, van den Bogert C, Bouter L. Causes of reporting bias: a theoretical framework [version 2; peer review: 2 approved]. F1000Res. 2019;8(280). doi: https://doi.org/10.12688/f1000research.18310.2.
Abt G, Boreham C, Davison G, Jackson R, Nevill A, Wallace E, et al. Power, precision, and sample size estimation in sport and exercise science research. J Sports Sci. 2020;38(17):1933-5. doi: https://doi.org/10.1080/02640414.2020.1776002.
Debray TPA, Moons KGM, Riley RD. Detecting small-study effects and funnel plot asymmetry in meta-analysis of survival data: A comparison of new and existing tests. Res Synth Methods. 2018;9(1):41-50. doi: https://doi.org/10.1002/jrsm.1266.
Stuck AE, Rubenstein LZ, Wieland D. Bias in meta-analysis detected by a simple, graphical test. Asymmetry detected in funnel plot was probably due to true heterogeneity. BMJ. 1998;316(7129):469; author reply 70-1.
Vandenbroucke JP. Bias in meta-analysis detected by a simple, graphical test. Experts' views are still needed. BMJ. 1998;316(7129):469-70; author reply 70-1.
Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Publication bias. In: Borenstein LV, Hedges JPT, Higgins JP, Rothstein HR, editors. Introduction to Meta‐Analysis. John Wiley & Sons, Ltd; 2009. p. 277-92.
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. J Clin Epidemiol. 2008;61(10):991-6. doi: https://doi.org/10.1016/j.jclinepi.2007.11.010.
Simmonds M. Quantifying the risk of error when interpreting funnel plots. Syst Rev. 2015;4:24. doi: https://doi.org/10.1186/s13643-015-0004-8.
Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ. 2006;333(7568):597-600. doi: https://doi.org/10.1136/bmj.333.7568.597.
van der Willik EM, van Zwet EW, Hoekstra T, van Ittersum FJ, Hemmelder MH, Zoccali C, et al. Funnel plots of patient-reported outcomes to evaluate health-care quality: Basic principles, pitfalls and considerations. Nephrology (Carlton). 2021;26(2):95-104. doi: https://doi.org/10.1111/nep.13761.
Furuya-Kanamori L, Barendregt JJ, Doi SAR. A new improved graphical and quantitative method for detecting bias in meta-analysis. JBI Evidence Implementation. 2018;16(4):195-203. doi: https://doi.org/10.1097/XEB.0000000000000141.
Begg CB, Mazumdar M. Operating Characteristics of a Rank Correlation Test for Publication Bias. Biometrics. 1994;50(4):1088-101. doi: https://doi.org/10.2307/2533446.
Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. Jama. 2006;295(6):676-80. doi: https://doi.org/10.1001/jama.295.6.676.
Rücker G, Schwarzer G, Carpenter JR, Binder H, Schumacher M. Treatment-effect estimates adjusted for small-study effects via a limit meta-analysis. Biostatistics. 2011;12(1):122-42. doi: https://doi.org/10.1093/biostatistics/kxq046.
Simonsohn U, Nelson LD, Simmons JP. p-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results. Perspect Psychol Sci. 2014;9(6):666-81. doi: https://doi.org/10.1177/1745691614553988.
Simonsohn U, Nelson LD, Simmons JP. P-curve: a key to the file-drawer. J Exp Psychol Gen. 2014;143(2):534-47. doi: https://doi.org/10.1037/a0033242.
Simonsohn U, Simmons JP, Nelson LD. Better P-curves: Making P-curve analysis more robust to errors, fraud, and ambitious P-hacking, a Reply to Ulrich and Miller (2015). J Exp Psychol Gen. 2015;144(6):1146-52. doi: https://doi.org/10.1037/xge0000104.
van Aert RCM, Wicherts JM, van Assen MALM. Conducting Meta-Analyses Based on p Values: Reservations and Recommendations for Applying p-Uniform and p-Curve. Perspectives on Psychological Science. 2016;11(5):713-29. doi: https://doi.org/10.1177/1745691616650874.
van Assen MALM, van Aert RCM, Wicherts JM. Meta-analysis using effect size distributions of only statistically significant studies. Psychol Methods. 2015;20(3):293-309. doi: https://doi.org/10.1037/met0000025.
Vevea JL, Woods CM. Publication bias in research synthesis: sensitivity analysis using a priori weight functions. Psychol Methods. 2005;10(4):428-43. doi: https://doi.org/10.1037/1082-989x.10.4.428.
Page MJ, Bero L, Kroeger CM, Dai Z, McDonald S, Forbes A, et al. Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol. F1000Res. 2019;8:1760. doi: https://doi.org/10.12688/f1000research.20726.2.
Page MJ, McKenzie JE, Green SE, Forbes AB. An empirical investigation of the potential impact of selective inclusion of results in systematic reviews of interventions: study protocol. Syst Rev. 2013;2(1):21. doi: https://doi.org/10.1186/2046-4053-2-21.
Kaneda H. Asymmetry in funnel plot. Am J Cardiol. 2011;108(2):330-1. doi: https://doi.org/10.1016/j.amjcard.2011.04.004.
Büttner F, Toomey E, McClean S, Roe M, Delahunt E. Are questionable research practices facilitating new discoveries in sport and exercise medicine? The proportion of supported hypotheses is implausibly high. Br J Sports Med. 2020;54(22):1365. doi: https://doi.org/10.1136/bjsports-2019-101863.
Irwig L, Macaskill P, Berry G, Glasziou P. Bias in meta-analysis detected by a simple, graphical test. Graphical test is itself biased. BMJ. 1998;316(7129):470; author reply -1.
Mayo-Wilson E, Fusco N, Li T, Hong H, Canner JK, Dickersin K. Multiple outcomes and analyses in clinical trials create challenges for interpretation and research synthesis. J Clin Epidemiol. 2017;86:39-50. doi: https://doi.org/10.1016/j.jclinepi.2017.05.007.
Seagroatt V, Stratton I. Bias in meta-analysis detected by a simple, graphical test. Test had 10% false positive rate. BMJ. 1998;316(7129):470; author reply -1.
Kadlec D, Sainani KL, Nimphius S. With Great Power Comes Great Responsibility: Common Errors in Meta-Analyses and Meta-Regressions in Strength & Conditioning Research. Sports Med. 2022. doi: 10.1007/s40279-022-01766-0.
Ruiz JR, Ortega FB, Gutierrez A, Meusel D, Sjöström M, Castillo MJ. Health-related fitness assessment in childhood and adolescence: a European approach based on the AVENA, EYHS and HELENA studies. Journal of Public Health. 2006;14(5):269-77. doi: https://doi.org/10.1007/s10389-006-0059-z.
Ruiz JR, Castro-Piñero J, España-Romero V, Artero EG, Ortega FB, Cuenca MM, et al. Field-based fitness assessment in young people: the ALPHA health-related fitness test battery for children and adolescents. Br J Sports Med. 2011;45(6):518-24. doi: https://doi.org/10.1136/bjsm.2010.075341.
Wikstrom EA, Mueller C, Cain MS. Lack of Consensus on Return-to-Sport Criteria Following Lateral Ankle Sprain: A Systematic Review of Expert Opinions. J Sport Rehabil. 2020;29(2):231-7. doi: https://doi.org/10.1123/jsr.2019-0038.
Winters M, Weir A. Grey matters; on the importance of publication bias in systematic reviews. Br J Sports Med. 2017;51(6):488. doi: https://doi.org/10.1136/bjsports-2016-096679.
Rohatgi A. WebPlotDigitizer, version 4.5. Pacifica, California2021.
Downloads
Posted
Versions
- 2023-05-19 (2)
- 2023-05-19 (1)
License
Copyright (c) 2023 José Afonso, Rodrigo Ramirez-Campillo, Filipe Manuel Clemente, Fionn Cléirigh Büttner, Renato Andrade (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
