Preface

         I have grown to despise systematic reviews/meta-analyses. I do not like writing them, I do not enjoy reading them (any more than I enjoy reading an instructional manual), and I especially hate explaining to people why they have so many limitations. I feel lazy and uninspired when I participate in systematic reviews. I hate that journals clamber to publish them, even papers with notable flaws. I was an author on a systematic review that included zero papers [1]; which I would argue is one of my better reviews. It disappoints me when people on social media assume that a review is biblical and fawn upon findings that support their biases or interests.

I do not like them. My goal in this blog is to explain why I loathe them so much.

Chad E Cook PT, PhD, FAPTA

Twitter @chadcookpt

Professor, Department of Orthopaedics, Duke University, Durham, NC. 27516

Competing interests: A portion of Dr Cook’s salary is funded by the NIH/VA/DoD and the Center of Excellence in Manual and Manipulative Therapy at Duke University.

What makes a review, systematic?

         Unlike a narrative review or an opinion paper, a systematic review has specific elements that are designed to minimize bias. These features include [2]:

  1. Clear aims with predetermined eligibility and relevance criteria for studies;
  2. Transparent, reproducible methods;
  3. Rigorous search designed to locate all eligible studies;
  4. An assessment of the validity of the findings of the included studies;
  5. A systematic presentation, and synthesis, of the included studies.

Ideally, the inclusion of these five assumptions improves the quality of the final review. Unfortunately, that is not always the case. 

Limitations of systematic review

Systematic reviews/meta-analyses have a number of limitations, a few of which are outlined below.

  1. When you combine multiple poor quality papers, you end up with a poor quality summary on a systematic review. In 1978, Hans Eysenck (of dimension of personality fame) [3] described doing this as “an exercise in mega-silliness”. Rarely do research adjust their review based on both quantitative and qualitative parameters of the included studies. It is important to recognize that performing a review is both an art and a science. If a paper does not fit (regardless of what the metrics say), it should not be included in the summary.
  2. Instead of performing a critical review, some systematic reviews are mostly descriptive. Although this is somewhat useful for clinicians who simply do not have the time to surf through all the literature on a topic, it fails to give boundaries on the quality of the work performed and which aspects of it are transferable to practice. It presents all studies as if they are of similar merit.
  3. Most reviews have heterogeneity of included studies. Types of heterogeneity can include: 1) variability in the participants, interventions and outcomes (known as clinical diversity), 2) variability in study design and risk of bias (known as methodological diversity), or 3) variability in the intervention effects being evaluated (known as statistical heterogeneity). Because the composition of clinical studies can differ so markedly, it is important to recognize that heterogeneity is likely the norm, not the exception.
  4. Several methodological discussions are critical for the meta-analysis [4]. Did the authors calculate mean differences or standardized mean differences? When the measured outcome in studies uses different units or different scales, a standardized mean difference is the correct choice. Nonetheless, there are cases where mean difference is used incorrectly. Did the authors used fixed or random effects? Fixed effects are appropriate if we assume there is one true effect size across all studies. A random effects model recognizes that the true effect could vary from study to study, which is usually the case in allied health studies.
  5. Conducting a meta-analysis does not overcome problems that were inherent in the design and execution of the primary studies. When I first learned how to run REVMAN™, I felt it would improve the likelihood of uncovering the truth in the menagerie of studies in the review. I was wrong since the quality of the primary study will always trump the use of elegant statistics. Noted Stanford epidemiologist John Ioannidis has stated that there is a glut of similar, but different studies that should not be combined analytically [5].
  6. Interpretation is highly influenced by author biases. Cochrane does an exceptional job of identifying ways to measure risk of studies assessed in their handbook [6]. Missing, is how to assess the risk of bias of those who are interpreting the summary of results. Because most authors of reviews or meta-analyses have an interest in the topic studied, they may interpret the findings in a way that supports their positive or negative bias. This is likely why we sometimes see different conclusions in distinct reviews of the same topic.

There are many more limitations; we do not have the space here to cover them.

The pot calling the kettle black?

Indeed, I have been involved in a number of systematic reviews. According to PubMed, I have been an author on 41 systematic reviews. An additional three are indexed only in Cinahl; thus, 44 total. Therefore, you may ask; “Mr. Hypocrite, why should we believe you when you say you hate systematic reviews?” It is a valid point, one that I think I can address. In most cases, I have been involved in reviews led by either PhD or DPT students and systematic reviews are a customary approach for researchers who are mastering a given topic. The truth is I have not been the first author of a review since 2014. My involvement has in reviews has markedly declined since 2017. I am confident in saying that unless we get better primary studies to appraise, I will likely never lead another systematic review.

References

Tousignant-Laflamme Y, Christopher S, Clewley D, Ledbetter L, Cook CJ, Cook CE. Does shared decision making results in better health related outcomes for individuals with painful musculoskeletal disorders? A systematic review. J Man Manip Ther. 2017 Jul;25(3):144-150

Clarke J. What is a systematic review? Evidence-Based Nursing 2011;14:64

Eysenck HJ. An exercise in mega-silliness. American Psychologist.  1978;33(5):517

Takeshima, N., Sozu, T., Tajika, A. et al. Which is more generalizable, powerful and interpretable in meta-analyses, mean difference or standardized mean difference?. BMC Med Res Methodol 14, 30 (2014). https://doi.org/10.1186/1471-2288-14-30

Ioannidis JP. Why most published research findings are false. PLoS Med. 2005 Aug;2(8):e124

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook