By: Chad E. Cook PT, PhD, FAPTA
We have problems: There are countless publications, editorials, and blogs indicating we have a notable problem with the peer review system used in scientific publications [1-4]. Concerns have included its inconsistency, its slow process, and the biases associated with reviewers (especially reviewer two) who have an axe to grind. These limitations and the knowledge that publishing companies are making record profit margins [5] off the free labor of reviewers, while authors are required to pay to publish, are especially difficult to stomach. This problem has been ongoing for some time but in my opinion, it seems to have worsened recently. Having been immersed in publishing for over 25 years as an author, and over 20 years as an editor-in-chief or associate editor for four journals, I’d like to outline my concerns that qualify my statement in the title that it’s “probably worse than you think”.
Journals are overwhelmed and subsequently, unresponsive: The last three publications I’ve submitted to peer reviewed journals took 11 months, 10 months, and 6 months, to receive the first set of reviewers’ comments. For those that are not familiar with peer-reviewed publishing, this is a very long time to wait for your first set of reviews. We pulled the paper that took 11 months over 6 months ago (because we were tired of the lack of responsiveness from the journal) and informed the editor-in-chief that we removed it from the review process, but they kept it within their system anyway, and eventually provided the reviews (11 months later). It had already been accepted in a different journal by then. We were informed by the editor-in-chief of the paper that took 6 months that they had reached out to 60 reviewers, to receive two reviewers’ comments. They eventually used the names of reviewers that we recommended. Two of the three examples were review articles and the editors had the audacity to recommend an updated search!
Quality has been sacrificed for quantity: It is estimated that there are 30,000 medical journals published around the world [6]. In 2016, about 1.92 million papers were indexed by the Scopus and Web of Science publication databases; In 2022, that number jumped to 2.82 million [7]. This equates to approximately two papers uploaded to PubMed every minute [8]. Subsequently, it is no secret that quantity has replaced quality. It is especially prevalent in open access journals in which revenue is dependent on an article processing charge (APC) and volume. On average, an article processing charge (APC) of $1,626 USD has been reported [9]. Whereas, this may not seem to be unreasonable, some journals charge over $11,000 USD (Nature Neuroscience [10]), whereas others (PLOS One [11]) have published over 30,000 papers in a given year. I think it is hard-pressed to assume that enough useful science is being created that demands 2.82 million unique papers.
Reviewers are overwhelmed and are abused: I feel it is my responsibility to review for journals, since I’m a user of the peer review system, and I do so without compensation. It generally takes me an hour to do a meaningful and respectful review; sometimes it takes me longer if I need to check the trial registration, review attached appendices, or some of the more important references. Although I serve as an associate editor for a journal, I try and limit my reviews to two manuscripts a week. Apparently, this isn’t enough. From March 1st through March 31st in 2024, I was asked to review 67 papers for scientific journals. That’s an average of nearly 2.2 requests per day-including non-business days. Interestingly, one journal in particular, in which I just published a paper (after 10 months of waiting for the first review), requested my review services 13 times. I averaged >four requests a week from this journal until I finally stopped responding. It is important to recognize that reviewers are overwhelmed and should be compensated for their work. Those who agree to review understand the sarcastic phrase “no good deed goes unpunished”.
Editors are Often Underpaid, Overworked, and Pressured to Publish: A 2020 survey found that more than one third of editors surveyed from core clinical journals did not receive compensation for their editorial roles [12]. As an editor-in-chief from 2006 through 2012, I contributed over 20 hours a week to the journal, and did receive a small stipend for my efforts. I calculated an average hourly salary of a little over three dollars. Further, previous work has exposed the pressure editors have to publish work [13], especially those who run open access journals, in which payment is required to publish within the journal. This leads to the acceptance of inferior work and a flooding of review requests for papers that should have likely been triaged by the editor.
Fake journals are numerous and are getting difficult to discriminate: Predatory journals are open-access publishers that actively solicit and publishes articles for a fee, with little or no real peer review [14]. I’ve written about these before and even wrote a fake paper (with Josh Cleland and Paul Mintken) about a dead person being brought back to life with spinal manipulation to show how these journals will accept anything [15]. There are some estimates that there are 15,000 predatory journals in existence [16]. A popular publishing company MDPI has recently been placed on Predatory-Reports.com’s predatory publishing list because of concerning behaviors in the peer-review process [17]. It is worth noting that many borderline predatory behaviors have made the distinction of predatory journals more difficult, as the competition to secure submissions has ramped up correspondingly with the number of new journals that have been created. Publishing low quality or questionable work has also undermined the promotion and tenure process in academic settings as appointment, promotion and tenure (APT) committee members are often asked to review portfolios of individuals outside of their professional field.
Retraction rates are on the rise. A retraction occurs when a previously published paper in an academic journal is flagged as seriously flawed to the extent that their results and/or conclusions are no longer valid. Retractions occur because of plagiarism, data manipulation and conflict of interest [18] and overall, they are not very common; for every 10,000 papers, 2.5 papers were retracted. Journals self-govern (with external assistance) and often identify flawed work and retract the papers. As such, most retractions occur in higher level journals. To date, data simply don’t exist that can provide an estimate of how many flawed papers are present in journals with no real peer review (predatory) and those that aren’t predatory but have questionable behaviors.
This sounds awful, what should we do: I do realize this blog is negative, but it’s important to understand the context around peer review, especially if you have not the opportunity to publish, review or edit in the peer review system. There are strategies that one can take on that may help navigate these challenges. First, I’d recommend that you read work from reputable journals that are affiliated with reputable societies (e.g., JOSPT, Physical Therapy, Journal of Physiotherapy, etc.). Second, I think it is healthy and reasonable to question results that are notably different from known information, results that were obtained from a group with a vested interest in the outcome of the study, and results that are substantially better than the comparison group, because that’s just not very common or likely. Third, it is appropriate to support the current inertia toward paying reviewers for their efforts as long as their work is of high quality. Fourth, it is good when editors triage papers that are unlikely to be published (or those that shouldn’t be published) as this reduces the burden on peer review. Lastly, it’s important to recognize that someone has to pay for open access journals; it is typically the author that pays.
References
Smith R. Peer review: a flawed process at the heart of science and journals. J R Soc Med. 2006 Apr;99(4):178-82.
Flaherty C. The Peer-Review Crisis. Inside Higher Ed. Available at: https://www.insidehighered.com/news/2022/06/13/peer-review-crisis-creates-problems-journals-and-scholars
Malcom D. It’s Time We Fix the Peer Review System. Am J Pharm Educ. 2018 Jun;82(5):7144.
Subbaraman N. What’s wrong with peer review. Wall Street Journal. Available at: https://www.wsj.com/science/whats-wrong-with-peer-review-e5d2d428
Ansede M. Scientists paid large publishers over $1 billion in four years to have their studies published with open access. El Pais. Available at: https://english.elpais.com/science-tech/2023-11-21/scientists-paid-large-publishers-over-1-billion-in-four-years-to-have-their-studies-published-with-open-access.html
Gower T. What Are Medical Journals? WebMD. Available at: https://www.webmd.com/a-to-z-guides/medical-journals
(no author) Scientists are publishing too many papers—and that’s bad for science. Science Advisor. Available at: https://www.science.org/content/article/scienceadviser-scientists-are-publishing-too-many-papers-and-s-bad-science#:~:text=In%20recent%20years%2C%20the%20number,had%20jumped%20to%202.82%20million.
Landhuis E. Scientific literature: Information overload. Nature. 2016;535:457–458.
Morrison H, (2021-06-24). “Open access article processing charges 2011 – 2021”. Sustaining the Knowledge Commons / Soutenir les savoirs communs. Retrieved 2022-02-18.
Du JS. Opinion: Is Open Access Worth the Cost? The Scientist. Available at: https://www.the-scientist.com/opinion-is-open-access-worth-the-cost-70049
Kayla Graham (January 6, 2014). “Thanking Our Peer Reviewers – EveryONEEveryONE”. Blogs.plos.org. Retrieved 2015-05-17.
Lee JCL, Watt J, Kelsall D, Straus S. Journal editors: How do their editing incomes compare? F1000Res. 2020;24;9:1027.
De Vrieze JOP. Open-access journal editors resign after alleged pressure to publish mediocre papers. Science Advisor. Available at: VRIEZEhttps://www.science.org/content/article/open-access-editors-resign-after-alleged-pressure-publish-mediocre-papers
Cook CE, Cleland JA, Mintken PE. Manual Therapy Cures Death: I Think I Read That Somewhere. J Orthop Sports Phys Ther. 2018 Nov;48(11):830-832.
Cook CE, Cleland J, Mintken P. Temporal Effect of Repeated Spinal Manipulation on Mortality Ratio: A Case Report. ARCH Women Health Care Volume. 2018. 1(1): 1–4.
Freeman E, Kurambayev B. Rising number of ‘predatory’ academic journals undermines research and public trust in scholarship. The Conversation. Available at: https://theconversation.com/rising-number-of-predatory-academic-journals-undermines-research-and-public-trust-in-scholarship-213107#:~:text=That%20is%20roughly%20the%20same,there%20were%2015%2C000%20predatory%20journals
(anonymous author) Is MDPI a predatory publisher? Publishing with Integrity. Available at: https://predatory-publishing.com/is-mdpi-a-predatory-publisher/
Conroy G. The biggest reason for biomedical research retractions. Detection software is not enough. Nature Index. Available at: https://www.nature.com/nature-index/news/the-biggest-reason-for-biomedical-retractions
