• All scholarly sources or peer-reviewed journals are not held in equally high regard.
• The most common measures of journal prestige, citation count and the related Impact Factor, have limitations and are not a direct measure of quality of the scholarship therein.

Notes for the instructor/librarian

This topic could fit well into instruction sessions that include significant treatment of source evaluation, and is one potential approach as you move beyond simple categorization of sources as scholarly/non-scholarly or primary/secondary. It stops short of a critical examination of construction of authority, but could be used to hint at greater subtlety and complexity. This topic has particular relevance for upper level undergraduates engaged in research, who may be starting to think about publication from an author’s perspective.

• Comparing journal prestige to university/college prestige: one way that reputations are built or maintained is through exclusivity and high rejection rate. Is this the same thing as quality?
• When showing students how to sort by times cited in a database: of course, most highly cited doesn’t mean best. A highly cited source might be controversial, or just published longer ago and had more time to get cited….
• There are incentives for journals to select manuscripts based on their likelihood to make a big splash and generate a lot of citations. This might actually result in publishing studies that are more likely to be overstated, flawed, or eventually retracted.
• Different numbers of researchers writing on a given topic and different citation practices in a given field can drive huge differences in citation rate. You can’t make comparisons across disciplines.

• How do you think your professor would evaluate a source in her area of expertise? Are you able to do that kind of evaluation with your current knowledge?
• What factors do you think would make an article likely to get a high number of citations?
• Are there reasons an article might get a high number of citations other than the quality of the research?
• If you only look at the most highly cited sources, what would you be likely to miss?

(See all content in Google Drive)

• Create a course-relevant visualization using one of the Scimago Journal Rank visualization tools. (SJR is based on Scopus data. Some details are available on their About page.) Alternatively, create a prompt for students to explore the visualizations themselves. As an example, you might use the Shape of Science visualization, setting the labels to Subject Areas, perhaps pruning to the top quartile, and zooming in to a given field of study. What questions can be answered, or not, based on these rankings and relationships?

• Use a satirical journal, such as Journal of Universal Rejection, Journal of Universal Acceptance, or Journal of Irreproducible Results, as fodder for discussion or activities. NPR Planet Money: “There exists a journal so prestigious — so elite and selective — that it has never published a single manuscript. It has rejected every submission it has every received.”
• Show example articles with different kinds of impact (citations, tweets, media coverage, etc.), on a topic related to the course or of broad interest. Lead a discussion of the possible value of these various kinds of impact, perhaps including the limitations of metrics and the difficulty of assessing impact that isn’t easily quantified. For example, here are some higher impact articles on climate change topics from PLOS ONE and PLOS Biology. PLOS is nice for this exercise because there is a Metrics tab on each article landing page with hyperlinked information.
  • The Role of Conspiracist Ideation and Worldviews in Predicting Rejection of Science
  • The Impact of Climate Change on Indigenous Arabica Coffee (Coffea arabica): Predicting Future Trends and Identifying Priorities
  • Climate Change and Range Expansion of the Asian Tiger Mosquito (Aedes albopictus) in Northeastern USA: Implications for Public Health Practitioners
  • Identifying the World’s Most Climate Change Vulnerable Species: A Systematic Trait-Based Assessment of all Birds, Amphibians and Corals
  • Climate-Related Local Extinctions Are Already Widespread among Plant and Animal Species

• Brembs, B. (2018). Prestigious Science Journals Struggle to Reach Even Average Reliability. Frontiers in Human Neuroscience, 12 (37). doi:10.3389/fnhum.2018.00037 This article could be shared with students or used for instructor/librarian background reading. It is particularly relevant for the sciences, but is only one example on the topic.
• Declaration on Research Assessment (DORA)
• This strong statement against improper use of journal impact factor (very widely endorsed by researchers, publishers, funders, and other groups) is probably only relevant to upper level undergraduates (and beyond) in the sciences, and if you’re able to devote significant time to metrics.