Source-linked AI summary
Deep Impact: Unintended consequences of journal rank
Björn Brembs, Marcus Munafò
TL;DR
The paper examines whether journal rank is a valid measure of scientific quality and whether the current scholarly communication system contributes to unreliable research. It reviews recent evidence across citations, methodological soundness, and retractions, finding weak utility prediction alongside stronger associations with unreliability. The authors therefore argue for replacing journal hierarchies with article-level scholarly communication.
Problem
The paper addresses limited empirical evidence about journal rank as a scientific impact and quality-assessment tool, amid concern about research unreliability.
Method
The paper reviews recent empirical data on journal rank and multiple measures of scientific quality, including citations, methodology, and retractions.
Results
Journal rank weakly predicts utility and perceived importance but moderately to strongly predicts intentional and unintentional scientific unreliability.
Takeaways & Limitations
The authors argue that journal rank should be abandoned in favor of a library-based system using technology for article-level selection, filtering, and ranking.
Takeaways & Limitations
The evidence includes concerns about Impact Factor validity and confounding factors, so the authors emphasize the need for future research.
Abstract
from arXiv · showhide
Most researchers acknowledge an intrinsic hierarchy in the scholarly journals ('journal rank') that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has been written about the negative effects of institutionalizing journal rank as an impact measure. So far, contributions to the debate concerning the limitations of journal rank as a scientific impact assessment tool have either lacked data, or relied on only a few studies. In this review, we present the most recent and pertinent data on the consequences of our current scholarly communication system with respect to various measures of scientific quality (such as utility/citations, methodological soundness, expert ratings or retractions). These data corroborate previous hypotheses: using journal rank as an assessment tool is bad scientific practice. Moreover, the data lead us to argue that any journal rank (not only the currently-favored Impact Factor) would have this negative impact. Therefore, we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary. This new system will use modern information technology to vastly improve the filter, sort and discovery functions of the current journal system.
Running Head: Consequences of Journal Rank
The passage identifies the paper’s authors: Björn Brembs, Katherine Button, and Marcus Munafò.
- Björn Brembs is listed as an author.
- Katherine Button is listed as an author.
- Marcus Munafò is listed as an author.
1. Institute of Zoology – Neurogenetics, University of Regensburg,
Science supports modern society and requires public trust, but scientific findings can be unreliable. The publication system also uses journal rank to evaluate researchers, making reliability and journal hierarchy connected concerns.
- Introduction: Scientific reliability matters because society increasingly depends on science and scientific research requires public trust.
- Introduction: Scientific publications can be unreliable because of honest error, statistical variability, misconduct, or fraud.
- Introduction: Peer review is one mechanism intended to increase the reliability of the scientific literature.
- Introduction: The publication system evaluates researchers partly by whether they publish in high-ranking journals.
- Introduction: Journal rank structures scientific communication and influences the composition of the scientific community.
Retractions and the Decline Effect
Retractions and unreliable findings have drawn increasing concern, while publication and career pressures may favor surprising results and prestigious venues. The paper asks whether this infrastructure contributes to scientific unreliability.
- Retractions and the Decline Effect: 0.001% to about 0.02%: the article retraction rate rose rapidly in the early 2000s after remaining stable since the 1970s.The passage describes this increase as a disturbing trend and notes that it gained wide public attention.
- Retractions and the Decline Effect: Retraction notices capture only the extreme end of a broader spectrum of scientific unreliability.The passage emphasizes that uncertainty is inherent to scientific work and that retractions are comparatively rare.
- Retractions and the Decline Effect: Researcher choices can increase false-positive risk, while publication bias favors surprising and novel effects over null results.
- Retractions and the Decline Effect: Null-result publication is uneven across disciplines and often receives less attention, contributing to a low frequency of replication studies.
- Retractions and the Decline Effect: The paper asks whether pressure to publish in prestigious journals contributes to scientific unreliability.
The Decline Effect and Journal Rank
The reviewed evidence links journal rank with inflated effects, fraud, retractions, and weak methodological quality, while its association with citations remains limited. These findings motivate scrutiny of journal rank as a quality signal.
- The Decline Effect and Journal Rank: Early studies in high-ranking journals reported larger effects, while later studies in lower-ranked journals often failed to replicate them or found weaker effects.
- The Decline Effect and Journal Rank: Higher-ranking journals are more likely than lower-ranking journals to publish fraudulent work among retracted publications.
- The Decline Effect and Journal Rank: Higher journal Impact Factor is associated with greater overestimation of likely true effect sizes in genetic association studies.
- The Decline Effect and Journal Rank: N=650; rs=-0.01; t=0.8: statistical power showed no significant correlation with journal rank.
- The Decline Effect and Journal Rank: Only about one out of five preclinical findings may be reproducible according to three replication studies.
- The Decline Effect and Journal Rank: Journal rank is also reported as a strong predictor of retraction rate, although the correlation may involve additional factors.The coefficient of determination between journal rank and retraction rate was 0.77, which the authors say is too strong to be explained solely by decreased reliability.
Social pressure and journal rank
Evidence links high journal rank with less reliable findings, while social pressure to publish positive results may contribute to misconduct and publication bias. Citation-based visibility may also contribute to error detection and retractions.
- High-ranking journals are more likely to publish fraudulent and less reliable findings than lower-ranking journals.The evidence includes inflated effects and findings that cannot subsequently be replicated.
- Pressure to publish in high-ranking journals may decrease ethical standards and increase publication bias in competitive fields.The paper connects broader career precariousness and competitiveness with publication bias across science.
- The correlation between journal rank and retraction rate cannot be explained exclusively by decreased reliability of high-ranking publications.The reported coefficient of determination is 0.77, suggesting additional contributing factors.
- Citation rates between journals can test whether greater visibility contributes to both citations and retractions.This approach tests the assumption that visibility contributing to retractions should also contribute to citations.
Journal Rank and Study Impact
Journal rank has only a weak association with future citations, and much of that association appears linked to visibility and reading habits rather than article-intrinsic importance. Citation evidence therefore provides little support for treating rank as a reliable measure of scientific impact or quality.
- High-ranking publications may receive citations through greater exposure, perceived value, and attention to novel or controversial findings.These factors supplement citations attributable to importance or novelty.
- In a sample of almost 30 million publications, the coefficient of determination between IF-based rank and two-year citations changed over publication history.The analysis examined correlations as a function of publication year.
- Journal rank’s association with future citations is weak, with coefficients of determination generally around 0.1 to 0.3.The paper describes the correlation as small but significant and insufficient to explain the stronger rank–retraction relationship.
- After the 1960s, rank–citation predictive power increased, then fell toward pre-1960s levels after internet search engines became widespread.The paper interprets these trends as consistent with changes in the influence of journal visibility on reading habits.
- The only citation-count measure strongly correlated negatively with journal rank is the number of uncited articles.This supports the view that fewer articles in high-ranking journals go unread.
- Independent citation measures do not compellingly establish journal rank as a reliable predictor of scientific impact or quality.Other measures instead indicate that rank may be at least as predictive of low reliability.
Practical consequences of Journal Rank
The pursuit of high-ranking journals slows dissemination, increases reviewing burdens, and can distort how scientific progress is communicated. Reported citation gains from resubmission are practically negligible, while selective attention can favor later or higher-ranked reports.
- Cascading submissions and rejections through journal hierarchies slow dissemination and increase the burden on reviewers.These costs arise even when the underlying study was performed to high standards.
- Resubmissions yield about 0.1 additional citations per article, a statistically significant but practically negligible effect.An earlier study failed to find such a citation benefit.
- Peer-review costs exceed €2.2 billion annually, while journal-rank competition contributes to rising subscription prices and closed access.The paper presents these as additional costs associated with the current system.
- Attention to high-ranking journals may distort communication of scientific progress inside and outside the scientific community.The paper notes that quantitative evidence for one specific citation effect is unavailable, while related media distortion has been reported.
- Two largely identical rodent Default-Mode Network reports received 19 and 5 citations, respectively, with the later report cited more.The paper cautions that the studies may differ in quality and that this particular effect lacks quantitative empirical study.
Impact Factor – Negotiated, irreproducible and unsound
The Impact Factor normalizes journal citations by the number of counted articles, but its calculation can be negotiated, irreproducible, and mathematically unsound. Skewed citation distributions make its arithmetic-mean basis inappropriate.
- The Impact Factor equals citations to journal articles divided by the number of counted articles.The numerator includes citations, while the denominator depends on which published items are counted.
- Impact Factor denominators can be negotiated, producing large changes in the resulting metric.For PLoS Medicine, the reported negotiation range was from 2 to about 11.
- Current Biology’s 2002/2003 Impact Factor increase was attributed mainly to a reduction in published items, with discrepancies across consecutive JCR editions.The table caption highlights the denominator change and inconsistent counts for items published in 2001.
- Published and received citation data have shown discrepancies that can alter journal rankings by up to 19%.Rockefeller University Press found numerous discrepancies between supplied citation data and published rankings.
- Reported errors and inconsistencies further undermine confidence in Impact Factor calculations.
- Citation counts are strongly left-skewed, yet the Impact Factor uses an arithmetic mean rather than a median.The paper argues that this makes the calculation inappropriate for the distribution.
Conclusions
The reviewed evidence supports four conclusions: journal rank weakly to moderately predicts utility and perceived importance, but more strongly predicts scientific unreliability. It is also costly and delays research, while Impact Factor violates basic scientific standards despite predicting subjective journal-quality judgments.
- Journal rank is a weak to moderate predictor of utility and perceived importance, but a moderate to strong predictor of intentional and unintentional scientific unreliability.
- Journal rank is expensive, delays science, and frustrates researchers.
- Impact Factor violates basic scientific standards while predicting subjective judgments of journal quality.
- Confounding arises because authors submit strong studies to prestigious journals, readers expect important work there, and field-specific citation and methodological practices differ.
- Despite these confounds, evidence consistently links journal rank with both scientific impact and unreliability, so the conclusions remain difficult to dismiss.
Potential long-term consequences of journal rank
The authors argue that journal rank creates pressures and expectations that can increase unreliable research and undermine confidence in science. They therefore characterize journal rank as, at best, unhelpful and, at worst, unscientific.
- Using journal rank is described as unhelpful at best and unscientific at worst because Impact Factor’s prestige signal lacks empirical support for predicting subsequent impact.
- Journal rank may contribute to decline effects and rising retraction rates by increasing pressure on authors to make less cautious conclusions.
- The reviewed discussion connects high-ranking-journal publication with misconduct-related retractions and a decline in reported null results.
- The implications extend beyond publication reliability because declining scientific reliability threatens public trust in science and scientists.
Alternatives
The authors propose replacing journal hierarchy with an archival, library-based system that evaluates and organizes research at the article level. Open interoperability would enable dynamic filtering, ranking, discovery, and post-publication editorial selection.
- Current technology can perform journal rank’s selection, filtering, and ranking functions dynamically and without bias at the per-article level.
- The proposed archival system would preserve peer-reviewed articles, software, raw data, and descriptions while using scientifically tested metrics in an evolving reputation system.
- The envisioned reputation system would make trying to produce the best science the only evolutionarily stable strategy.
- Fragmentation, limited interoperability, and missing standards currently prevent article-level technologies from replacing journal rank.
- Professional editors could continue selecting publications after release, competing through track records for identifying important discoveries.
- Subscription funds could finance the initial conversion to the proposed scholarly communication system.