Source-linked AI summary
The Journal Impact Factor: A brief history, critique, and discussion of adverse effects
Vincent Lariviere, Cassidy R. Sugimoto
TL;DR
The chapter examines how the widely used JIF shapes research evaluation despite technical limitations and adverse effects. Drawing on prior literature and original data, it reviews the indicator, its critiques, alternatives, and future use. It concludes that responsible interpretation is essential because the JIF is likely to remain part of the research ecosystem.
Problem
The JIF is pervasive in research evaluation, but its limitations and applications can distort assessment and contribute to adverse effects across the research system.
Method
The chapter combines existing literature with original data to review the JIF, empirically discuss its critiques, examine adverse effects, and describe alternative journal indicators.
Results
The chapter identifies technical and interpretive limitations, documents adverse effects, and describes alternative journal-based indicators for addressing some limitations.
Takeaways & Limitations
The JIF should be interpreted and applied responsibly, with awareness of its consequences, because journal reputation and indicators will likely remain central to research evaluation.
Takeaways & Limitations
Because citation distributions are skewed, the JIF is a weak predictor of individual papers’ citation rates and should not be used to evaluate individual researchers or papers.
Abstract
from arXiv · showhide
The Journal Impact Factor (JIF) is, by far, the most discussed bibliometric indicator. Since its introduction over 40 years ago, it has had enormous effects on the scientific ecosystem: transforming the publishing industry, shaping hiring practices and the allocation of resources, and, as a result, reorienting the research activities and dissemination practices of scholars. Given both the ubiquity and impact of the indicator, the JIF has been widely dissected and debated by scholars of every disciplinary orientation. Drawing on the existing literature as well as on original research, this chapter provides a brief history of the indicator and highlights well-known limitations-such as the asymmetry between the numerator and the denominator, differences across disciplines, the insufficient citation window, and the skewness of the underlying citation distributions. The inflation of the JIF and the weakening predictive power is discussed, as well as the adverse effects on the behaviors of individual actors and the research enterprise. Alternative journal-based indicators are described and the chapter concludes with a call for responsible application and a commentary on future developments in journal indicators.
1. Introduction
The JCR introduced comprehensive journal-level citation reporting, expanding from a publication tool into a foundation for scientometrics and research evaluation. The chapter examines how the JIF became influential across science while emphasizing its limitations and consequences.
- History: The 1975 JCR was ISI’s first comprehensive journal-level report, built from over 4.2 million references in 1974 papers across about 2,400 journals.Garfield had proposed the impact-factor concept earlier, but the 1975 JCR provided the first comprehensive reporting at journal level.
- Original purposes: Garfield originally presented the JCR as a tool for locating journals and studying science, including research fronts, disciplines, and interdisciplinary relationships.He cautioned against treating it primarily as a way to identify elite journals.
- Institutional uptake: The JCR became a backbone of scientometrics and was adopted alongside growing emphasis on research evaluation.Its applications extended to quantitative questions concerning science’s planning, evaluation, sociology, and history.
- Consequences: JIF-driven evaluation influenced research topics, dissemination practices, university hiring, and the broader science system.These effects followed the growing use of journal indicators to assess research success.
- Scope: The JIF attracted pervasive attention across scientific and medical fields, with more than 5,800 Web of Science Core Collection articles mentioning it by August 2017.The chapter focuses on central limitations raised in the literature and scientific community rather than surveying that literature exhaustively.
- Chapter aims: The chapter combines existing literature with original data to examine technical and interpretive critiques, adverse effects, and future journal-based measures.Critiques include numerator–denominator asymmetry, self-citations, citation-window length, skewness, and field- and time-dependence.
2. Calculation and reproduction
The JIF is a short-window ratio of citations to recent journal content, but its calculation embeds classification and timing choices. Reproduction analysis shows that the official value can be closely approximated from standard Web of Science indexes with careful data cleaning.
- Calculation: The 2016 JIF divides citations received in 2016 by 2014–2015 items by the number of citable items published in 2014–2015.The numerator counts citations to the two preceding publication years, while the denominator counts citable items from those years.
- Calculation: Citable items are limited to articles and reviews in the denominator but not the numerator, so the JIF is not exactly a mean citation count.It is generally interpreted as the short-term mean number of citations received by papers in a journal.
- Timing: The two-year publication window combines papers with nearly three years of citation opportunity and papers with only slightly more than one year.Presenting the resulting JIF to three decimals has been criticized as false precision.
- Coverage: JIFs are assigned annually to journals indexed in SCIE and SSCI, whereas AHCI journals generally lack JIFs because their citations have longer half-lives.Some social-history journals indexed in SSCI are included.
- Reproduction: With access to the data and careful cleaning, the JIF can be reproduced using the standard citation indexes in the Web of Science Core Collection.The WOS-derived JIF was very similar to the official JCR JIF, and the analysis found no evidence that Conference Proceedings or Book Citation indexes were included.
3. Critiques
The JIF has technical and interpretive limitations that affect how its values should be understood across journals, disciplines, and time. Evidence concerns numerator–denominator asymmetry, self-citations, short citation windows, skewed citation distributions, weakening article-level prediction, and field differences.
- 3.1 The numerator / denominator asymmetry: The numerator–denominator asymmetry varies with front material and citation completeness, and is less consequential for disciplinary journals than for Science and Nature.Non-citable items and unmatched citations comprise 9.8%–20.6% of citations in the examined journals; the effect of non-citable items is greater for interdisciplinary journals.
- 3.2 Journal self-citations: Including journal self-citations creates manipulation concerns, but excluding them can penalize niche journals and certain specialties.The authors describe self-citation treatment as a trade-off rather than a uniformly correct adjustment.
- 3.3 Length of citation window: Citation accumulation differs substantially across disciplines, making the two-year window capture only 7%–16% of citations received over 30 years.The first two years capture 16% for Physics, 15% for Biomedical Research, 8% for Social Science, and 7% for Psychology papers.
- 3.4 Skewness of citation distributions: Citation distributions are highly skewed: most papers receive few citations, while a small minority receive many, so journal averages poorly represent individual articles.Across journals, only about one-third of articles are likely to reach the corresponding JIF value; only 1.3% of journals have at least half their articles reaching it.
- 3.4 Skewness of citation distributions: The JIF’s relationship with article citedness exists but has weakened over time, limiting its predictive use at the article level.A matched-paper study found that papers in the highest-JIF journals received twice as many citations as twins in the lowest-JIF journals, but the relationship weakened over time.
- 3.5 Disciplinary comparison: Because citing practices differ across fields, the JIF cannot validly compare disciplines; higher values often reflect reference-list length and reference age.Medical researchers are more likely than mathematicians or social scientists to publish in high-JIF journals because of disciplinary publication and referencing practices.
4. Systemic Effects
The JIF shapes research evaluation and scholarly behavior, while incentives tied to it encourage manipulation, strategic submission, and the prioritization of rankings over scientific considerations. Its skewed distribution also makes individual-level evaluation especially problematic, and its prestige has spawned opaque imitation metrics.
- Metrics shape policy, resource allocation, competition, short-termism, and scholars’ research and dissemination practices.
- JIF engineering exploits calculation features through front material, excessive self-citation, citation coercion, and citation cartels.A study of nearly 7,000 scholars found that most would acquiesce to editorial coercion, while 20% reported coercive self-citation; a follow-up study reported 14.1%.
- Evaluation policies and cash-based rewards make institutions and individuals complicit in maximizing JIF-related outcomes.Cash incentives were correlated with increased submissions but not increased publication, while authors often submit first to higher-JIF journals before moving down the ranking ladder.
- Emphasizing JIFs prioritizes Web of Science coverage biases and incentivizes English-language natural and medical-science journals over other venues.
- Because citation distributions are skewed, the JIF weakly predicts individual article citation rates and should not evaluate individual researchers or papers.Across journals, only about one-third of articles are likely to reach their journal’s JIF value.
- The JIF’s status as a brand has produced fake impact factors and other opaque indicators associated with predatory publishing.More than 50 organizations were identified as providing questionable or misleading metrics, while some products lack transparency in how they are compiled.
5. What are the alternatives?
Several journal indicators complement or compete with the JIF by weighting citation networks, normalizing for disciplinary citation potential, or simplifying document-type treatment. Despite these alternatives, the JIF remains dominant because evaluation systems have standardized and institutionalized its use.
- Eigenfactor Metrics: Eigenfactor Score and Article Influence Score rank journals using citation-network structure and weight citations from central journals.
- SNIP: SNIP normalizes citation impact across fields using citing-paper sets and avoids JIF asymmetries created by non-citable items.SNIP still includes self-citations and can be biased toward journals with many review articles.
- SJR: SJR uses eigenvector centrality to calculate journal prestige by weighting links according to co-citation closeness.
- CiteScore: CiteScore averages citations received in one year by all document types published during the preceding three years.Including all document types removes concerns about asymmetries between cited and citing items, although it introduces other concerns discussed in the chapter.
- None of these indicators has displaced the JIF, partly because scholars and institutions have internalized its standardized role in hiring, promotion, and grant evaluation.
6. The future of journal impact indicators
Future journal impact indicators must address both technical limitations and the consequences of how indicators are applied. Although alternatives and refinements exist, responsible interpretation remains necessary because journal reputation continues to function as academic capital.
- Technical corrections to the JIF are possible, but adding alternative indicators instead of replacing the original creates standardization problems.Tailor-made indicator systems can weaken communication across researchers, administrators, evaluators, and policymakers.
- Skewed citation distributions, disciplinary differences, and JIF inflation restrict the indicator to ranking contemporary journals within the same discipline.Because fewer than one-third of articles are likely to reach the JIF citation value, the indicator should not evaluate individual articles or scholars.
- Systemic disruption arises primarily from misapplication within research evaluation systems, where the JIF has become associated with academic capital.These effects are linked not only to the indicator itself but also to journals' role as vectors of scientific capital.
- Garfield recommended using JCR information within a broader decision framework and rarely in isolation from other objective and subjective factors.He specifically cautioned against comparing citation rates across disciplines and noted factors including author reputation, accessibility, circulation, and cost.
- Despite documented limitations and consequences, journal-based indicators will likely remain part of research evaluation while journals remain central to knowledge dissemination.The chapter therefore calls for users to interpret and apply indicators responsibly and with awareness of their consequences.