Source-linked AI summary
The weakening relationship between the Impact Factor and papers' citations in the digital age
George A. Lozano, Vincent Lariviere, Yves Gingras
TL;DR
The paper examines how electronic dissemination changes the relationship between journal Impact Factors and papers’ citations. It finds that this relationship strengthened through most of the 20th century but has weakened since the digital age, potentially undermining the Impact Factor’s usefulness.
Problem
Electronic dissemination lets researchers access papers individually rather than reading entire journals, raising questions about the journal-level basis of Impact Factors.
Method
The study compares the relationship between journal Impact Factors and papers’ citations across the sciences, including physics and social sciences.
Results
The link between Impact Factors and paper citations strengthened through most of the 20th century but has weakened steadily since the beginning of the electronic age.
Takeaways & Limitations
If this pattern continues, the Impact Factor may cease to be a meaningful measure of the quality of journals, papers, and researchers.
Takeaways & Limitations
The method used to calculate Impact Factors is not entirely clear and is therefore irreproducible.
Abstract
from arXiv · showhide
Historically, papers have been physically bound to the journal in which they were published but in the electronic age papers are available individually, no longer tied to their respective journals. Hence, papers now can be read and cited based on their own merits, independently of the journal's physical availability, reputation, or Impact Factor. We compare the strength of the relationship between journals' Impact Factors and the actual citations received by their respective papers from 1902 to 2009. Throughout most of the 20th century, papers' citation rates were increasingly linked to their respective journals' Impact Factors. However, since 1990, the advent of the digital age, the strength of the relation between Impact Factors and paper citations has been decreasing. This decrease began sooner in physics, a field that was quicker to make the transition into the electronic domain. Furthermore, since 1990, the proportion of highly cited papers coming from highly cited journals has been decreasing, and accordingly, the proportion of highly cited papers not coming from highly cited journals has also been increasing. Should this pattern continue, it might bring an end to the use of the Impact Factor as a way to evaluate the quality of journals, papers and researchers.
Introduction
Impact Factor (IF) became widely used as a proxy for papers’ eventual citations and as a measure of journal, paper, and researcher quality. The electronic age enables topic-based searches and article-level access, potentially weakening the link between journals’ IFs and individual paper citations.
- Introduction: Researchers, editors, and publishers increasingly use and promote IFs as measures or proxies of journal and paper quality.Researchers consider IFs when choosing publication outlets, while editors and publishers use them strategically and promotionaly.
- Introduction: Before electronic dissemination, high-profile journals with broad circulation gave their papers greater opportunities to be read and cited.Printed-journal access tied papers’ visibility to journal circulation and availability.
- Introduction: Electronic searches let researchers find specific articles across journals, so papers can be read and cited based on their own merits.Online availability and database indexing reduce dependence on a journal’s physical availability, reputation, or IF.
- Introduction: The paper tests whether changing literature-use practices have reduced the correlation between paper citation rates and journal IFs over time.It also examines whether highly cited papers increasingly come from journals outside the highest-IF group.
- Introduction: The analysis covers natural and medical sciences together, physics, and social sciences from 1900 to 2011.These disciplinary groups support comparisons of the transition toward electronic dissemination.
Methods
The study combines Web of Science records with recalculated journal IFs and paper-level citation measures for 1902–2009. It evaluates the IF–citation relationship using correlation and concentration indicators while addressing uncited papers and data limitations.
- Methods: The analysis covers natural and medical sciences, physics, and social sciences, using discipline classifications adapted from the U.S. National Science Foundation.Each journal is assigned to one discipline and specialty.
- Methods: Papers were included when published in journals for which an IF could be calculated, while citations from excluded papers still counted for included papers.Humanities papers were excluded because of long citation windows and high uncitedness, but citations in humanities journals were included.
- Methods: The study recalculated IFs by correcting numerator–denominator asymmetry and counting citations during the full two years after publication.This produces comparable IF and citation-rate measures over the complete period.
- Methods: Analyses were repeated without uncited papers to reduce their influence on early-period relationships.The study also notes that Thomson Reuters does not provide IFs for the full period and its exact calculation method is not entirely clear or reproducible.
- Methods: The coefficient of determination (r2) relates each paper’s journal IF to its citations during the following two years for each year.A second indicator measures the yearly percentages of highly cited papers published in, or outside, highly cited journals.
Results
The relationship between journal IFs and paper citations strengthened through much of the twentieth century but weakened from around 1990 onward. Highly cited papers increasingly appeared outside the highest-IF journals, with the decline beginning earlier in physics.
- Results: Since scientific information began being disseminated electronically around 1990, the relationship between journal IF and paper citation rates has weakened.This pattern appears across the natural and medical sciences, physics, and social sciences.
- Results: In physics, the decline appears to begin toward the end of the 1980s, earlier than the comparable pattern in social sciences.The disciplinary difference was not significantly different, and smaller samples produced greater variation in r2.
- Results: Removing uncited papers produces a clearer trend with fewer fluctuations but does not strengthen the IF–citation relationship.The relationship remains within the same order of magnitude.
- Results: Since 1990, the share of top 10% papers published in top 10% journals fell from about 5.25% to 4.50%.The corresponding share of top 10% papers outside the top-IF journals rose from 52% to about 56%.
- Results: For the top 5%, papers in top-5% journals fell from about 2.25% in 1990 to 1.90% in 2009, while papers outside those journals rose from about 55% to 62%.The results indicate that highly cited literature increasingly comes from a broader range of journals.
Discussion and conclusions
The relationship between Impact Factors and paper citations strengthened through most of the 20th century but has weakened since the electronic age began, especially in physics. As highly cited papers increasingly appear outside the highest-IF journals, the IF may cease to meaningfully measure journal, paper, or researcher quality.
- Impact Factor use and criticism: Impact Factors were originally developed to help libraries make purchasing decisions, but later became widely used as measures of journal quality.The paper notes that the IF has also been criticized on methodological grounds, including citation-counting practices, disciplinary differences, database coverage, non-normal citation distributions, arbitrary time windows, and editorial manipulation.
- Changing citation relationship: Throughout most of the 20th century, the link between journal Impact Factors and papers’ citation rates became stronger.The analysis identifies a subsequent weakening of this relationship as scientific literature moved into the electronic domain.
- Changing citation relationship: Since the beginning of the electronic age, this link has weakened steadily, beginning slightly earlier in physics because the field adopted electronic dissemination sooner.The timing is associated with changes in how researchers access scientific literature.
- Distribution of highly cited papers: The percentage of highly cited papers coming from the highest-IF journals has decreased, while citations have become more widely spread among journals.This indicates that high-IF journals are losing their previous stronghold as the sole repositories of highly cited papers.
- Implications: Electronic searches let researchers find individual papers across journals, weakening the basis for transferring journal reputation to papers or researchers.The authors argue that direct assessment of papers, including reading them, should replace automatic reliance on journal IFs; open-access journals and repositories may gain prominence.
- Implications: If the pattern continues, the Impact Factor may cease to be a meaningful measure of journal, paper, and researcher quality.The paper connects this possibility to the increasing appearance of important papers in more diverse venues and to the changing modes of scientific dissemination and access.