Source-linked AI summary
Differences in Impact Factor Across Fields and Over Time
Benjamin M. Althouse, Jevin D. West, Theodore Bergstrom, Carl T. Bergstrom
TL;DR
The paper asks why impact factors rise over time and differ across fields, then decomposes these patterns using JCR citation and article data from 1994–2005. It finds that longer reference lists chiefly explain inflation over time, while the fraction of citations to JCR-indexed literature chiefly explains cross-field differences.
Problem
Impact factors are widely used as indicators of journal influence, yet they have increased over time and vary widely across academic fields.
Method
The paper decomposes impact factors into field growth, citations per paper, recent-citation fraction, and the fraction of citations to JCR-indexed items using JCR data.
Results
Increasing citations in reference lists are the greatest contributor to impact-factor inflation, while JCR-indexed citation fractions contribute most to cross-field differences.
Takeaways & Limitations
Scientific-literature growth and cross-field differences in field size and growth rate have little influence on impact-factor inflation or cross-field impact-factor differences.
Abstract
from arXiv · showhide
The bibliometric measure impact factor is a leading indicator of journal influence, and impact factors are routinely used in making decisions ranging from selecting journal subscriptions to allocating research funding to deciding tenure cases. Yet journal impact factors have increased gradually over time, and moreover impact factors vary widely across academic disciplines. Here we quantify inflation over time and differences across fields in impact factor scores and determine the sources of these differences. We find that the average number of citations in reference lists has increased gradually, and this is the predominant factor responsible for the inflation of impact factor scores over time. Field-specific variation in the fraction of citations to literature indexed by Thomson Scientific's Journal Citation Reports is the single greatest contributor to differences among the impact factors of journals in different fields. The growth rate of the scientific literature as a whole, and cross-field differences in net size and growth rate of individual fields, have had very little influence on impact factor inflation or on cross-field differences in impact factor.
I. INTRODUCTION
Impact factors have risen over time and differ substantially across fields, motivating an analysis of the sources of both patterns using JCR data from 1994–2005.
- 109 journals had impact factors of at least 10 in 2006, compared with 7 journals in 1969.Science’s impact factor rose from 3.0 in 1969 to 30.0 in 2006.
- Average impact factor increased about 2.6% per year from 1994 to 2005.
- In 2006, average impact factor was 0.8 in economics and 4.8 in molecular and cell biology, a sixfold difference.The field maxima were 4.7 and 47.4, respectively.
- The paper examines why impact factors increase over time and why they differ across fields using citation and article counts from JCR editions.The data cover 1994–2005 and include the Science and Social Science editions.
II. CHANGES IN IMPACT FACTOR OVER TIME
The paper defines impact factor as citations received by recent articles, normalized by the number of articles published in the journal during the target window.
- Impact factor counts citations made during a census period to articles published in an earlier target window.
- Thomson Scientific’s impact factor uses a one-year census period and the two preceding years as the target window.
- For journal i in year t, n_i^t counts citations to its t−1 and t−2 volumes.
- A_i^t denotes the number of articles appearing in journal i during year t.
A. Impact factors of individual journals
Across 4,300 continuously indexed journals, impact factors generally increased from 1994 to 2005, with gains occurring across the distribution and largely independent of initial scores.
- About 80% of the 4,300 continuously indexed journals increased in impact factor over the eleven years.
- Impact factor scores increased annually, predominantly through the midrange of the distribution.
- Most journals experienced steady impact-factor increases independently of their initial impact factors.
B. Weighted average impact factor
The weighted average gives greater influence to journals publishing more articles. Across the JCR, weighted impact factor increased from 1994 to 2005, while the figure tracks journal-level and distribution-wide changes.
- Journals publishing more articles receive larger weights when calculating the average rate of impact-factor change.
- Weights are based on articles published during the target years, with A_i^t summed across all journals in S_t.
- Figure 1 compares 1994 with 2005 scores for 4,300 continuously listed journals and shows rank-order distributions across years.Darker shading indicates higher point density in panel (a) and later years in panel (b).
- 2.6% per year was the weighted average impact-factor increase for all JCR-listed journals from 1994 to 2005.For journals indexed throughout the period, the average annual increase was 1.6%.
C. Decomposing changes in average impact factor
The paper decomposes average impact factor into four multiplicative components to identify what drives its increase over time. Increasing reference-list length is the predominant contributor, while article growth and citation-timing or indexing fractions contribute little or negatively.
- Decomposition: The decomposition expresses average impact factor as the product of four factors: article-count ratio, citations per article, recent-citation fraction, and JCR-indexed citation fraction.These components are denoted α, c, p, and v.
- Article growth: Article-count growth cannot explain impact-factor inflation: the α component contributed −0.001 on average, implying roughly 0.1% annual decline if acting alone.A constant article-growth rate produces a constant impact factor rather than inflation.
- Citation timing and coverage: The recent-citation fraction reduced impact factor by about 1% per year, while the indexed-citation fraction contributed only about 0.1% per year.These effects correspond to the p and v components of the decomposition.
- Reference-list length: 3.6% average annual growth in citations per article adequately explains the 2.6% annual impact-factor increase despite declines from other components.The authors identify increasing reference-list length as the single greatest contributor to inflation from 1994–2005.
- Interpretive boundary: The analysis finds no evidence that larger fields or increasing numbers of citable articles explain longer reference lists, and it does not distinguish among other proposed causes.Possible causes include field expansion, easier electronic literature access, and citation pressures from referees or authors.
D. Natural Selection?
The authors test whether journal turnover in the JCR inflated average impact factors. Although entering journals initially scored higher than exiting journals, turnover reduced the full-set average; later growth among entrants was stronger than average.
- 4,202 journals entered and 2,415 exited the JCR from 1994 to 2005, creating substantial journal turnover.
- Entering journals had significantly higher impact factors than exiting journals, but their averages remained below the full-JCR average.The difference was statistically significant (D = 0.074, p = 5.6e-7).
- Because nearly twice as many journals entered as exited, JCR turnover decreased the average impact factor of the full journal set.
- 18,200 citations per year was the average cost of journal turnover during 1995–2004, so natural selection did not contribute to impact-factor inflation.
- 6% was the average annual growth rate of journals entering during 1995–2004, more than twice the overall dataset’s rate.
III. DIFFERENCES IN IMPACT FACTOR ACROSS FIELDS
Impact-factor differences across fields are analyzed by decomposing them into field growth, citation-list length, citation timing, and the share of citations directed to JCR-indexed literature. The share directed to indexed literature explains the largest portion of variation, while field size and growth contribute little.
- Impact-factor variation across disciplines may reflect citation practices, citation lag, and the fraction of citations directed to JCR-indexed literature.The fields are delineated into 88 non-overlapping science and social-science categories using citation patterns.
- Mathematics has a weighted impact factor of IF = 0.56, versus 4.76 for Molecular and Cell Biology, an eight-fold difference.
- The field-level decomposition expresses impact-factor growth as ρt(IFF) = ρt(αF) + ρt(cF) + ρt(pF) + ρt(vF).The components represent field growth, citations per paper, citation timing, and citations to JCR-listed items.
- The fraction of citations to JCR-indexed literature, vF, explains the greatest fraction of variation among the four components.
- The four-variable model has r2 = 0.855, but its prediction is imperfect because citations do not all come from the same field.Multiple regressions also indicate that α has little, if any, predictive power.
- 50% of explained variance in IF is accounted for by v, while c accounts for an additional 27%.Fields citing heavily within the ISI dataset have higher scores, whereas fields citing less heavily within it have lower scores.
A. Inflation differences across fields
Inflation rates differ substantially across fields, but these differences are not explained by field size or by the fields’ weighted impact-factor levels.
- Impact-factor inflation rates vary substantially across fields.
- Field size explains almost none of the variation in inflation rate, with r2 = 0.001.
- Initial weighted impact-factor scores likewise explain little of inflation-rate variation, with r2 = 0.018.
Summary
Impact factors vary across fields and over time, with different mechanisms dominating each pattern. Increasing citations per article chiefly drives inflation over time, whereas the share of citations to JCR-indexed literature chiefly explains cross-field differences.
- Impact factors are decomposed into field growth, citations per paper, recent-citation share, and citations to JCR-listed items.
- Increasing citations in reference lists is the greatest contributor to impact-factor inflation over time.
- Differences in the fraction of citations to JCR-indexed literature are the greatest contributor to cross-field impact-factor differences.
- Cross-field differences also reflect citations per paper and the fraction of references published within two years.
- Scientific-literature growth and cross-field differences in field size and growth rate have very little influence on either pattern.
Competing interests
The authors are identified as developers of Eigenfactor, a method for ranking journal influence using citation network data.
- The authors developed Eigenfactor.Eigenfactor ranks journal influence using citation network data.
- Eigenfactor is a method for ranking journal influence.The method uses citation network data.
- The method is based on citation network data.The passage identifies Eigenfactor as the authors’ journal-influence ranking method.
APPENDIX A: Deriving αt, ct, pt and vt from the JCR data
The appendix derives α_t, c_t, p_t, and v_t from JCR article and citation data. It defines field-specific and dataset-wide calculations using article counts, outgoing citations, recent citations, and citations into JCR-listed literature.
- A_t averages article counts for year t reported in the t + 1 and t + 2 JCR datasets.The averaged count is used because the two later datasets typically disagree on the number of articles published in year t.
- α_t is calculated as A_t/(A_t−1 + A_t−2).
- c_t divides total outgoing citations for year t by total articles for year t.This measures citations relative to the number of articles published in that year.
- p_t divides citations to papers from the previous two years by total outgoing citations in year t.The cited papers were published in years t −1 and t −2.
- For the entire dataset, v_t divides incoming citations to the previous two years by total outgoing citations over that period.The denominator combines the relevant citation flows represented by arrows A + C in Figure 3.
- For field F, v_t,F divides two-year citations from F into JCR literature by all two-year outgoing citations from F.The numerator includes citations within F and from F to the rest of JCR; the denominator includes arrows A + B + C.