Source-linked AI summary
Scopus's Source Normalized Impact per Paper (SNIP) versus a Journal Impact Factor based on Fractional Counting of Citations
Loet Leydesdorff, Tobias Opthof
TL;DR
The paper examines field-specific citation behavior and the lack of significance testing in impact-factor comparisons. It proposes fractional citation counting to contextualize citations and test aggregated impacts, finding that Annals of Mathematics is not significantly lower despite a five-fold impact-factor difference.
Problem
Impact-factor variants face field-specific citation effects and lack statistical tests for whether differences are significant.
Method
Fractional citation counting contextualizes citations at the paper level and aggregates weighted citation impacts for statistical testing.
Results
Annals of Mathematics is not significantly lower in citation impact despite a five-fold difference in impact factors.
Takeaways & Limitations
Weighted citation impact can address field-specific citation behavior while allowing significance testing across sets of documents.
Takeaways & Limitations
The SNIP indicator does not solve both identified problems, and its normalization decisions prevent significance testing.
Abstract
from arXiv · showhide
Impact factors (and similar measures such as the Scimago Journal Rankings) suffer from two problems: (i) citation behavior varies among fields of science and therefore leads to systematic differences, and (ii) there are no statistics to inform us whether differences are significant. The recently introduced SNIP indicator of Scopus tries to remedy the first of these two problems, but a number of normalization decisions are involved which makes it impossible to test for significance. Using fractional counting of citations-based on the assumption that impact is proportionate to the number of references in the citing documents-citations can be contextualized at the paper level and aggregated impacts of sets can be tested for their significance. It can be shown that the weighted impact of Annals of Mathematics (0.247) is not so much lower than that of Molecular Cell (0.386) despite a five-fold difference between their impact factors (2.793 and 13.156, respectively).
Introduction
The paper examines field-specific citation differences and argues that fractional, paper-level citation weighting can address both field normalization and statistical testing limitations. It accepts SNIP’s normalization goal but criticizes its aggregation procedure.
- Citation frequencies differ systematically between fields such as mathematics and biomedicine, complicating comparisons based on conventional impact factors.
- A citing paper with n references contributes 1/n to a cited paper’s weighted impact, contextualizing citations at the paper level.
- SNIP aggregates citing papers at the citing-journal level, whereas the paper argues normalization should precede addition.
- Weighting citations before averaging produces an impact factor that, unlike the ISI-IF or SNIP, permits statistical significance testing.
- The proposed approach addresses both field-specific citation behavior and the lack of statistics, while the authors describe SNIP’s elaboration as problematic.
The SNIP indicator
SNIP divides a raw impact-per-paper measure by a relative database citation potential, applying normalization through aggregated journal-level quantities. The paper argues that normalizing before averaging better supports statistical testing.
- The SNIP indicator: SNIP is defined as Raw Impact per paper divided by Relative Database Citation Potential.
- The SNIP indicator: Raw Impact per paper is technically similar to a three-year Impact Factor, although Scopus counts articles, proceedings papers, and reviews in numerator and denominator.
- The SNIP indicator: Relative Database Citation Potential uses the median of journals’ Database Citation Potentials, based on mean one- to three-year-old cited references per paper in citing journals.
- The SNIP indicator: SNIP performs normalization in both numerator and denominator, including averaging skewed distributions and taking a median of means.
- The SNIP indicator: Normalizing first and then averaging normalized values allows statistical testing in research evaluation.
Weighting the citation impact in terms of the citing papers (Methods)
The paper retrieves cited and citing records under defined publication-year and document-type constraints, then fractionally weights citations by the reference counts of citing papers. This produces a field-sensitive impact measure and changes the comparison between journal indicators.
- Retrieval: Citations are retrieved for specified cited journals, publication years, and document types, with 2,996 cited-side records and 629 citing documents in the example.The procedure uses database search fields and restricts both cited and citing records to selected document types and publication-year conditions.
- Fractional weighting: Each citation contributes 1/n, where n is the number of references in the citing paper.Fractional counting is performed at the retrieved-paper level rather than by averaging citation counts for each citing journal.
- Fractional weighting: The Journal of Electronic Materials has a weighted citation impact of 0.099, calculated as 78.926 weighted citations divided by 794 citable items.The cited journal was referenced 629 times in 151 journals, whose documents contained 19,459 cited references, including 1,740 to this journal.
- Indicator comparison: Weighted impact preserves the rank order of the three journals relative to the ISI-IF and three-year impact factor, whereas SNIP reverses Annals of Mathematics and Molecular Cell.The paper uses this difference to illustrate that changing the order of normalization operations can affect rankings.
- Indicator comparison: Molecular Cell scores less than twice Annals of Mathematics on weighted impact, despite a five-fold ISI impact-factor difference.The paper attributes much of the ISI difference to larger reference lists in biomedicine than in mathematics and concludes that weighted impact normalizes field-specific citation behavior.
- Indicator comparison: The weighted impact correlates with ISI-IF (r = 0.90; p < 0.05) and three-year IF (r = 0.94; p < 0.05), but SNIP is not significantly correlated with these indicators.For the five-journal set, weighted impact and SNIP are not significantly correlated in 2007 (r = 0.75; n.s.).
Statistics
The fractional-counting approach assigns paper-specific citation weights and supports statistical testing of citation distributions. Non-parametric comparisons identify significant differences and groups of journals with similar citation patterns.
- Statistical basis: Citation weights are paper-specific and therefore field-specific, without requiring an external subject-category index.The citing paper determines the relevant citation context through its references and field position.
- Statistical basis: The method provides full-fledged statistics for testing whether citation-distribution differences are significant.The distributions can be tested for any set, although the paper focuses on journals.
- Statistical tests: Kruskall-Wallis testing shows that the three journals differ significantly in their being-cited patterns.A post-hoc Bonferroni test also permits pairwise comparison of journal means.
- Statistical tests: The three journals also differ significantly from one another under the Bonferroni comparison.The reported comparison uses a .05 significance criterion for mean differences.
- Journal grouping: Annals of Mathematics is not significantly different from Inventiones Mathematicae in citation pattern.The paper notes that the measures are non-parametric and that dividing by the number of cited papers can affect the tested impact values.
- Journal grouping: The non-parametric test organizes journals into precise, though not necessarily distinct, groups with significantly similar being-cited patterns.The analysis extends the set with two mathematics journals selected as intellectually similar to Inventiones Mathematicae.
Conclusions and summary
The proposed weighted citation-impact method contextualizes citations at the article level, aggregates impacts across document sets, and enables statistical significance testing. It addresses field-specific citation behavior and the lack of comparative statistics that affect conventional impact measures, whereas SNIP does not solve both problems.
- Weighted citation impact contextualizes citations at the article level and aggregates impacts across sets of documents.
- Statistical testing allows the impacts of document sets, including journals, to be compared for significance.
- The method addresses field-specific citation behavior at the article level.
- The method also addresses the lack of statistics for comparing journal impacts under ISI Impact Factors and similar measures.
- SNIP does not solve the problems of field-specific citation behavior and missing significance statistics.
- The authors therefore suggest reconsidering SNIP in favor of the simpler weighted measure.