Source-linked AI summary
Towards a new crown indicator: An empirical analysis
Ludo Waltman, Nees Jan van Eck, Thed N. van Leeuwen, Martijn S. Visser, Anthony F. J. van Raan
TL;DR
Citation counts require normalization because they vary across fields and publication years. The paper empirically compares CWTS’s current CPP/FCSm crown indicator with its planned MNCS replacement across aggregation levels and examines recent publications. The mechanisms differ little for large institutions and countries but more at research-group and journal levels, while recent publications require special attention.
Problem
Citation counts vary across fields and publication ages, preventing direct comparison without controlling for field and publication year.
Method
The paper empirically compares CWTS’s current CPP/FCSm normalization mechanism with the planned MNCS mechanism across four aggregation levels, emphasizing recent publications.
Results
Differences between CPP/FCSm and MNCS are very small for large research institutions and countries but somewhat larger for research groups and journals.
Takeaways & Limitations
The comparison supports CWTS’s planned move from CPP/FCSm to MNCS while indicating that aggregation level matters when interpreting differences between the indicators.
Takeaways & Limitations
Because MNCS weights recent publications equally with older ones despite their low and weakly predictive citation counts, recent publications can introduce substantial noise, especially at lower aggregation levels.
Abstract
from arXiv · showhide
We present an empirical comparison between two normalization mechanisms for citation-based indicators of research performance. These mechanisms aim to normalize citation counts for the field and the year in which a publication was published. One mechanism is applied in the current so-called crown indicator of our institute. The other mechanism is applied in the new crown indicator that our institute is planning to adopt. We find that at high aggregation levels, such as at the level of large research institutions or at the level of countries, the differences between the two mechanisms are very small. At lower aggregation levels, such as at the level of research groups or at the level of journals, the differences between the two mechanisms are somewhat larger. We pay special attention to the way in which recent publications are handled. These publications typically have very low citation counts and should therefore be handled with special care.
1. Introduction
Citation counts vary across scientific fields and publication ages, so direct comparisons are inappropriate. The paper empirically compares the current and planned CWTS normalization mechanisms across aggregation levels, with special attention to recent publications.
- Citation counts cannot be directly compared across publications from different fields or years because citation averages vary by field and age.
- The crown indicator normalizes citation counts using expected citations for publications of the same document type, field, and year.
- The alternative mechanism averages each publication’s ratio of actual to expected citations rather than dividing aggregate actual citations by aggregate expected citations.
- Prior theoretical work concluded that the alternative mechanism has more satisfactory field-correction properties because it weights all publications equally.
- The empirical comparison examines the current and alternative mechanisms at research-group, institution, country, and journal levels, emphasizing how recent publications are handled.
2. Definitions of indicators
This section defines the current CPP/FCSm crown indicator and the planned MNCS crown indicator. Both use field- and year-based expected citation counts, but they differ in whether normalization uses a ratio of averages or an average of ratios.
- CPP/FCSm means citations per publication divided by mean field citation score, while MNCS means mean normalized citation score.
- CPP/FCSm is the current CWTS crown indicator, whereas MNCS is the new crown indicator CWTS is going to adopt.
- For publication i, expected citations equal the average citations of publications in the same field and year.
- CWTS defines fields normally using subject categories in the Web of Science database.
- CPP/FCSm normalizes through a ratio of averages, while MNCS normalizes through an average of ratios.
3. How to handle recent publications?
Recent publications require special handling because their low citation counts provide weak evidence about long-run impact, while the MNCS gives them equal weight. Excluding the newest publications can reduce noise, especially at low aggregation levels.
- The MNCS weights recent and older publications equally, unlike CPP/FCSm, whose recent-publication effects are typically small.Equal weighting gives recent publications a stronger influence despite their low and less predictive citation counts.
- Citation counts vary greatly across fields, with biochemistry and molecular biology roughly an order of magnitude above mathematics.This demonstrates why field normalization is necessary.
- Publications receive almost no citations in their publication year because the citation process is delayed by writing, review, revision, copyediting, and publication backlogs.In mathematics, publications are often unlikely to be cited in the following year as well.
- Short-run citation counts can weakly predict long-run impact: mathematics reached 0.25 for citations by 1999 versus 2008, while biochemistry and molecular biology reached only 0.55.For mathematics, the 2000-to-2008 correlation was 0.59, still described as moderate.
- At lower aggregation levels, recent publications can introduce significant noise because only a limited number of publications are available.The paper identifies research groups and individual researchers as examples of such levels.
- Leaving out publications with less than one year to earn citations may reduce noise, although it also discards relevant information.This is presented as a possible way to alleviate the problem in MNCS calculations.
4. Empirical comparison
The study compares CPP/FCSm with two MNCS variants across four aggregation levels, focusing on how excluding very recent publications affects indicator relationships. Differences are very small for research institutions and countries but larger for research groups and journals.
- Data and indicators: MNCS1 includes all publications, whereas MNCS2 excludes publications that had less than one year to earn citations.CPP/FCSm, MNCS1, and MNCS2 are compared using Pearson and Spearman correlations and scatter plots.
- Data and indicators: The comparison covers research groups, research institutions, countries, and journals, using Web of Science publications and excluding arts and humanities.The study uses article, note, and review publications; the data-set characteristics are listed in Table 3.
- Research groups: Research groups show a moderately strong CPP/FCSm–MNCS1 relation, while the relation with MNCS2 is considerably stronger.Several groups display substantial score differences, especially when recent publications have high normalized citation scores.
- Research groups: Recent publications can strongly affect MNCS2: a publication with five citations produced a high normalized score because its expected citation count was small.The equal weighting of publications in MNCS2 makes such recent publications influential.
- Research institutions and countries: Research institutions and countries show very strong relations among CPP/FCSm, MNCS1, and MNCS2, with the three country indicators yielding very similar rankings.A highly cited recent University of Göttingen article creates a large CPP/FCSm–MNCS1 difference, but its MNCS2 score is close to CPP/FCSm.
- Journals: Journals generally show strong CPP/FCSm–MNCS1 relations, but many more large differences occur than with MNCS2; remaining MNCS2 exceptions are a small minority.The journal data set contains more than 8000 journals.
5. Conclusions
The empirical comparison finds minimal differences between CPP/FCSm and MNCS at high aggregation levels, but larger differences for research groups and journals. Recent publications require special attention because their low citation counts can add noise to MNCS, especially at lower aggregation levels.
- Journal-level differences: 17.68, 32.28, and 2.14 are the CPP/FCSm, MNCS1, and MNCS2 scores, respectively, for the journal with the highest CPP/FCSm score.An article cited 3489 times explains the unusually different scores because it receives different weight across the indicators.
- Aggregation levels: At high aggregation levels, differences between the CPP/FCSm and MNCS indicators are very small.This applies to large research institutions and countries.
- Aggregation levels: At lower aggregation levels, differences between the CPP/FCSm and MNCS indicators are somewhat larger.The paper identifies research groups and journals as examples.
- Recent publications: Recent publications can introduce substantial noise into MNCS because they receive equal weight despite having low citation counts and poorly predictable long-run impact.Leaving out publications with less than one year to earn citations substantially strengthens the relation between CPP/FCSm and MNCS at lower aggregation levels.