PaperlayerView exact revision on arXiv

Source-linked AI summary

Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts?

Lutz Bornmann, Werner Marx

arXiv:1410.5772v2cs.DLstat.OT

TL;DR

Discipline and publication period affect citation counts independently of publication quality, raising the question of which normalization methods best reflect peer assessments. The study compares cited- and citing-side indicators against F1000 recommendations. The indicators, including observed citation counts, show very similar relationships with peer assessments, though the biomedical focus limits the study’s scope.

  • Problem

    Citation counts vary with discipline and publication period independently of publication quality, so it is unclear which normalization methods best correspond to peer assessments.

    Source: §1 · p. 3

  • Method

    The study compares cited- and citing-side normalized indicators with expert peer assessments from F1000.

    Source: §1 · p. 4, §5 · p. 23

  • Results

    All indicators show very similar relationships with peer judgements, and observed citation counts perform similarly to normalized indicators.

    Source: §5 · p. 24

  • Takeaways & Limitations

    The results support treating the indicators as measuring normalized impact similarly when peer judgement is used as an external validity criterion.

    Source: §5 · p. 24

    Because F1000 papers are biomedical, these results do not reflect science’s full disciplinary diversity and require validation in other or multidisciplinary datasets.

    Source: §5 · p. 24

Abstract

from arXiv · show

Evaluative bibliometrics compares the citation impact of researchers, research groups and institutions with each other across time scales and disciplines. Both factors - discipline and period - have an influence on the citation count which is independent of the quality of the publication. Normalizing the citation impact of papers for these two factors started in the mid-1980s. Since then, a range of different methods have been presented for producing normalized citation impact scores. The current study uses a data set of over 50,000 records to test which of the methods so far presented correlate better with the assessment of papers by peers. The peer assessments come from F1000Prime - a post-publication peer review system of the biomedical literature. Of the normalized indicators, the current study involves not only cited-side indicators, such as the mean normalized citation score, but also citing-side indicators. As the results show, the correlations of the indicators with the peer assessments all turn out to be very similar. Since F1000 focuses on biomedicine, it is important that the results of this study are validated by other studies based on datasets from other disciplines or (ideally) based on multi-disciplinary datasets.

1 Introduction

Citation counts vary independently of publication quality with discipline and publication period, motivating normalization. This study compares cited- and citing-side indicators with peer assessments to examine how well they reflect expert judgements.

1 Introduction

  • Citation counts vary with discipline and publication period for reasons independent of publication quality, motivating normalization across both factors.Methods for normalizing citation impact have been developed since the mid-1980s.Source: §1 · p. 3
  • Cited-side normalization compares a paper’s citation count with similar publications from the same subject area and publication year.The comparison uses a reference set of similar publications.Source: §1 · p. 3
  • Citing-side normalization instead adjusts individual citations by the subject area and publication year of the citing publication.This approach is oriented toward citing rather than cited publications.Source: §1 · p. 3
  • The study tests which citation-normalization methods correspond more closely to peer assessments, a recognized way to validate bibliometric indicators.The authors frame the choice among methods as important because each has methodological advantages and disadvantages.Source: §1 · p. 3
  • The study extends Bornmann and Leydesdorff (2013) with a considerably larger dataset and citing-side indicators, alongside cited-side indicators and observed citation counts.The comparison asks whether normalized indicators measure research impact more accurately than unnormalized citation counts.Source: §1 · pp. 3–4

2 Normalization of citation impact

This section describes cited-side and citing-side methods for normalizing citation impact across disciplines and publication years, including indicators based on cited publications and methods that weight citations by their citing environments.

2 Normalization of citation impact

  • Average citation impact varies significantly by subject area: engineering and technology average 10.77 citations versus 16.85 in medical and health sciences; it also depends on publication year, with fewer citations expected for newer publications.Source: §2 · p. 4

2.1 Cited-side normalization of citation impact

  • Cited-side normalization compares a focal paper’s observed citations with expected citations for papers in the same subject category and publication year; a quotient of 1 represents average impact.The expected value is the average citation impact of papers in that reference set, and the observed-to-expected quotient is described as the current bibliometric standard.Source: §2.1 · p. 6
  • MNCS values are close to 1 across all subject categories, ranging from 0.87 to 1; this indicates normalization of citation impact with respect to both time and discipline.Source: §2.1 · p. 7
  • Hazen percentiles average around 50 across disciplines, indicating the same average normalized citation impact and the desired normalization effect.The underlying data set includes all Web of Science articles from 2007 to 2010.Source: Figure 1 · p. 5, §2.1 · p. 8
  • P100 averages are similar at about 8 in several disciplines but about 4 in humanities, showing that its normalization is not successful in all disciplines.The passage identifies medical and health sciences, agricultural sciences, and social sciences as disciplines with similar averages around 8.Source: Figure 1 · p. 5, §2.1 · p. 9
  • Mutz (in press) and Schreiber (2014) identify P100 weaknesses, including that a paper’s scale value can rise when another paper receives an additional citation.Source: §2.1 · p. 9
  • Bornmann and Mutz (in press) suggest P100’ as an improvement that accounts for the frequency of papers with equal citation counts when assigning ranks.Each rank is divided by the highest rank and multiplied by 100: 100*(i/imax).Source: §2.1 · p. 9

2.2 Citing-side normalization of citation impact – the weighting of individual

  • Because papers receive citations from multiple fields with distinct citation cultures, citing-side normalization adjusts citation weights for differences in expected citation rates among citing papers.Source: §2.2 · pp. 10–11
  • SNCS1 weights citations using journal-year average linked references, SNCS2 uses the citing paper’s linked-reference count, and SNCS3 combines the two approaches.Source: §2.2 · pp. 12–13
  • The linked-reference restriction may disadvantage fields citing material outside Web of Science; social sciences have significantly fewer linked than overall cited references.Source: §2.2 · p. 12
  • SNCS3 average scores were approximately 1 across disciplines, suggesting that the normalization method basically works.Source: §2.2 · p. 14

3 Methods

The methods use F1000 peer recommendations, link those records to bibliometric data, and compare indicators with recommendations using correlations and regression analyses.

3.1 Peer ratings provided by F1000

  • F1000 papers are selected by a peer-nominated global faculty of leading scientists and clinicians, who rate them Good, Very good, or Exceptional (scores 1, 2, or 3); only a restricted set of covered medical and biological journal papers is reviewed, and most are not.Source: §3.1 · pp. 14–15

3.2 Formation of the data set to which bibliometric data and altmetrics are attached

  • Matching F1000 records to bibliographic data yielded bibliometric data for 94,641 of 104,633 papers, or 91%.Matching used DOI and, when DOI matching failed, author, journal, volume, and issue information.Source: §3.2 · p. 16
  • Because citing-side scores were available only for 2007–2010, the analysis dataset was reduced to 50,082 records.The matched F1000 recommendations span 1980–2013.Source: §3.2 · p. 16

3.3 Statistical procedures and software used

  • The analyses calculate Spearman rank correlations and regression models between recommendations and each indicator, with indicator scores z-transformed for model comparison.Clustered errors account for recommendations nested within papers, and bootstrap estimation is used for standard errors.Source: §3.3 · p. 17
  • Predictive margins from the fitted models are presented graphically to convey the practical significance and direction of estimated effects.The margins supplement statistical significance tests with interpretable predictions.Source: §3.3 · pp. 17–18

4 Results

Mean citation rates differed substantially across subject categories, while normalized indicators showed broadly similar correlations with F1000 recommendations. Regression margins agreed for “good” papers but diverged more for higher-rated papers.

4.1 Mean citation rates

  • 14.69 citations per paper in anaesthesiology versus 107.22 in medicine, general & internal, showing large mean-rate differences across subject categories.Table 2 reports mean, minimum, and maximum citation counts using a three-year citation window.Source: §4.1 · p. 18, Table 2 · p. 18

4.2 Correlation

  • Using only each paper’s first recommendation reduced correlation coefficients for every indicator.The authors suggest this reflects recommendations for the same paper being more similar than recommendations for different papers.Source: §4.2 · p. 20
  • The citation indicator had the largest correlation with recommendation scores, though coefficient-height differences among indicators were slight within the two recommendation groups; coefficients ranged between small (r=.1) and medium (r=.3).Source: §4.2 · p. 20

4.3 Regression model

  • For “good” recommendations, indicator predictive margins agreed at around -1.6, about one and a half standard deviations below the mean.The paper reports this as agreement about the later impact of papers assessed as “good.”Source: §4.3 · p. 21

5 Discussion

Using peer judgments as an external criterion, the study finds that the indicators generally measure normalized impact similarly, with observed citation counts and SNCSs appearing stronger for papers rated “exceptional.” The authors caution that the biomedical F1000 data and its rating process limit how decisively the methods can be compared.

5 Discussion

  • The study tests concurrent validity by comparing indicators with peer judgments, assuming those judgments are independent of subject category and publication year.The authors treat stronger correlation with peer judgments as evidence that an indicator is better suited to measuring impact.Source: §5 · pp. 23–24
  • For “exceptional” papers, observed citation counts and SNCSs seem to have an advantage over other indicators for impact measurement.The differences between indicators are greater for this rating than for “good” or “very good” papers.Source: §5 · p. 24
  • Overall, all indicators involved measure normalized impact similarly when peer judgments serve as the external validity criterion.Validity is especially similar for papers rated “good” or “very good.”Source: §5 · p. 24
  • The similar results may mean normalization has only slight influence on validity, or that it does not improve peer-judgment correlations for the highest-quality papers.The authors also suggest that normalization choices, such as WoS subject categories, may sometimes distort outcomes and leave normalized indicators inferior to observed citations.Source: §5 · pp. 24–25
  • F1000 covers biomedical research rather than science’s full disciplinary diversity, and its three-level rating scale limits evaluation of indicator differences.F1000 was described as a unique exception among available datasets with peer judgments for individual papers.Source: §5 · p. 24
  • Observed citations’ good result may reflect reviewers’ access to citation counts and journal impact data, not only their reading of each paper.The cited data include short-window citation counts and the publishing journal’s JIF.Source: §5 · p. 25
  • Low reliability and disagreement among expert judgments may prevent the dataset from distinguishing indicators; future comparisons need broader disciplinary coverage.A publication can receive different ratings from different reviewers, or no recommendation from another reviewer.Source: §5 · p. 25
  • Cited-side normalization still faces unresolved choices about journal sets and field or subfield delineation, as well as other citation-impact factors beyond subject and year.Journal sets can be problematic for small fields and multidisciplinary journals, and the appropriate level of field delineation remains unsettled.Source: §5 · pp. 25–26
Loading 1410.5772v2…