Source-linked AI summary
The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000
Lutz Bornmann, Loet Leydesdorff
TL;DR
The study addresses limited comparisons between citation metrics and independent expert assessments of quality. It compares F1000 peer ratings with InCites bibliometric indicators and finds the strongest correlation for Percentile in Subject Area, while three other indicators show at least medium effect sizes.
Problem
Prior studies had not compared different citation-based metrics with an independent expert-based quality indicator, and citation counts vary in expected rates across fields.
Source: §1 · pp. 3–4Method
The study compares F1000 peer ratings with InCites citation indicators, including category-normalized citations and a reversed Percentile in Subject Area measure.
Source: §2.2 · pp. 7–8, §4 · p. 11Results
Percentile in Subject Area has the highest correlation with FFa; Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations each have a true correlation reaching at least a medium effect size.
Source: §4 · p. 12Takeaways & Limitations
Among the normalized indicators considered, Percentile in Subject Area and Category Actual/Expected Citations should be preferred in research assessment.
Source: §4 · p. 12The study covers only two subject categories and German papers, so more comprehensive studies are needed.
Source: §4 · p. 12
Abstract
from arXiv · showhide
The data of F1000 provide us with the unique opportunity to investigate the relationship between peers' ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the seven indicators offered by InCites, we analyzed the validity of raw citation counts (Times Cited, 2nd Generation Citations, and 2nd Generation Citations per Citing Document), normalized indicators (Journal Actual/Expected Citations, Category Actual/Expected Citations, and Percentile in Subject Area), and a journal based indicator (Journal Impact Factor). The data set consists of 125 papers published in 2008 and belonging to the subject category cell biology or immunology. As the results show, Percentile in Subject Area achieves the highest correlation with F1000 ratings; we can assert that for further three other indicators (Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations) the 'true' correlation with the ratings reaches at least a medium effect size.
1 Introduction
Prior work had not compared different citation-based metrics with an independent expert-based quality indicator, leaving the validity of advanced indicators unclear. This study compares citation metrics with F1000 peer ratings and treats the highest-correlating metric as preferable for research evaluation.
1 Introduction
- Different citation-based metrics had not been compared with an independent, expert-based indicator of research quality.Although peer evaluation had been used to validate metrics, earlier metric comparisons did not answer which citation-normalization technique should be preferred.Source: §1 · pp. 3–4
- Earlier studies either used simple, non-field-normalized metrics or focused on single journals, leaving a need to test advanced indicators on a larger scale.Field differences in expected citation rates mean raw citation counts cannot be directly compared across fields or subfields.Source: §1 · pp. 3–4
- The study calculates Spearman rank-order correlations between field-normalized citation impact values and F1000 ratings for published papers.F1000 researchers review and rate biomedical papers in their specialist areas.Source: §1 · p. 4
- The authors propose preferring the bibliometric metric with the highest correlation coefficient in research evaluation.They cite F1000 as a potential complement to alternative metrics at paper, individual, and journal levels.Source: §1 · p. 5
2 Methods
The study uses F1000 peer ratings to assess seven InCites indicators for papers in two biomedical subject categories. It correlates the ratings and metrics using Spearman rank-order coefficients and reports confidence intervals while accounting for the dataset's sampling characteristics.
2.1 Peer ratings provided by F1000
- F1000 ratings come from a post-publication review system in which a peer-nominated Faculty of experts selects biomedical papers, rates them, and explains their importance.The system covers only a restricted set of papers, and most papers in the biomedical journals it covers are not reviewed.Source: §2.1 · p. 5
2.2 Data set for the study
- For the 5,204 InCites papers published in 2008 across cell biology (2,657) and immunology (2,547), FFa could be retrieved for 125 papers (2.4%); downloaded papers had at least one German address.Source: §2.2 · p. 6
- Percentile in Subject Area values were reversed as 100 minus the InCites percentile because smaller original values indicate higher citation impact.For journals assigned to multiple subject areas, the percentile uses the area in which the paper performs best, meaning the lowest value.Source: §2.2 · p. 8
2.3 Statistical procedures
- Because the variables were not approximately normal, the authors used Spearman rank-order correlations and calculated 95% confidence intervals for each coefficient.They used a Bonferroni adjustment to address multiple comparisons between FFa and several metrics.Source: §2.3 · pp. 8–9
- The authors calculated unadjusted correlation coefficients despite ties, citing software constraints and evidence that adjusted and unadjusted values differ very little.Because the papers were not randomly sampled, significance tests address whether relationships could arise from a random data-generating process, not population generalization.Source: §2.3 · pp. 8–9
3 Results
Figure 1 compares the correlations between F1000 ratings and seven bibliometric metrics, with Percentile in Subject Area highest and Journal Actual/Expected Citations lowest. Percentile in Subject Area accounts for 20% of rating variance, approaching the stated 33% theoretical maximum for citation impact alone.
3 Results
- Percentile in Subject Area has the highest correlation with F1000 ratings, while Journal Actual/Expected Citations has the lowest.Figure 1 reports Spearman rank-order correlation coefficients with 95% confidence intervals.Source: §3 · p. 9
- The variance accounted for does not establish a direction of causality between citation metrics and F1000 ratings.The paper notes that attribution of causality is unclear and treats citations as measuring impact, while peers may assess importance and accuracy as well.Source: §3 · p. 10
- Journal Actual/Expected Citations explains 1% of F1000 rating variance, compared with 20% for Percentile in Subject Area.The paper notes a theoretical maximum of 33% (r2=33% or r=.57) if citations measure only the impact aspect of peer-assessed quality.Source: §3 · p. 10
- The relationship between Percentile in Subject Area and F1000 ratings has at least a medium and at best a large effect size.The paper frames the 33% variance figure as a maximum and says Percentile in Subject Area's r2=20% comes close to it.Source: §3 · p. 10
4 Discussion
Citation indicators capture research impact, while peer assessments can also evaluate accuracy and importance. In this F1000 comparison, Percentile in Subject Area correlated most strongly with ratings, and three other indicators had true correlations of at least medium effect size.
4 Discussion
- The study compared F1000 ratings with seven InCites indicators spanning raw citation counts, normalized indicators, and Journal Impact Factor.F1000 ratings supplied the peer-assessment measure for this comparison.Source: §4 · p. 11
- Percentile in Subject Area had the highest correlation with FFa; Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations had true correlations of at least medium effect size.The authors report this as the study’s principal comparison.Source: §4 · p. 12
- The authors attribute relatively high correlations for raw-count indicators partly to selecting papers from cell biology and immunology, which have above-world citation impact.InCites impact relative to World is reported as 2.3 for cell biology and 1.87 for immunology.Source: §4 · p. 12
- Among normalized indicators, the authors recommend Percentile in Subject Area and Category Actual/Expected Citations, but not Journal Actual/Expected Citations, for bibliometric assessment.They call for more comprehensive future studies given the study’s limited database and scope.Source: §4 · p. 12
- The findings are constrained by a small F1000 sample, two subject categories, and indicators restricted to German papers; the authors call for more comprehensive studies.Only a small fraction of papers in cell biology and immunology could be found in F1000.Source: §4 · p. 12