Source-linked AI summary
The inconsistency of the h-index
Ludo Waltman, Nees Jan van Eck
TL;DR
The paper addresses whether the h-index appropriately measures overall scientific impact, given concerns about unsystematic indicator development and limited theoretical analysis. It uses a theoretical critique to show inconsistent rankings and discusses scoring rules, especially highly cited publications, as alternatives. The authors conclude that the h-index is not appropriate for this purpose, while the highly cited publications indicator avoids the same inconsistency problem.
Problem
Existing h-index research is often unsystematic and empirically oriented, leaving a fundamental theoretical problem in measuring overall scientific impact insufficiently recognized.
Method
The paper theoretically analyzes how the h-index aggregates publication and citation statistics and discusses scoring rules as alternative indicators.
Results
The h-index can produce inconsistent scientist rankings and therefore cannot be considered an appropriate indicator of overall scientific impact.
Takeaways & Limitations
Scoring rules provide a broad family of theoretically grounded alternatives, including a highly cited publications indicator that does not produce inconsistent rankings.
Takeaways & Limitations
Some non-scoring-rule indicators can also avoid inconsistency, but they are fairly complicated and violate Marchant’s Archimedeanness axiom, limiting their practical usefulness.
Abstract
from arXiv · showhide
The h-index is a popular bibliometric indicator for assessing individual scientists. We criticize the h-index from a theoretical point of view. We argue that for the purpose of measuring the overall scientific impact of a scientist (or some other unit of analysis) the h-index behaves in a counterintuitive way. In certain cases, the mechanism used by the h-index to aggregate publication and citation statistics into a single number leads to inconsistencies in the way in which scientists are ranked. Our conclusion is that the h-index cannot be considered an appropriate indicator of a scientist's overall scientific impact. Based on recent theoretical insights, we discuss what kind of indicators can be used as an alternative to the h-index. We pay special attention to the highly cited publications indicator. This indicator has a lot in common with the h-index, but unlike the h-index it does not produce inconsistent rankings.
1. Introduction
The paper critiques the h-index and its variants theoretically, arguing that their aggregation mechanism can rank scientists inconsistently when measuring overall scientific impact. It discusses theoretically well-founded alternatives, especially the highly cited publications indicator, without endorsing a single replacement.
- 1. Introduction: H-index research often develops new indicators ad hoc by addressing isolated undesirable properties without systematically comparing overall indicator properties.The authors characterize the literature as unsystematic and strongly empirically oriented.
- 1. Introduction: The paper argues theoretically that the h-index can behave counterintuitively and produce inconsistent rankings of scientists.The claimed problem concerns how publication and citation statistics are aggregated into one number.
- 1. Introduction: The authors conclude that the h-index is inappropriate for measuring a scientist’s overall scientific impact.The conclusion applies to scientists and, more generally, other units of analysis.
- 1. Introduction: The paper discusses a broad family of theoretically well-founded indicators that avoid the h-index’s fundamental inconsistency problem.These alternatives are presented without selecting one universally preferred indicator.
- 1. Introduction: The highly cited publications indicator resembles the h-index but does not produce inconsistent rankings.The paper gives this indicator special attention among the proposed alternatives.
2. Definition of the h-index
The h-index counts how many publications meet a matching citation threshold, using the intersection of a sorted citation curve with a 45-degree line. Its appeal includes robustness to extreme citation counts, although the threshold construction is presented as arbitrary.
- 2. Definition of the h-index: A scientist has h-index h when h publications each have at least h citations and every remaining publication has fewer than h + 1 citations.The definition also applies to research groups and journals, not only individual scientists.
- 2. Definition of the h-index: The h-index is calculated by sorting publications by decreasing citations and locating the intersection of the citation curve with a 45-degree line.The corresponding publication count gives the index; the illustrated example has an h-index of six.
- 2. Definition of the h-index: The h-index avoids the total-citation indicator’s sensitivity to one or a few highly cited publications.This robustness is presented as an advantage over simply counting total citations.
- 2. Definition of the h-index: The h-index can also replace highly cited-publication counts, whose citation threshold is considered arbitrary and potentially favorable or unfavorable to individuals.Hirsch’s argument is that the h-index avoids dependence on a parameter with an arbitrary value.
- 2. Definition of the h-index: The choice of a 45-degree line is arbitrary because alternative lines, such as 30 or 60 degrees, could be used in the graphical construction.Figure 3 illustrates this arbitrariness directly.
3. Inconsistency of the h-index
The paper argues theoretically that the h-index can produce counterintuitive and inconsistent rankings when used to measure overall scientific impact. Three examples show problems involving equal relative improvements, equal absolute improvements, and aggregation from scientists to research groups.
- Example 1: Equal relative performance improvements can reverse the h-index ranking of two scientists.Scientists X and Y both doubled their publications and citations, yet their ranking changed.
- Example 2: Equal absolute performance improvements can also reverse the ranking of two scientists under the h-index.In the example, both scientists received two publications with six citations each, but their ranking reversed.
- Example 3: The h-index violates consistency between individual and group rankings.The paper shows that two scientists can each outrank their counterparts while their combined research group does not outrank the other group.
- The h-index aggregates publication and citation statistics in ways that can produce inconsistent results when measuring overall impact.The authors conclude that it is therefore inappropriate as an indicator of the overall scientific impact of a set of publications.
- Discussion: The inconsistency problem extends to h-index variants, extensions, and some unrelated indicators.The authors state that resolving it requires a different class of indicators rather than merely modifying the h-index framework.
4. Alternatives to the h-index
The paper considers scoring rules as a broad class of alternatives to the h-index because they avoid inconsistency problems. It focuses especially on the highly cited publications indicator, which shares several practical features with the h-index while avoiding its inconsistency.
- Scoring rules: Scoring rules aggregate publication-level citation scores, with each score determined by citations and increasing when citations increase.For citation counts c1, c2, …, cn, the rule sums f(c1) + f(c2) + … + f(cn), where f is increasing.
- Scoring rules: Scoring rules do not suffer from inconsistency problems, while almost all indicators outside this class do.The paper therefore restricts its search for h-index alternatives largely to scoring rules.
- Highly cited publications indicator: The highly cited publications indicator counts publications meeting a citation threshold, assigning equal positive scores to qualifying publications and zero to the rest.It is therefore insensitive to the exact citation counts of highly cited publications.
- Highly cited publications indicator: The highly cited publications indicator resembles the h-index through simple calculation and insensitivity to highly cited outliers and many lowly cited or uncited publications.The paper presents it as a scoring rule with similar practical behavior to the h-index.
- Highly cited publications indicator: The highly cited publications indicator is regarded as more appropriate than the h-index because it avoids inconsistency problems.The paper does not designate a single scoring rule as universally best.
- Comparison with the h-index: Arbitrariness does not distinguish the two indicators: both depend on arbitrary elements, although only the highly cited publications indicator has an explicit parameter.The paper rejects arbitrariness as a reason to prefer one indicator over the other.
5. Discussion and conclusion
The paper concludes that the h-index can produce inconsistent scientist rankings and is therefore inappropriate for measuring overall scientific impact. It presents scoring rules—especially the highly cited publications indicator—as theoretically consistent alternatives, while noting that empirical comparisons should include robust alternatives and that multiple indicators or citation distributions may be preferable.
- Conclusion: The h-index cannot be considered an appropriate indicator of overall scientific impact because its aggregation mechanism can produce inconsistent scientist rankings.This conclusion concerns cases in which publication and citation statistics are combined into a single h-index value.
- Alternatives: Scoring rules form a large family of bibliometric indicators that, unlike the h-index, do not suffer from inconsistency problems.The paper discusses this family rather than endorsing one universal replacement.
- Alternatives: The highly cited publications indicator combines robustness to extreme citation counts with the advantage of avoiding the h-index's inconsistency problems.The authors regard it as an attractive alternative when robustness to very highly cited and barely cited publications is desirable.
- Caveats: The h-index may have advantages for credit allocation in co-authored high-impact publications, but this argument depends on giving most credit to senior authors.The authors question whether that advantage justifies sacrificing consistency of measurement.
- Empirical evidence: Existing empirical support for the h-index is limited in scope, and future studies should compare it with robust scoring-rule indicators.Hirsch's study supported the h-index for applications in physics but did not include the highly cited publications indicator or other robust alternatives.
- Practical implications: Practical evaluation is usually better served by a set of indicators, or sometimes by examining citation distributions directly, than by relying on one number.The authors state that citation distributions provide the most comprehensive picture available from publication and citation data.