PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing scientific research performance and impact with single indices

John Panaretos, Chrisovaladis Malesios

arXiv:0812.4542v3cs.IRphysics.soc-ph

TL;DR

Assessing research output and impact with one index is a limited undertaking, and the h-index itself has attracted many modifications. This paper reviews the h-index, its important modifications, and similar indicators, and recommends combining h-type indices while noting that peer review can improve assessment validity.

  • Problem

    The paper addresses how to assess research output and impact when a single h-index value is limited as an assessment tool.

    Source: §1 · pp. 1–2, §9 · p. 37

  • Method

    The authors critically review the h-index, its important literature-proposed modifications, and other indicators of research output and impact, and present extensions of some indices.

    Source: §9 · p. 37

  • Results

    The review concludes that peer reviews can significantly improve the validity of assessments based on the single h-index value.

    Source: §9 · p. 37

  • Takeaways & Limitations

    For assessing a researcher’s citation impact, the authors suggest using the h-index together with other h-type indices.

    Source: §9 · p. 37

    The validity of an assessment provided by the single h-index value can be improved by peer review.

    Source: §9 · p. 37

Abstract

from arXiv · show

We provide a comprehensive and critical review of the h-index and its most important modifications proposed in the literature, as well as of other similar indicators measuring research output and impact. Extensions of some of these indices are presented and illustrated.

1. Introduction

Accountability pressures have driven the use of rankings and single research-performance indices, even as their methodological sensitivity and coarseness draw criticism. The h-index was introduced to combine research productivity and citation impact, prompting extensive work on its modifications and limitations.

1. Introduction

  • Rankings and single-index assessments have drawn criticism because they are sensitive to methodological assumptions and reduce research value to one coarse measure.An EU-commissioned report cited in the introduction notes the sensitivity of popular world rankings to methodological assumptions.Source: §1 · p. 2
  • Traditional measures such as total papers or citations can miss research impact or be disproportionately affected by one highly influential publication.Source: §1 · p. 2
  • The h-index was proposed to assess scientific performance, and its introduction was followed by many proposed modifications and implementations.Hirsch proposed it to measure scientific output and impact, with some attention to quality, sustainability, and diversity.Source: §1 · p. 2
  • The h-index combines a scientist’s publication activity and citation impact by focusing on their most-cited papers and the citations those papers receive.A scientist has index h when h papers each have at least h citations; the papers in this h-core meet that threshold.Source: §1 · p. 3

2. Bibliographic Data Sources

Citation databases and automated tools can calculate h-indices, but their coverage, records, and author identification differ. The review recommends combining citation sources and checking records while recognizing that researcher mobility complicates affiliation-based matching.

2. Bibliographic Data Sources

  • Google Scholar and Scopus can produce higher citation counts than Web of Knowledge, while Google Scholar’s counts can include gray literature.A cited information-science study reports average increases of 160 per cent for Google Scholar and 35 per cent for Scopus compared with Web of Knowledge.Source: §2 · p. 4
  • Author-name similarities can cause papers by different researchers to enter one h-index calculation, making manual disambiguation time-consuming.The text notes that searches using only an author field cannot rule out papers by a different scientist with the same last name.Source: §2 · p. 5
  • Using multiple citation sources is described as necessary, though not sufficient, for valid h-index results; additional sources can help identify irrelevant records and duplicates.The authors also caution that searching with an author’s initials can omit valid citations when publications use only a first initial.Source: §2 · pp. 5–6
  • Affiliation-based checks are difficult to apply when researchers move between institutions or universities.Source: §2 · p. 6

3. Disadvantages of the h-index

The h-index’s simplicity leaves important aspects of citation records and research performance unrepresented. Its limitations include career-length and field differences, sensitivity to self-citation and database records, and the risk of treating a complex activity as one number.

3. Disadvantages of the h-index

  • A single metric can omit crucial information about citation records, and using it in decisions may damage careers or encourage researchers to maximize the index at the expense of quality work.The cited discussion also notes that citations can serve rhetorical purposes, including negative citations or citations added mainly to fill out an introduction.Source: §3 · pp. 6, 7, 9
  • The h-index is bounded by publication count, disadvantaging early-career researchers regardless of the importance of their discoveries.Source: §3 · p. 7
  • Self-citations can make a scientist’s work appear more widely accepted, and researchers with many collaborators may receive more of them than those publishing alone.Source: §3 · p. 7
  • The h-index can disadvantage highly cited outputs: two scientists may share h = 30, with 30 articles cited at least 30 times, yet one may have 20 cited more than 1000 times.Source: §3 · p. 8
  • Database coverage and name-matching errors can distort h-index calculations, including overestimation when records combine researchers with similar names.Source: §3 · p. 8
  • Differences in average citation numbers across fields limit the h-index’s use for comparing researchers from different disciplines.Source: §3 · p. 8

4. Some Generalizations/ Modifications of the h-index

This section reviews generalizations and modifications of the h-index that address its limitations and adapt it to differences in citation patterns, publication age, career length, research field, co-authorship, self-citations, and research topics. It describes their definitions and uses across researchers, journals, and institutions, and reports findings and limitations concerning their behavior.

4.1 h-type indices adjusting for the robustness of h-index to the number of h-core citations

  • The g-index is at least h and increases with highly skewed citation distributions, capturing changes within the Hirsch core that h misses but requiring a longer, less precise citation list [Egghe, 2006a, 2006b, 2006c].It addresses the h-index’s insensitivity to whether h-core papers receive 10 or 100 or 10000 citations.Source: §4 · p. 10
  • The A-index is the average number of citations received by articles in the Hirsch core; the R-index is described as an improvement intended to eliminate the A-index disadvantage of dividing by h.With the same A-index, it can penalize the scientist with the higher h-index because the A-index involves division by h.Source: The A-index · pp. 10–11, The R-index · p. 11
  • The hw-index uses weighted ranks in place of citation ranks to capture both the total h-core citations and their distribution [Egghe and Rousseau, 2007].The authors define discrete and continuous versions; the discrete construction is described as the more practical one.Source: The hw-index · p. 11
  • Kosmulski [2006] described the h(2)-index as an attempt to circumvent name-similarity problems; compared with two other Hirsch indices, it lacked discriminatory power.It is defined by k of Np papers having at least k^2 citations and the other (Np-h) papers having at least h^2 citations.Source: The A(2) index · pp. 11–12
  • Wu’s w-index emphasizes influence from top papers, and analysis of 20 astrophysicists found h≅4w [Wu, 2008].It is defined as w papers each having at least 10w citations.Source: The A(2) index · p. 12
  • The maxprod index selects the highest i×ci; examples suggest outstanding scientists typically reach that value at i much smaller than h, unlike typical scientists, for whom i≈h [Kosmulski, 2007].The passage also associates i much larger than h with scientists producing many lightly cited articles.Source: The i×ci index (maxprod index) · p. 12
  • Tol’s two h-index modifications resemble the g-index but use harmonic and geometric rather than arithmetic averages [Tol, 2007].They were introduced in an effort to address disadvantages of the h-index.Source: The t- and f-indices · p. 13

4. Some Generalizations/ Modifications of the h-index

  • The f- and t-indices are described as more discriminatory than h and g, yet real-data comparisons show strong correlations among all four and no significant ranking change.Source: The t- and f-indices · p. 13
  • Scientist G, with 185 citations, has the lowest values across all indicators; scientist D's g-, A-, and R-index values are 18, 34, and 18,4, respectively.Source: An illustrative example · p. 14
  • The hw-index ranks A first and F second with moderate variation, while A(2) and w lack discrimination relative to the h-, g-, A-, R-, and hw-indices.The reported hw-index values for C and G are 12.6 and 12.2, respectively.Source: An illustrative example · p. 15
  • The i×ci index ranks A first but assigns C and G the same value, 112, despite their different citation profiles.The passage attributes the tie to the index's failure to clearly discriminate the remaining outputs.Source: An illustrative example · p. 15

4.2. h-type indices for correcting for the age of publications

  • Sidiropoulos et al. [2006] define contemporary h using citation scores adjusted for article age and trend h using exponentially decaying weights on each citation.These modifications are intended to reveal significant young scientists and trendsetters, respectively.Source: Contemporary h-index, trend h-index,… · pp. 15–16
  • The normalized h-index is defined as h/√Np, correcting for publication count and favoring scientists with few but highly cited publications.It retains the h-index condition that h articles have at least h citations each and the remainder no more than h.Source: Contemporary h-index, trend h-index,… · pp. 15–16
  • The age-dependent AR-index can decrease as articles age, complementing an h-index that may keep increasing through citations even after a researcher stops producing new work.Source: The AR-index · pp. 16–17

4. Some Generalizations/ Modifications of the h-index

  • Hirsch [2005] defines m as h divided by years since a scientist’s first publication; values near 1, 2, and 3 indicate successful, outstanding, and truly unique scientists, respectively.The quotient is useful for comparing scientists with different career lengths; related discussions include Bornmann et al. [2008] and Imperial and Rodriguez-Navarro [2007].Source: The m quotient (or m parameter) · p. 17

4.3. Correcting the h-index for different fields of research

  • After field correction, h-values became more homogeneous, with the correction particularly useful for mathematics HCRs, whose h-values were considerably lower than those in other disciplines.Iglesias and Pecharromán [2007a] propose scaling h using field citation rates relative to physics.Source: §4.3 · p. 18

4.4. h-type indices for Journals

  • Braun et al. [2006] proposed h-type indices as a supplement to impact factors because h-index is robust to accidental citations and combines publication quantity with citation impact.Source: The h-index of a journal · p. 19
  • Braun et al. [2006] found that journal rankings by h-index and impact factor differed significantly, reflecting different dimensions indicated by the two indices.Their comparison calculated h-indices for 21 journals using Web of Science.Source: The h-index of a journal · p. 19
  • Rousseau [2007a] found that JASIS’s yearly h-index was influenced by its current-year article count and proposed dividing h by that count to obtain a relative h-index.Orbay et al. [2007] applied a similar relative h-index to the Turkish Journal of Chemistry.Source: The h-index of a journal · p. 20
  • Saad [2006] found significant correlations between impact factor and h-index in two datasets of business and marketing journals.Source: The h-index of a journal · p. 20

4. Some Generalizations/ Modifications of the h-index

  • Barendse [2007] proposed the strike rate index to account for journal size and found it rarely correlates with impact factor, while SR index relates linearly to article volume.The passage attributes the weak SRI–IF correlation to the general behavior of the two indices.Source: The h-index of a journal · p. 20
  • Molinari and Molinari [2008] found that journal m-curves comprise an initial straight line for few papers and a second curve for larger publication counts.They examined h-values against paper counts for three journals across countries.Source: The impact index hm for… · p. 21
  • Molinari and Molinari [2008] express h_m as h/N^β, with β approximately 0.4, and propose the impact index for comparing journals.Similar results were reported for ranking institutions.Source: The impact index hm for… · p. 21
  • Kinney [2007] found that higher h_m values were assigned to top-ranked US academic institutions.The comparison covered US institutions and science centers in physics and engineering.Source: The impact index hm for… · p. 21
  • Batista et al. [2005] define hI by dividing h by the mean number of authors in its h-core, and propose the median to avoid unfairness to scientists with few heavily co-authored papers.The mean-based correction can disadvantage scientists with a few but largely co-authored articles.Source: §4.5 · p. 22
  • The pure h-index divides a researcher’s h-index by the h-core’s average number of co-authors; the adapted version uses a larger number of articles, adjusted to observed citation data.Source: §4.5 · p. 22
  • Schreiber [2008a] argues that the fractional-counting h_m index is more robust than hI to extreme co-author counts and does not decrease as citations increase.Its effective ranking is based on each article's number of co-authors.Source: §4.5 · pp. 22–23

4.6 The self-citation issue

  • Schreiber [2007a] examines how self-citations affect h and distinguishes researchers' own citations from citations made by possible co-authors.The passage states that the h-index should in principle exclude self-citations [see, e.g., Schreiber, 2007b].Source: §4.6 · p. 23
  • Schreiber [2007b] argues that the g-index characterizes citation data better than the h-index and is more affected by self-citations.He proposes excluding self-citations from the g-index.Source: §4.6 · p. 24
  • Excluding self-citations generally does not change scientists’ rankings in Cronin and Meho’s [2006] mature-age cohort.Their data mainly covered scientists with large citation counts, supporting Schreiber’s view that self-citations have no major influence on mature researchers’ h-indices.Source: §4.6 · p. 24

4.7 The successive h-index

  • The successive h-index applies the h-index to ordered individual h-indices to summarize a university’s or institution’s research output.For theoretical modeling, Egghe [2008a] studies successive indices under a Lotkaian system, with each step multiplying the previous exponent by 1/α, where α is the Lotka exponent.Source: §4.7 · p. 24
  • Liang [2006] proposes h-index sequences and matrices to reveal career differences obscured by a single h-index, and finds that career-long h-index trends vary among 11 physicists.The study describes specific sequences but uses no statistical analysis to generalize their patterns to a broader population.Source: §4.8 · pp. 25–26

4.9 The research status index

  • Symonds et al. [2006] find men published almost 40% more papers on average and that the h-index favored female scientists less; their residual h showed no gender difference.Residual h, or research status, is the y-residual from the least-squares regression of h on publication count.Source: §4.9 · p. 26
  • The research status index can change with the addition of a few low-cited papers and appears to disregard research quantity.Source: §4.9 · p. 27

4.10 The h-b index for topics or compounds

  • Banks [2006] proposes the h-b index to measure a scientific topic’s or compound’s impact and to identify interesting or “hot” research topics.The passage notes young researchers at the start of their careers in describing the index’s purpose.Source: §4.10 · p. 27
  • The h-b index can be calculated by searching a selected scientific topic rather than a scientist’s name, helping identify topics with little interest or exhausted research.The passage frames this information as useful to PhD students advancing their work.Source: §4.10 · p. 27

5. Theoretical Approximations to the h-index

The section surveys theoretical models that relate the h-index to citation distributions, author productivity, career length, and time. It also notes that statistical inference for the theoretical h-index remains less developed than empirical applications.

Scope and examples

  • The section presents theoretical h-index models as attempts to study properties that empirical applications alone do not establish, including statistical validity and uncertainty.It illustrates the proposed methodology with citation data for D.R. Cox and P. Erdös.Source: §5 · pp. 27, 29

Distribution-based models

  • Glänzel [2006] defines a theoretical h-index using extreme-value theory under an assumed citation distribution, illustrating it with discrete Pareto and Price distributions.Schubert and Glänzel [2007] report that the theoretical Paretian model fitted their journal citation data perfectly.Source: §5 · pp. 27–28

Stochastic publication-citation model

  • Burrell [2007a] models publications with a Poisson distribution and citation rates with a gamma distribution, finding h approximately proportional to career length and linearly related to log productivity and average citation rates.These relationships are reported from simulated scenarios using varied model parameters.Source: §5 · p. 28

Information Production Process

  • Under the Information Production Process model, the unique h-index depends on the number of sources T and Lotka exponent α as h = T^(1/α) [Egghe and Rousseau, 2006].In this terminology, sources correspond to published articles and produced items to citations.Source: §5 · pp. 28–29

Dynamic h-index

  • Egghe [2008b] adds time and citation ageing rate b to the Information Production Process expression; as t approaches infinity, it reduces to the earlier Egghe and Rousseau [2006] expression.The section identifies t as time, T as number of articles, α as Lotka’s exponent, and b as citation ageing rate.Source: §5 · pp. 28–29

Statistical properties of the theoretical h-index

  • Beirlant and Einmehl [2007] establish asymptotic normality for the theoretical h-index and construct empirical h-index confidence intervals under Pareto and Weibull citation distributions.The section frames this as a response to limited work on confidence intervals, hypothesis tests, and checks of statistical properties.Source: §5 · p. 29

6. Other Developments Related to the h-index

The section reviews studies applying the h-index to prediction, performance assessment, and comparisons with other indicators. Their findings highlight both its usefulness in some settings and limits to treating it as a standalone measure.

Predicting future research

  • The h-index was slightly superior to total citations Nc for predicting future achievements, while both outperformed publication count Np.The study examined correlations between h-indices across publishing-life time windows alongside Nc, Np, and citations per paper.Source: §6 · pp. 29–30

Factor analyses of h-type measures

  • A nine-indicator factor analysis explained more than 95% of variability with two factors: h-core measures and impact of papers in the h-core [Bornmann et al., 2008].The factors were then used as independent variables in logistic regression to predict peer assessment.Source: §6 · p. 30

Limits of a single measure

  • The h-index lacks the accuracy and precision needed to serve as a complete measure of research performance, and it captures only a small portion of information about researchers’ work [Costas and Bordons, 2007].Their analysis of Spanish natural-resources scientists found the h-index, publication count, and citation count in one factor, while other factors contained relative indicators.Source: §6 · pp. 29–31

Program and discipline comparisons

  • The h-index performs better when comparing graduate than undergraduate research programs, but a non-normalized index is inappropriate across disciplines and databases may incompletely cover some fields.The cited departmental evaluation specifically notes psychology as an example of inadequate database coverage.Source: §6 · p. 31

Missing publications

  • Under a theoretical Zipf-distribution model, the h-index is generally unaffected by small numbers of missing highly cited articles, especially relative to missing publications [Rousseau, 2007].The finding concerns the modelled effect of missing records on h-index calculation.Source: §6 · p. 31

Relations to other indicators

  • A study of 147 university chemistry research groups in the Netherlands found that citation counts were proportional to the square of the h-index.The reported relationship is Ncitations = αh^2, with α for physics ranging between 3 and 5.Source: §6 · pp. 31–32

Factors associated with h-index

  • After controlling for scientific age, females were found to have lower h-index values than male scientists, while h-index also correlated with scientific age [Kelly and Jennions, 2006].The authors offer several possible explanations for the gender difference, including paper citability and publication volume.Source: §6 · p. 32

7. A New Classification for Index Comparisons

The section proposes considering the spread of a researcher’s citations across papers alongside publication and citation totals. An artificial comparison of eight publication records shows that h-type indices can rank the same researchers differently.

Three-factor classification

  • The proposed classification adds citation spread across a researcher’s papers to publication and citation totals: widely distributed citations favor a larger index than citations concentrated in a few papers.The example contrasts many articles with substantial citations spread broadly against a small number of highly cited papers.Source: §7 · pp. 32–33

Index comparisons

  • The g-index favors authors ACE and ACF, who combine many articles with many citations, whereas the A-index favors BCE and ACE, whose citations are concentrated in the h-core.The comparison uses four indices—h, g, A, and R—on the constructed publication records of eight authors.Source: §7 · p. 34
  • The h-index ranks ACF and BCF highest among the example authors with many widely distributed citations, regardless of whether they have 20 or 10 articles.The example also shows h-index values of 4 for BDE and 3 for ADF, although ADF has twice as many articles.Source: §7 · p. 34

A-index limitation

  • For equal h-core citation totals, the A-index assigns a higher value to the researcher with the smaller h-index, a consequence of dividing h-core citations by h.The section identifies this as a disadvantage of the A-index.Source: §7 · p. 34

Index comparisons

  • The R-index places BCE and BCF highest in the example, favoring authors with fewer publications but many citations.It is presented as an alternative to the A-index.Source: §7 · p. 34

8. Some Modifications

The section introduces two modifications to existing indices. Artificial examples show that the R-index can still undervalue dispersed citations, while Rm changes rankings and a CV adjustment penalizes citation variability.

8. Some Modifications

  • Rm modifies the R-index by taking the square root of the sum of square roots of citations in the h-core.The resulting ranking is shown for the example's eight scientists in Table 5.Source: §8 · p. 35, Table 5 · p. 35
  • In the example, R favors ACE (13.5) over ACF (11.4), despite ACE's citations being concentrated in a few articles.The authors say this fails to correct the A-index's disadvantage.Source: §8 · p. 35
  • For BCE versus BCF, R is nearly tied (14 for BCE; 14.1 for BCF) and gives no significant advantage to the more dispersed profile.BCF has more scattered citations, but the R-index difference remains small.Source: §8 · p. 35
  • Rm slightly improves rankings, placing BCF and ACF at the top and ADF and BDF at the bottom.The ranking result is presented for the eight scientists in Table 5.Source: Table 5 · p. 35, §8 · p. 36
  • Rm also distinguishes ACE from ACF and BCE from BCF, giving higher values to scientists with more widely accepted work.The index values differ for both pairs.Source: §8 · p. 36
  • To account for h-core citation variability, the authors subtract its coefficient of variation (CV) from the previously presented index.They choose CV because dividing by the mean removes magnitude as a factor.Source: §8 · p. 36
  • The CV penalty makes differences more visible: ACE's concentrated h-core has higher variation than ACF's more scattered citations.Table 6 presents the Rm-cv results for the eight researchers.Source: §8 · p. 36

9. Conclusions

The article critically reviews the h-index and related indicators, and presents h-index modifications for special circumstances. It recommends combining indicators with other evidence while taking citation and co-authorship issues into account.

9. Conclusions

  • The article presents an extensive critical review of h-index modifications and other indicators, including modifications intended to improve performance in special circumstances.The review covers literature on the h-index, its important modifications, and other research-output indicators.Source: §9 · p. 37
  • The authors recommend using h with hw, g, R, and A-index values to identify significant variations in h-core outputs.They suggest combined use for more representative results.Source: §9 · p. 37
  • Combining h-index measures with publication and citation totals or peer review can improve assessment validity over relying on the h-index alone.The authors also say A(2), w, and i×ci indices can provide extra insight.Source: §9 · p. 37
  • Excluding self-citations, especially in small citation sets, is recommended to improve assessment accuracy and fairness.Source: §9 · p. 37
  • Co-authorship effects should be considered, particularly in disciplines such as medicine where multiple co-authorship is common.The authors suggest using h-type indices presented in the article to correct for co-authorship.Source: §9 · p. 37
  • Citation data from different sources should be treated cautiously, while comparing their citation outputs may help produce more credible results.Source: §9 · p. 37
Loading 0812.4542v3…