PaperlayerView exact revision on arXiv

Source-linked AI summary

Field-normalized citation impact indicators and the choice of an appropriate counting method

Ludo Waltman, Nees Jan van Eck

arXiv:1501.04431v1cs.DL

TL;DR

Bibliometric comparisons across fields require field-normalized citation indicators, but the paper examines whether publication counting methods support that goal. Through theoretical and extensive empirical analysis, it compares full and fractional counting and concludes that full counting cannot yield properly field-normalized results, whereas fractional counting can. It consequently recommends fractional counting, especially for country- and organization-level studies, and generally favors author-level or address-level variants.

  • Problem

    Field-normalized citation indicators require counting methods that handle co-authored publications appropriately for valid comparisons between scientific fields.

  • Method

    The paper develops a theoretical argument and conducts an extensive empirical analysis comparing full counting with fractional counting and its variants.

  • Results

    Full counting cannot produce properly field-normalized results, whereas fractional counting does provide properly field-normalized results.

  • Takeaways & Limitations

    Bibliometric studies requiring field normalization should use fractional counting, especially at country and research-organization levels, generally favoring author-level or address-level variants.

  • Takeaways & Limitations

    Full counting results carry a serious risk of misinterpretation.

Abstract

from arXiv · show

Bibliometric studies often rely on field-normalized citation impact indicators in order to make comparisons between scientific fields. We discuss the connection between field normalization and the choice of a counting method for handling publications with multiple co-authors. Our focus is on the choice between full counting and fractional counting. Based on an extensive theoretical and empirical analysis, we argue that properly field-normalized results cannot be obtained when full counting is used. Fractional counting does provide results that are properly field normalized. We therefore recommend the use of fractional counting in bibliometric studies that require field normalization, especially in studies at the level of countries and research organizations. We also compare different variants of fractional counting. In general, it seems best to use either the author-level or the address-level variant of fractional counting.

1. Introduction

Field normalization and counting methods are closely connected because valid between-field citation comparisons depend on how co-authored publications are counted. The paper argues that full counting cannot produce properly field-normalized results, whereas fractional counting can.

  • Field normalization aims to correct citation-practice differences so citation-based indicators permit valid comparisons between scientific fields.
  • Counting methods determine how co-authored publications are allocated, such as fully to each country or fractionally among countries.
  • Proper field normalization requires a suitable counting method, linking two topics usually discussed separately.
  • Full counting assigns co-authored publications fully to every co-author, so publications are counted multiple times.
  • This multiple counting biases results toward fields with extensive co-authorship when co-authorship correlates with additional citations.
  • The paper develops the argument through theoretical discussion, an extensive empirical analysis, comparisons of counting methods, and responses to arguments favoring full counting.

2. Counting methods

The paper compares full, fractional, first-author, and corresponding-author counting, illustrating how publication weights differ across author, organization, and country analyses. Fractional counting variants distribute a publication's total weight across co-authors or affiliations, with author-level and address-level approaches receiving particular support.

  • Counting methods: Full counting gives every co-author a full publication, while fractional counting assigns shares whose weights sum to one.
  • Counting methods: A publication co-authored by four countries receives weight 1 / 4 = 0.25 for each country under fractional counting.
  • Counting methods: Author-level fractional counting gives each author equal weight, whereas first-author and corresponding-author counting assign weight one to one author and zero to the others.
  • Counting variants: Address-level, organization-level, and country-level fractional counting give equal weight to listed addresses, organizations, or countries, respectively.
  • Author analysis: For a five-author example, full counting assigns weight one to each author, while author-level fractional counting assigns 1 / 5 = 0.20 to each.
  • Organization analysis: For organizations, address-level counting can weight an organization by the number of its listed addresses, while organization-level counting gives each organization equal weight.
  • Choosing variants: When an author is affiliated with two organizations, author-level fractional counting may better reflect contributions than organization-level fractional counting.
  • Choosing variants: The paper argues that author-level or address-level fractional counting is generally preferable to organization-level or country-level variants.

3. Relation between counting methods and field normalization

The paper demonstrates that counting methods are closely connected to field normalization: full counting can distort cross-field comparisons, whereas fractional counting satisfies strong field normalization.

  • 3.1. Example involving a single field: In a single-field example, full counting places both countries above the world average, whereas fractional counting distinguishes one above-average and one below-average country.The full-counting conclusion would imply that every country performs above average, while fractional counting gives country A above-average and country B below-average performance.
  • 3.1. Example involving a single field: Fractional counting makes the weighted average of countries’ MNCSs exactly one, a general property that full counting does not guarantee.The weights are given by each country’s fractional number of publications.
  • Conclusions based on the examples: Full counting can yield a world average of 1.20 rather than one, and such deviations have serious consequences when comparing fields.In the example, interpreting 1.20 as the world average restores the conclusion that country A is above average and country B below average.
  • Motivation: Full counting is fundamentally inconsistent with field normalization because co-authored publications are counted once for each co-author.This creates a bias favoring fields with extensive co-authorship and citation advantages associated with co-authorship.
  • Conclusions based on the examples: Fractional counting permits valid between-field comparisons because its results are compatible with strong field normalization.The paper presents this conclusion for country-level comparisons and notes that the underlying examples generalize to authors, organizations, and other fractional-counting variants.

4. Empirical analysis of the full counting bonus

The empirical analysis shows that the full counting bonus varies with co-authorship patterns, citation relationships, and scientific field. It is positive and substantial overall, especially for author- and organization-level analyses, creating systematic differences across fields and analysis levels.

  • Determinants of the bonus: The full counting bonus depends on variation in the number of authors, organizations, or countries and on how those counts relate to citation scores.If all publications have the same number of contributors, or contributor counts are unrelated to citation scores, the corresponding bonus can disappear.
  • Empirical basis: Publications with more authors, organizations, or countries generally have higher citation scores, with the strongest relationship at the country level.For publications with two to five authors, however, citation scores show little dependence on author count, and three- or four-author publications score slightly below two-author publications.
  • Results by analysis level: At author, organization, and country levels, the MNCS full counting bonus is positive and significant, generally highest for authors and lowest for countries.The organization-level bonus is generally intermediate, although it exceeds the author-level bonus in Biomedical and health sciences and Life and earth sciences.
  • Results by indicator: The PPtop 10% full counting bonuses resemble the MNCS results but are consistently higher.This pattern is reported across the corresponding analysis levels and fields.
  • Interpretation over time: For countries, rising full-counting MNCS values do not necessarily indicate improved performance relative to other countries.Increasing international co-authorship and the high citation impact of internationally co-authored publications produce an overall upward trend in countries’ full-counting MNCSs.

5. Empirical comparison of counting methods

The empirical comparisons show that full and fractional counting often produce correlated results but can differ materially for universities and countries. Fractional counting lowers impact indicators in most comparisons and changes some rankings, particularly where international collaboration and high-bonus fields are concentrated.

  • Comparison design: The comparisons examine full versus address-level fractional counting for organizations and seven counting methods for countries.The organization comparison uses 500 universities, while the country comparison analyzes 25 countries.
  • Organizations: For almost all universities, fractional-counting PPtop 10% is lower than full-counting PPtop 10%, by almost 2 percentage points on average.The two indicators nevertheless show a strong overall correlation.
  • Organizations: Differences between university indicators are larger for institutions with a greater share of Biomedical and health sciences publications.Universities ranked by decreasing field share show a corresponding decrease in the average gap between full and fractional counting.
  • Countries: For all 25 countries, address-level fractional counting yields a lower MNCS than full counting.The decrease ranges from 9% for Turkey to 42% for Switzerland and Scotland.
  • Countries: Country rankings can change with the counting method: the United States moves from eighth with MNCS 1.34 under full counting to fourth with MNCS 1.30 under address-level fractional counting.The country-level MNCS difference is especially sensitive for the United States.
  • Countries: Full counting tends to benefit countries strongly involved in international collaboration, whereas fractional counting benefits countries less involved in international collaboration.Countries with larger publication-output decreases also tend to show larger MNCS decreases when moving to fractional counting.
  • Overall comparison: Country-level results are relatively insensitive to the choice among alternatives to full counting and to the choice between MNCS and PPtop 10%.The analysis states that results become more sensitive to these choices at other analysis levels.

6. Commonly used arguments in favor of full counting

The paper evaluates common arguments for full counting, including its intuitive treatment of collaboration and differing interpretations of participation versus contribution. It concludes that fractional counting better supports field-normalized citation impact comparisons, while collaboration should be measured with separate indicators.

  • Scope of the methods: The paper acknowledges that equal weighting of co-authors is arbitrary and that fractional counting may fail to represent unequal individual contributions.Both methods are considered valid for different analytical purposes, depending on what the analysis intends to measure.
  • Scope of approximation: For large publication sets, the authors assume that overestimation and underestimation of organizational or national contributions tend to cancel out.They regard the resulting error as likely to remain within an acceptable margin for organizations or countries, but not necessarily for individual publications.
  • Collaboration: The authors argue that full counting gives collaborative publications an unfair advantage, especially in fields with a significant full counting bonus.Fractional counting is presented as neutral toward collaboration, while full assignment to every co-author reinforces the advantage.
  • Measurement principle: Citation impact and collaboration are treated as separate dimensions of scientific performance that should generally be measured with separate indicators.The paper favors fractional counting for accurate citation-impact measurement and additional indicators for collaboration.
  • Intuitiveness: Fractional counting gives non-integer publication and citation counts, making it less intuitive in some individual-researcher contexts.A researcher may regard co-authored and single-author publications as similarly important, although fractional counting assigns less weight to co-authored work.
  • Interpretability: Full counting is widely used, but the paper argues that it can produce field-normalized indicators that are difficult to interpret correctly.Fractional-counting indicators are described as compatible with strong field normalization and easier to interpret for cross-field comparisons.
  • Participation versus contribution: The paper distinguishes participation from contribution: full counting measures publications in which an entity participates, whereas fractional counting measures its partial contribution.This distinction applies to publication output and citation impact at the country level.

7. Conclusions

The paper connects counting methods to field normalization and concludes that fractional counting is preferable to full counting, particularly for highly aggregated analyses. It also recommends author-level or address-level fractional counting variants.

  • The paper presents field normalization as closely connected to the choice of counting method for citation-based indicators.
  • Full counting cannot produce properly field-normalized results and can make field-normalized indicators easy to misinterpret.
  • Co-authored publications are counted multiple times under full counting, advantaging fields with extensive co-authorship and disadvantaging others.
  • Fractional counting counts each publication once regardless of its number of co-authors, enabling unbiased comparisons between fields.
  • Practical implications: At the country or organization level, the authors consider fractional counting essential because full-counting results carry a serious risk of misinterpretation.
  • Practical implications: At low aggregation levels, field-normalized indicators may be insufficiently accurate, reducing the relevance of the paper’s fractional-counting argument.
  • Practical implications: Country-level differences among fractional-counting variants are relatively small compared with differences between full and fractional counting.
  • Practical implications: The authors generally recommend either author-level or address-level fractional counting; first-author and corresponding-author counting also yield properly field-normalized results.

Appendix

The appendix documents the datasets, counting-method abbreviations, assumptions, and tables used for the empirical results. It also records data-availability constraints affecting some counting methods.

  • The appendix reports detailed analysis results in Tables A1–A3 and provides results for 150 countries in an Excel file.
  • The appendix defines seven methods, including full counting, author-level and address-level fractional counting, and first- and corresponding-author counting.
  • The Web of Science dataset contains 2.46 million articles and reviews from 2009–2010, but 1.9% lack address information.
  • Only 2.41 million publications can be assigned to one or more countries because some publications lack address information.
  • Corresponding-author counting uses the reprint address, while other methods use regular addresses when available.
  • When author–address links are unavailable, author-level fractional and first-author counting cannot be implemented, so each author is assumed affiliated with every address.
  • Tables A1, A2, and A3 present country publication counts, MNCS values, and PPtop 10% values, respectively.
Loading 1501.04431v1…