Source-linked AI summary
Citation gaming induced by bibliometric evaluation: a country-level comparative analysis
Alberto Baccini, Giuseppe De Nicolao, Eugenio Petrovich
TL;DR
Research evaluation systems increasingly use quantitative indicators, but how evaluated researchers respond to them remains an important question. This paper compares Italy’s inwardness with other G10 countries after the new rules, finding a marked rise that is consistent with widespread strategic citation behavior.
Problem
The paper asks how researchers respond to indicator-based evaluation, focusing on Italy where bibliometric rules govern key career passages and strategic citation behavior is under investigation.
Source: §1 · pp. 2–3Method
Results
After 2009, Italy’s inwardness accelerated, reaching 30.73% in 2016 and ranking second in the G10; results indicate increased inwardness globally and across many fields.
Source: §6 · p. 7, §7 · p. 9, §8 · p. 13Takeaways & Limitations
The Italian case provides empirical insight into how indicator-based evaluation systems may shape researchers’ citation behavior.
Source: §8 · p. 13The authors call for further research to understand the possibility that peer-review procedures may collapse into indicator-centric approaches and advise extreme caution in using indicators in science policy.
Source: §8 · p. 14
Abstract
from arXiv · showhide
It is several years since national research evaluation systems around the globe started making use of quantitative indicators to measure the performance of researchers. Nevertheless, the effects on these systems on the behavior of the evaluated researchers are still largely unknown. We attempt to shed light on this topic by investigating how Italian researchers reacted to the introduction in 2011 of national regulations in which key passages of professional careers are governed by bibliometric indicators. A new inwardness measure, able to gauge the degree of scientific self-referentiality of a country, is defined as the proportion of citations coming from the country itself compared to the total number of citations gathered by the country. Compared to the trends of the other G10 countries in the period 2000-2016, Italy's inwardness shows a net increase after the introduction of the new evaluation rules. Indeed, globally and also for a large majority of the research fields, Italy became the European country with the highest inwardness. Possible explanations are proposed and discussed, concluding that the observed trends are strongly suggestive of a generalized strategic use of citations, both in the form of author self-citations and of citation clubs. We argue that the Italian case offers crucial insights on the constitutive effects of evaluation systems. As such, it could become a paradigmatic case in the debate about the use of indicators in science-policy contexts.
1 Introduction
The section frames quantitative indicators as tools for evaluating research performance, while noting debate over their incentives and potential to encourage strategic behavior. It motivates an empirical study of Italy, where bibliometric indicators govern key career passages, to examine whether citation strategies became widespread.
1 Introduction
- Quantitative publication and citation indicators have become part of national research evaluation systems alongside peer review.The section describes systems implemented from the late 1980s onward.Source: §1 · p. 2
- Supporters say metrics can improve research quantity and quality through incentives, while critics warn that incentives can prompt gaming such as slicing publications or citation gaming.The section also presents responsible use of metrics as a mediating position.Source: §1 · p. 2
- The literature describes constitutive effects of indicators, including goal displacement, risk avoidance, and task reduction, but their evidence had been mainly anecdotal until recent empirical work.The paper positions its Italian case study as a contribution to the empirical study of these effects.Source: §1 · p. 2
- Italy is a useful case because bibliometric rules govern key career passages, and the paper tests whether researchers widely adopted strategic citation practices to boost indicators.The authors define pervasive behavior as visible in the great majority of scientific fields at the national level.Source: §1 · p. 3
- The paper examines whether Italian researchers used citations strategically at a national scale and across most scientific fields.The remaining sections introduce the Italian context, review citation strategies, define inwardness, present results, and discuss explanations.Source: §1 · p. 3
2 The Italian case
Italy’s 2010 university reform established national research assessment and habilitation procedures that rely substantially on bibliometric indicators. These rules also influenced university recruitment and promotion, and citation indicators counted self-citations.
2 The Italian case
- The reform created ANVUR, which began the VQR research assessment in 2011 and a second assessment in 2015; submitted articles were largely evaluated using citation-indicator algorithms.Books and other outputs were evaluated by peer review.Source: §2 · p. 3
- The National Scientific Habilitation became mandatory for academic hiring and promotion, with candidates required to pass two of three ANVUR thresholds.For hard, life, and engineering sciences the indicators were article count, citation count, and h-index; social sciences and humanities used research outputs, monographs, and class A journal papers.Source: §2 · p. 3
- The first ASN thresholds were based on the median indicator values of permanent academics at the relevant rank, and successful candidates then faced a five-referee committee decision.Exceptions to the two-threshold rule were possible in specific circumstances only in the first edition.Source: §2 · pp. 3–4
- Bibliometric criteria also reached university recruitment committees, whose members had to consider metrics and themselves meet bibliometric thresholds.The section describes indicators as central to both national research assessment and recruitment procedures from 2011.Source: §2 · p. 4
- Because citation indicators included self-citations, researchers could raise their scores by citing their own work; prior studies reported increased self-citation after 2010 and an engineering-area peak in 2013.The studies covered four Italian research areas and the engineering area, respectively.Source: §2 · p. 4
3 Strategic behaviors and country self-citations
Country self-citations can capture both author self-citation and citation exchanges among researchers in the same country, including exchanges between non-coauthors. They are not inherently strategic, so unusually rising levels may be informative only in context.
3 Strategic behaviors and country self-citations
- Country self-citations include citations where citing and cited papers share at least one author country, capturing both author self-citation and same-country citation-club exchanges.Unlike author self-citation, this measure can include exchanges between researchers who are not co-authors.Source: §3 · pp. 4–5
- A citation club is an informal group that strategically exchanges citations to raise members’ citation scores.Such exchanges may not qualify as author self-citations because club members need not be co-authors.Source: §3 · p. 4
- Country self-citations can reflect legitimate internal knowledge use, not only opportunistic behavior.The passage notes that author self-citations can also be legitimate, for example when building on prior results or avoiding repetition.Source: §3 · p. 5
- International collaboration raises country self-citation counts because citations to a jointly authored paper can count for each author country.The paper therefore treats an anomalous increase relative to other countries—not the level alone—as suggestive of strategic behavior.Source: §3 · p. 5
4 The inwardness indicator
The paper defines inwardness as the percentage of a country’s citations that are country self-citations, offering a size-normalized indicator of domestic citation flows. Because collaboration also raises inwardness, the authors say its trend must be considered before attributing a rise to strategic behavior.
4 The inwardness indicator
- Inwardness is the percentage ratio of a country’s country self-citations to its total citations in a given year.It is introduced as a normalized measure of country self-citations.Source: §4 · p. 5
- The indicator ranges from 0 when a country has no self-citations to 100 when all its citations are self-citations, and it is a variant of the country Relative Citation Impact.The cited definition of Relative Citation Impact compares a country’s average citations per paper with the world average.Source: §4 · p. 5
- Inwardness is normalized for publication volume and describes how much a country’s research is cited by its own later publications rather than by other countries.The complement, 1 − Ic, indicates the share flowing to publications produced by other countries.Source: §4 · p. 6
- Strategic citation behavior may raise inwardness, but international collaboration can also raise it, so collaboration trends must be controlled before interpreting an increase.The authors associate a country-level start of strategic citation use with an anomalous rise in inwardness.Source: §4 · p. 6
5 Data
The study calculates inwardness from SCIval citation counts for G10 countries across 2000–2016, including all Scopus fields and each of 27 subject categories. It also tracks international collaboration and reports inwardness deltas.
5 Data
- SCIval supplied citation counts including and excluding self-citations; their difference yielded country self-citations, and articles, reviews, and conference papers were included.The included-with-self-citations count represented total citations.Source: §5 · p. 6
- The data covered 11 G10 countries, 2000–2016, all Scopus fields aggregated, and each of 27 Scopus Main Categories.The export also included each country's percentage of international collaboration.Source: §5 · p. 6
- The SCIval data were exported on October 16, 2018, using the Scopus update dated September 21, 2018.Source: §5 · p. 6
- A country self-citation had binary, non-fractional weight 1, even when a citing publication involved multiple countries.The same weight applied to an Italian-to-Italian citation and one involving an international Italo-French-Chinese publication.Source: §5 · p. 6
- Table 1 defines inwardness delta as the simple difference, in percentage points, between the period's last and first year.Source: Table 1 · p. 7
- International collaboration was the share of a country's publications coauthored with at least one other country.The graphs were implemented in R using ggplot2.Source: §5 · p. 7
6 Results
Italy’s inwardness accelerated after about 2010, unlike the relatively stable country rankings observed earlier. The increase was prominent across fields and was not matched by a parallel rise in international collaboration.
6 Results
- Across the countries, inwardness rose slowly and regularly, while their yearly inwardness ranks remained remarkably stable.The passage notes that rankings are partly explained by scientific production size.Source: §6 · p. 7
- Italy’s inwardness rose from 20.62% (sixth) in 2000 to 30.73% (second) in 2016; after growing parallel to the UK, Germany, and France until 2009, it accelerated around 2010, becoming first in Europe and second in G10.Source: §6 · p. 7
- 8.29 p.p. was Italy’s 2008–2016 inwardness delta, over 4 p.p. above the G10 average; its 2000–2016 delta was 10.11 p.p. versus 5.22 for the G10 average.Source: §6 · p. 7
- Inwardness rose with international collaboration across countries, but Italy ended with the highest European inwardness and lowest collaboration; Figure 2 compares inwardness with a three-year collaboration average.The average begins in the inwardness year being plotted.Source: §6 · pp. 7, 9
- Italy had the largest inwardness increase in 23 of 27 fields in 2008–2016, whereas it was generally in line with other G10 countries in 2000–2008.Source: §6 · p. 9
- At period end, Italy had both the highest inwardness and lowest international collaboration in 11 categories, and the highest inwardness but not lowest collaboration in nine others.Arts and humanities and social sciences are only partially covered by Scopus, so findings for those fields require great caution.Source: §6 · p. 9
7 Discussion
Italy’s inwardness began accelerating after 2009, around the introduction of citation-based evaluation rules, unlike the trends in the other G10 countries. The increase appeared across most fields and was not accompanied by a peculiar rise in international collaboration.
7 Discussion
- After 2009, Italy’s inwardness grew faster than in the other G10 countries, with acceleration roughly synchronous with the new assessment and habilitation rules.The paper attributes the slightly earlier observed start to citation measures’ backward effect.Source: §7 · pp. 9, 12
- Post-2008 inwardness acceleration appeared in most Italian research areas, with especially large increases in engineering, mathematics, and veterinary science.The authors find no direct evidence falsifying specialization, but consider an Italy-wide, cross-field shift implausible.Source: Figure 3 · p. 11, §7 · p. 12
- Figure 2 compares inwardness with international collaboration, and the authors report no peculiar Italian increase in collaboration that could explain the acceleration.International collaboration is represented by a three-year moving average.Source: Figure 2 · p. 10, §7 · p. 12
8 Conclusion
The authors find that Italian researchers’ citation behavior changed after the evaluation system’s introduction, with inwardness rising nationally and across most fields. They interpret the pattern as strategic citation behavior and draw policy lessons about metrics in research evaluation.
8 Conclusion
- Italian researchers’ behavior changed after the evaluation system’s introduction, with the change visible nationally and in most scientific fields.The authors attribute the pattern to strategic author self-citation and citation clubs, while noting that further research is needed to assess their respective weight.Source: §8 · p. 13
- After 2010, inwardness rose as international collaboration remained stable and comparatively low, while citations to Italian articles from non-Italian-authored papers sharply decreased.The analysis found this pattern both globally and across many research fields.Source: §8 · p. 13
- The Italian case suggests researchers respond quickly to incentive systems, and that bibliometric indicators actively shape evaluated researchers’ behavior rather than neutrally measuring performance.The authors urge careful design and implementation of policies that introduce or modify such incentives.Source: §8 · p. 13
- Strategic behaviors emerged in Italy even though researchers were evaluated against multiple indicators, as recommended by the Leiden Manifesto.The authors cite the manifesto’s principle 9, which recommends a suite of indicators instead of a single one to prevent gaming and goal displacement.Source: §8 · p. 13
- The authors warn that indicators in mixed evaluation systems may structurally foster opportunistic strategies, even when peer review also contributes to assessment.They identify a concrete risk that the indicator-based component could overtake peer review and call for further research and extreme caution in science policy.Source: §8 · pp. 13–14