Source-linked AI summary
The field-standardized average impact of national research systems compared to world average: the case of Italy
Giovanni Abramo, Ciriaco Andrea D'Angelo, Fulvio Viel
TL;DR
International research comparisons matter, but aggregate comparisons and rankings can be distorted when they do not account for field differences. This study analyzes Italian public research organizations’ hard-science publications from 2001–2006 using field-standardized citation impact. Italian publications received 12% more citations than the world average, with the difference reaching a peak of +30% in the final year examined.
Problem
International research comparisons are needed, but aggregate comparisons and organization rankings can be distorted by field differences and size-dependency.
Source: §1 · pp. 2–3Method
Results
12% more citations than the world average, rising to a peak of +30% in the final year examined, for Italian publications.
Source: §4 · p. 14Takeaways & Limitations
The methodology can support policy and institutional analysis of strengths and weak fields and is presented as essentially replicable in other national contexts.
Source: §4 · p. 14Average impact per publication does not establish productivity; policy evaluation should also consider labor and capital inputs and other organizational responsibilities.
Source: §1 · p. 4, §4 · p. 14
Abstract
from arXiv · showhide
The study presents a time-series analysis of field-standardized average impact of Italian research compared to the world average. The approach is purely bibliometric, based on census of the full scientific production from all Italian public research organizations active in 2001-2006 (hard sciences only). The analysis is conducted both at sectorial level (aggregated, by scientific discipline and for single fields within disciplines) and at organizational level (by type of organization and for single organizations). The essence of the methodology should be replicable in all other national contexts. Its offers support to policy-makers and administrators for strategic analysis aimed at identifying strengths and weaknesses of national research systems and institutions.
1. Introduction
The introduction argues that international research comparisons are valuable but often lack scientific rigor when they ignore differences among fields and organizations. The study instead compares Italy’s field-standardized average research impact with the world average, across sectors and organizations.
1. Introduction
- International comparisons can help identify research strengths, weaknesses, centers of excellence, and opportunities, but producing them rigorously is exceptionally complex.The study notes that comparisons are often traded off against scientific rigor.Source: §1 · p. 2
- Aggregate output-to-input measures and organization rankings can distort performance comparisons when they omit field differences and, for rankings, size-dependency.The ARWU, for example, derives over 90% of its performance result from university size.Source: §1 · pp. 2–3
- The authors present their analysis as a modest, accurate comparison of average impact, not a measure of research productivity.Above-world-average impact per publication does not establish greater total impact per researcher when publication counts per researcher are unknown.Source: §1 · p. 4
- Only two extensive national time-series analyses of field-standardized average impact were identified, alongside Dutch methodological studies validated on limited fields.The Dutch studies covered medicine, natural sciences and mathematics at Leiden, and chemistry-related publications from ten universities.Source: §1 · p. 4
- The study compares field-standardized average impact of Italian hard-science publications with the world average across 345 U&PROs active during 2001–2006.Analysis spans aggregate, inter-temporal, disciplinary, field, single-organization, and organization-type levels.Source: §1 · p. 4
- The findings are intended to inform policy and institutional benchmarking, and the methodology is presented as essentially replicable in other national contexts.Inter-temporal analysis may help examine the effectiveness of interventions.Source: §1 · p. 4
2. Data and method
The study uses Web of Science publications and citation benchmarks to measure Italian hard-science research impact against field-specific world averages. It aggregates publication-level standardized impact across years, fields, disciplines, and organizations.
2. Data and method
- The dataset covers over 250,000 Italian U&PRO publications in the hard sciences indexed in Web of Science from 2001 to 2006.It includes articles, review articles, and conference proceedings from 78 universities, 75 research institutions, and 192 hospitals and health care research organizations.Source: §2 · p. 5
- Expected Citation Rate (XCR) benchmarks citations against world publications in the same year and field, while Journal Expected Citation Rate (JXCR) benchmarks publications against the same journal and year.JXCR is used for publications in top-decile impact-factor journals within each field.Source: §2 · p. 5
- The analysis averages each publication’s field-standardized impact after aggregating publications at multiple levels.The benchmark standardization enables comparison of relative impact at the world level.Source: §2 · p. 5
3. Impact of Italian research compared to world average
Across Italy’s public research system, field-standardized citation impact rose above the world average, with substantial variation by discipline, organization type, institution, and research unit. The analysis moves from national and disciplinary patterns to organizational comparisons, where it also identifies scope limits for interpreting rankings.
3. Impact of Italian research compared to world average
- Italian publications grew 26% over six years, while conference proceedings retreated in the final year after earlier growth.The analysis covers 250,000 records from 2001–2006; articles made up over two thirds of publications, and reviews exceeded 5% in 2006. Italian journals indexed in the 2008 JCR accounted for roughly 1% of JCR journals.Source: §3 · p. 6, Table 1 · p. 6
- 12% more citations than the world average over 2001–2006, rising to 30% more in 2006, in the field-standardized Cites/XCR measure.The series rose steadily; the percentage of publications in top journals also increased, and those publications received 9% more citations than same-year, same-journal averages. 2003 values were particularly low, possibly related to a peak in lower-cited conference proceedings.Source: §3 · pp. 6–7, Table 2 · p. 6
3.1.1 Impact per discipline
- Chemistry and clinical medicine had the highest disciplinary Cites/XCR values, 19% and 20% above the world average; engineering was the only discipline below it.Clinical medicine, engineering, and physics together accounted for roughly 57% of publications. Standardized average impact increased in all eight disciplines; Cites/XCR rose at an average annual rate of 5.9%, while Cites/JXCR changes were mixed.Source: §3.1.1 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 8
3.1.2 Impact per field
- In biomedical research, one field achieved the highest Cites/XCR at 1.44, followed by hematology (1.31) and allergy (1.22); anatomy and morphology was 29% below average.Cites/XCR increased in all 14 fields, most in infectious diseases (+14.5%); Cites/JXCR ranged from 0.9 for virology to 1.21 for hematology.Source: §3.1.2 · p. 9
3.2.1 Impact per type of institution
- Hospitals and healthcare research organizations (HCROs) produced 20% more citations than the world average, versus 16% for research institutions and 8% for universities.Hospitals-HCROs also had the highest top-journal share (15.1%) and Cites/JXCR (1.16); universities accounted for over 67% of output and 90% of mathematics publications.Source: §3.2.1 · p. 10, Table 8 · p. 10, Table 7 · p. 10
3.2.2 Impact per individual organization
- Among the 15 organizations producing two thirds of output, INAF had the highest Cites/XCR at 1.36; the top ten organizations meeting the 50-publication threshold contained no universities.The Italian Space Agency led that top-ten list at 2.91, but only 4.4% of its publications appeared in top journals. Nine of the ten ranked institutions had relatively few publications, so their Cites/JXCR rankings could depend on very few articles.Source: §3.2.2 · pp. 11–12, Table 9 · p. 11, Table 10 · p. 12
3.2.3 Impact of research units within individual institutions
- The authors present the analysis as a benchmark for identifying organizational strengths and weaknesses, with field- and time-series detail available for targeted comparisons.Inter-temporal analysis is described as suitable for evaluating intervention effectiveness; the analysis can also be extended to research groups and organizational units.Source: §3.2.3 · pp. 12–13
- Among CNR institutes, ITAE led Cites/XCR at 2.92 and Cites/JXCR at 1.96, while the Institute for Biomedicine and Immunology led top-journal concentration at almost 40%.The CNR example covers institutes with 120–436 publications over six years; the authors note that reconciling variant institutional addresses enabled the analysis.Source: §3.2.3 · p. 13, Table 11 · p. 13
4. Discussion and conclusions
The study compares field-standardized impact across Italian research over 2001–2006 and examines findings by discipline and organization. It presents a replicable bibliometric approach while emphasizing that impact indicators alone are insufficient for policy decisions.
4. Discussion and conclusions
- Italian publications received 12% more citations than the world average over the six-year period, rising to 30% more in 2006.Clinical medicine, biomedical research, and chemistry were identified as drivers of the improving national performance.Source: §4 · p. 14
- Hospitals and health research organizations led universities in average impact, while universities produced over two thirds of national research output.The hospital and health research organization group numbered almost 200 and focused particularly on biomedical research and clinical medicine.Source: §4 · p. 14
- National scientific production grew at almost 5% per year over the six years observed.Source: §4 · p. 14
- The study illustrates diagnostic analyses of performance at the levels of individual organizations, disciplines, and detailed fields.Source: §4 · p. 14
- The methodology is described as replicable in other countries and potentially useful for policy interventions and assessing their effectiveness over time.The authors also extend these considerations to individual universities and public research organizations.Source: §4 · p. 14
- Impact findings alone should not guide policy without considering other evaluation dimensions, including research efficiency and labor and capital inputs.International research-productivity comparisons also require comparable measures from other nations.Source: §4 · p. 14
APPENDIX – further analysis
The appendix illustrates organization-level rankings using top-journal publication shares, top-decile impact, and discipline-specific analyses in physics and oncology.
APPENDIX – further analysis
- All but one of the top ten organizations by top-journal publication share were hospitals, and all but one had higher citations than other works in those journals.The table covers organizations with at least 50 publications during 2001–2006.Source: APPENDIX – further analysis · p. 16, Table 12 · p. 16
- Seven organizations appear in both the top-ten Cites/XCR ranking for top-decile publications and Table 10, with the same three organizations occupying the top three places.The table ranks organizations with at least 50 publications by average Cites/XCR among their top 10% of publications.Source: APPENDIX – further analysis · p. 16, Table 13 · p. 16
- In physics, the top three organizations by Cites/XCR were research institutes, each scoring above two; the top ten also included four universities.Two of those universities were schools for advanced studies.Source: APPENDIX – further analysis · pp. 16–17, Table 14 · p. 17
- The appendix also presents a top-ten organization ranking for oncology research, based on Cites/XCR and restricted to organizations with at least 50 publications.Source: APPENDIX – further analysis · p. 17, Table 15 · p. 17