Source-linked AI summary
The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation
Ludo Waltman, Clara Calero-Medina, Joost Kosten, Ed C. M. Noyons, Robert J. W. Tijssen, Nees Jan van Eck, Thed N. van Leeuwen, Anthony F. J. van Raan, Martijn S. Visser, Paul Wouters
TL;DR
The paper introduces and evaluates the Leiden Ranking 2011/2012 as a bibliometric alternative to global rankings that combine heterogeneous performance dimensions. It documents the ranking’s data collection, indicators, comparisons, and innovations, including highly cited-publication measures, fractional counting, language exclusion, and stability intervals. The paper concludes that Leiden is more focused and transparent, while emphasizing that its scope is limited to selected aspects of scientific performance.
Problem
Global rankings often combine heterogeneous university-performance dimensions and data sources, making their aggregate indicators difficult to interpret and compare.
Method
The paper documents the Leiden Ranking 2011/2012, its bibliometric data collection and indicators, comparisons with other rankings, and four methodological innovations.
Results
The Leiden Ranking focuses exclusively on scientific performance using bibliometric data and avoids university-supplied data, questionable surveys, and aggregate multi-dimensional scoring.
Takeaways & Limitations
Leiden offers a more focused and extensively documented basis for interpreting university scientific performance than broader global rankings.
Takeaways & Limitations
The ranking does not capture teaching performance or several other aspects of scientific performance, including societal impact and research in non-Web-of-Science journals.
Abstract
from arXiv · showhide
The Leiden Ranking 2011/2012 is a ranking of universities based on bibliometric indicators of publication output, citation impact, and scientific collaboration. The ranking includes 500 major universities from 41 different countries. This paper provides an extensive discussion of the Leiden Ranking 2011/2012. The ranking is compared with other global university rankings, in particular the Academic Ranking of World Universities (commonly known as the Shanghai Ranking) and the Times Higher Education World University Rankings. Also, a detailed description is offered of the data collection methodology of the Leiden Ranking 2011/2012 and of the indicators used in the ranking. Various innovations in the Leiden Ranking 2011/2012 are presented. These innovations include (1) an indicator based on counting a university's highly cited publications, (2) indicators based on fractional rather than full counting of collaborative publications, (3) the possibility of excluding non-English language publications, and (4) the use of stability intervals. Finally, some comments are made on the interpretation of the ranking, and a number of limitations of the ranking are pointed out.
1. Introduction
The Leiden Ranking 2011/2012 addresses methodological problems in global university rankings by focusing exclusively on scientifically measured performance. It introduces bibliometric and reporting innovations intended to improve comparison and interpretation.
- Global university rankings inform strategic decisions, university management, and student selection despite longstanding methodological problems.
- The Leiden Ranking restricts measurement to scientific performance rather than combining distinct university-performance dimensions in one aggregate indicator.It uses Web of Science bibliometric data for 500 major universities and no university-supplied data.
- The 2011/2012 edition addresses shortcomings of earlier ranking editions and other university rankings through several innovations.
- The PPtop 10% indicator counts highly cited publications and is less sensitive to extremely highly cited publications than other citation-impact indicators.
- Fractional counting, exclusion of non-English publications, and stability intervals provide alternative handling of collaboration, language, and indicator sensitivity.The paper presents fractional counting as producing more accurate university comparisons; stability intervals show sensitivity to the underlying publication set.
2. Comparison with other university rankings
The paper compares the Leiden Ranking with ARWU, THE, and SIR, emphasizing differences in dimensions, data sources, indicators, and transparency. It presents Leiden as a bibliometric ranking designed to avoid several problems identified in broader aggregate rankings.
- ARWU: ARWU combines education, faculty, research output, and per-capita performance through six indicators, including prizes, highly cited researchers, journals, and Web of Science publications.
- ARWU: The paper identifies ARWU limitations including arbitrary indicator weights, field and age disadvantages, problematic prize affiliations, and emphasis on university size.
- THE: THE combines 13 indicators across teaching, research, citations, industry income, and international outlook, using reputational surveys and substantial university-supplied data.Its citation normalization is documented incompletely.
- THE: The paper criticizes THE for aggregate multi-dimensional scoring, dependence on potentially nonstandardized or manipulable university data, and weaknesses in reputational surveys.The discussion notes that reputational feedback may advantage already well-known universities.
- Leiden Ranking: Compared with these rankings, Leiden avoids aggregate multi-dimensional indicators, university-supplied data, questionable surveys, and limited documentation.
- SIR: SIR is more similar to Leiden because both rely exclusively on bibliometric data and focus on scientific performance.
3. Data collection
The Leiden Ranking collects Web of Science publications for major universities using defined document, time-period, disciplinary, and output criteria, then assigns publications through name matching and collaboration analysis. Its 2011/2012 coverage expanded especially in China and South Korea, while assignment remained imperfect.
- Data sources and selection: Publication data came from Web of Science and covered articles, letters, and reviews published from 2005 to 2009 in the sciences and social sciences.Arts and humanities publications were excluded, and universities generally needed at least 500 publications in each year.
- Publication assignment: Publication assignment first matched explicit university names, recognized variants and abbreviations, and assigned publications from university institutes to their parent universities.Name variants occurring at least five times were included in the first round.
- Publication assignment: A second assignment round used strong author–university collaboration links to attribute some academic-hospital publications lacking an explicit university name.This was intended to reduce the effect of differing university–hospital relationships on international comparisons.
- Limitations: Assignment was not error-free because infrequent name variants were omitted and hospital-based attribution was inherently inaccurate.The authors expected more false negatives than false positives and estimated that universities generally had no more than 5% false negatives.
- Data sources and selection: The ranking included only the 500 universities with the largest Web of Science publication output, spanning 41 countries and 3.4 million publications.These publications represented 61.3% of all Web of Science publications from 2005–2009.
- Coverage changes: Compared with the 2010 edition, coverage increased from 18 to 31 Chinese universities and from 8 to 18 South Korean universities.Australia, Brazil, India, and Taiwan each gained at least three universities.
4. Indicators
The Leiden Ranking uses bibliometric indicators for publication output, citation impact, and scientific collaboration, with options for fractional counting, language filtering, and stability intervals. Results illustrate how indicator design changes university comparisons and captures distinct aspects of research performance.
- Overview: The ranking measures publication output, citation impact, and scientific collaboration, while supporting full or fractional counting and exclusion of non-English publications.Stability intervals can also accompany indicators.
- 4.1. Impact indicators: MNCS normalization substantially changes citation-impact assessment relative to raw MCS values by accounting for differences in citation density across fields.MIT has MNCS 2.17 and ranks 1st, whereas University of Massachusetts Medical School has MNCS 1.40 and ranks 50th despite similar MCS values.
- 4.1. Impact indicators: The PPtop 10% indicator is less sensitive than MNCS to a single extremely highly cited publication, as shown by University of Göttingen’s contrasting rankings.Göttingen ranks 2nd on MNCS but 238th on PPtop 10%; removing one publication changes its MNCS from 2.04 to 1.09 and its rank from 2nd to 219th.
- 4.2. Collaboration indicators: The four collaboration indicators measure different aspects of scientific collaboration, with distance-based measures complementing publication-based collaboration measures.MGCD and PP>1000 km show a more-or-less linear relation but diverge for approximately 30 universities, revealing characteristics of universities in peripheral locations.
- 4.3. Counting method: Fractional counting provides a benchmark of one at the organization level, whereas full counting can inflate averages and favor organizations concentrated in clinical medicine.Full counting makes average MNCS values exceed one and average PPtop 10% values exceed 10%, because collaborative publications are counted multiple times and are cited more frequently.
- 4.3. Counting method: Full and fractional PPtop 10% indicators correlate highly (r = 0.97), yet their university-level differences can be substantial; excluding non-English publications usually changes results little but significantly benefits some French and German universities.Stability intervals are generally small for the high-output universities shown, because larger publication output generally produces smaller intervals.
5. Interpretation of the Leiden Ranking
The Leiden Ranking should be interpreted as a focused measure of selected scientific-performance dimensions, not as a comprehensive measure of university quality. Indicator values and scope matter more than rank positions alone.
- Scope: The ranking excludes teaching performance and covers only publication output and citation impact in Web of Science journals.It also omits societal impact and impact in national journals not covered by Web of Science.
- Scope: The ranking uses research outputs rather than inputs such as research staff numbers or research funding.The paper notes that ideally scientific performance would be measured using both input and output variables.
- Interpreting positions: A ten-position rank increase is more meaningful near the top than lower down because indicator values are unevenly distributed across universities.The paper gives an almost 0.5 MNCS difference between ranks 1 and 10, versus less than 0.1 between ranks 200 and 300.
- Interpreting positions: Direct indicator comparisons are more accurate than comparisons based only on rank positions.For example, the paper prefers stating that one university performs 20% better than another rather than saying it ranks 20 positions higher.
- Scope: The ranking assesses universities as wholes and cannot support conclusions about individual departments, research groups, or institutes.Different units within the same university may differ substantially in performance.
6. Conclusion
The paper introduces the Leiden Ranking 2011/2012 as a transparent, bibliometric ranking focused on scientific performance. It also acknowledges important scope limitations and proposes extensions involving coverage, statistics, and indicators.
- Contribution: The paper documents the ranking’s data collection methodology, indicators, and interpretation for the 2011/2012 edition.The edition is presented as a global university ranking based on bibliometric data.
- Advantages: Compared with ARWU and THE, the Leiden Ranking avoids arbitrary aggregation, university-supplied data, and questionable survey data.Its extensive documentation is presented as making it more transparent than many other rankings.
- Limitations: The ranking’s exclusive focus on scientific performance limits its usefulness for prospective undergraduate students and omits societal impact.Citation impact is only one element of scientific performance and has methodological difficulties across databases, indicators, and disciplines.
- Future development: Planned extensions include more universities and institution types, university classifications, longitudinal trends, disciplinary breakdowns, and improved or additional indicators.Examples include improved normalization for field differences and a possible university-industry collaboration indicator.
- Future development: Some Leiden Ranking innovations are expected to influence U-Multirank, a user-driven tool for classifying, benchmarking, and ranking universities worldwide.The authors’ institute is involved in developing U-Multirank.