Source-linked AI summary
The new Excellence Indicator in the World Report of the SCImago Institutions Rankings 2011
Lutz Bornmann, Felix de Moya-Anegón, Loet Leydesdorff
TL;DR
The paper introduces the Excellence Indicator to compare institutions’ shares of top-10% cited papers against a 10% expectation and test whether differences are statistically significant. It applies proportion-based significance testing to institutional comparisons, while noting that journal classifications simplify inter-journal and inter-discipline relations.
Problem
Institutional ranking differences require a way to determine whether observed Excellence Indicator differences exceed the 10% reference expectation or random variation.
Method
The paper uses the Excellence Indicator—the percentage of an institution’s papers in the normalized top-10% citation set—and z-tests for independent proportions.
Results
Institutions above 10% perform above expectation, while statistically significant differences can be interpreted beyond the analyzed sample data.
Takeaways & Limitations
The Excellence Indicator supports comparisons against a reference standard and between institutions using significance tests.
Takeaways & Limitations
Journal classifications reduce the complexity of inter-journal and inter-discipline relations and cannot be expected to match them one-by-one.
Abstract
from arXiv · showhide
The new excellence indicator in the World Report of the SCImago Institutions Rankings (SIR) makes it possible to test differences in the ranking in terms of statistical significance. For example, at the 17th position of these rankings, UCLA has an output of 37,994 papers with an excellence indicator of 28.9. Stanford University follows at the 19th position with 37,885 papers and 29.1 excellence, and z = - 0.607. The difference between these two institution thus is not statistically significant. We provide a calculator at http://www.leydesdorff.net/scimago11/scimago11.xls in which one can fill out this test for any two institutions and also for each institution on whether its score is significantly above or below expectation (assuming that 10% of the papers are for stochastic reasons in the top-10% set).