Source-linked AI summary
How are excellent (highly cited) papers defined in bibliometrics? A quantitative analysis of the literature
Lutz Bornmann
TL;DR
Bibliometric studies use varied methods to identify excellent papers, leaving no clear account of common practice. This study analyzes bibliometric literature to characterize those definitions and thresholds; percentile classes average the top 7.6% of papers, with a median of 3%, and the top 1% is most frequent.
Problem
Bibliometric studies use varied definitions of excellent papers, and the methods and thresholds most commonly used have not been clearly established.
Source: §4 · p. 12Method
The study quantitatively analyzes empirical bibliometric papers to identify definitions and thresholds used by bibliometricians.
Source: §1 · p. 5, §2.1 · p. 5Results
Percentile classes average the top 7.6% of papers and have a median of 3%; the top 1% class is most frequent, followed by the top 10%.
Source: §4 · p. 14Takeaways & Limitations
The reported thresholds offer guide values for identifying excellent papers according to average or most frequent bibliometric practice.
Source: §4 · p. 14The analysis operationalizes excellence as high citedness and focuses on papers in information and library science.
Source: §2.1 · p. 5, §1 · p. 5
Abstract
from arXiv · showhide
As the subject of research excellence has received increasing attention (in science policy) over the last few decades, increasing numbers of bibliometric studies have been published dealing with excellent papers. However, many different methods have been used in these studies to identify excellent papers. The present quantitative analysis of the literature has been carried out in order to acquire an overview of these methods and an indication of an "average" or "most frequent" bibliometric practice. The search in the Web of Science yielded 321 papers dealing with "highly cited", "most cited", "top cited" and "most frequently cited". Of the 321 papers, 16 could not be used in this study. In around 80% of the papers analyzed in this study, a quantitative definition has been provided with which to identify excellent papers. With definitions which relate to an absolute number, either a certain number of top cited papers (58%) or papers with a minimum number of citations are selected (17%). Around 23% worked with percentile rank classes. Over these papers, there is an arithmetic average of the top 7.6% (arithmetic average) or of the top 3% (median). The top 1% is used most frequently in the papers, followed by the top 10%. With the thresholds presented in this study, in future, it will be possible to identify excellent papers based on an "average" or "most frequent" practice among bibliometricians.
1 Introduction
Bibliometric attention has shifted toward highly cited papers, but studies have used varied definitions of excellence. This study maps those methods and the thresholds bibliometricians use, treating excellence as a normative attribution grounded in empirical practice.
- Highly cited papers are a small, closely studied part of a citation distribution that is generally skewed, with many rarely cited papers.Source: §1 · p. 3
- Science-policy and public interest in research excellence has grown alongside the use of indicators that identify highly cited papers and institutions.Source: §1 · p. 3
- Earlier studies describe highly cited papers as drawing on influential prior work and often involving many authors and international collaboration.The cited related work also reports that highly cited papers receive citations from a wide range of journals and fields.Source: §1 · pp. 3–4
- The study maps the methods and quantitative thresholds used in empirical bibliometric papers to identify excellent papers.It focuses on expert practice because choosing a bibliometric definition of excellence is a normative attribution.Source: §1 · pp. 4–5
2 Methods
The study searched Web of Science for papers about highly or most-cited work, then coded their definitions and study characteristics. It used association tests and Cramér’s V to analyze categorical data.
2.1 Data
- The Web of Science search covered specified citation-related phrases in information and library science; 321 papers were found, 16 excluded, and 365 definition entries analyzed.Source: §2.1 · pp. 5–6
2.2 Coding the data for quantitative analysis2
- The coding recorded how excellence was defined, the subject area, research unit, and terminology, with data coded twice to check and correct inconsistencies.Source: §2.2 · pp. 6–7
2.3 Statistical procedures
- Associations between categorical variables were tested with Pearson’s chi-square, and Cramér’s V was used to assess association strength.Source: §2.3 · p. 7
3 Results
Most studies used quantitative definitions, especially fixed counts of top-cited papers; percentile classes and citation-count thresholds were also common. Threshold values and definition choices varied across measures and subject areas.
- Database definitions were especially common in life sciences, while physical-science studies most often used quantitative definitions; fixed top-paper counts were less common in multidisciplinary studies.Database definitions appeared in 17.39% of life-science studies and 0% of health-science studies; quantitative definitions appeared in 36.11% of physical-science studies, where qualitative definitions appeared in 11.11%.Source: §3 · pp. 8–9
- Most studies concerned individual papers (64.84%); “highly cited” was the most common label (51.59%), followed by “most cited” (32.56%).Source: §3 · p. 9
- 57.55% of papers specified a number of top-cited papers, around a quarter used percentile classes, about 17% set a citation-count threshold, and around 2% used distance from a mean or CSS.Source: §3 · p. 10
- For fixed-count definitions, the median was 13.5 papers despite an arithmetic mean of about 856; the most frequent counts were 1 and 10 papers.Source: §3 · p. 11
- Percentile classes averaged the top 7.6% of papers and had a median of 3%; the top 1% was the most frequent class, followed by the top 10%.Source: §3 · p. 11
- Citation-count thresholds averaged 93 citations and had a median of 50; median thresholds ranged from 189 citations in life sciences to 10 in social sciences.Source: §3 · pp. 11–12
4 Discussion
The discussion surveys varied ways of defining excellent papers and identifies commonly used thresholds, while favoring relative definitions for comparisons across fields and time periods.
- Around a third of the papers provided a quantitative definition, and absolute-number definitions were used most often, although relative definitions are preferred for cross-field and cross-time comparisons.Source: §4 · p. 12
- Only 2% of papers using quantitative methods cited thresholds based on a distance from the mean.Source: §4 · p. 12
- Percentile rank classes appeared in around half of papers using quantitative methods, and the discussion presents them as a preferred way to identify an elite set.Source: §4 · p. 13
- Percentile methods allow expected shares of top-ranked papers to be compared without calculating means from skewed citation counts.Source: §4 · p. 13
- The top 7.6% is the arithmetic average threshold and the top 3% the median; the top 1% is used most often, followed by the top 10%.Source: §4 · pp. 13–14
- The study aims to make future identification of excellent papers possible using average or most-frequent bibliometric practice, while noting that excellence has no generally accepted formula.Source: §4 · p. 14