Source-linked AI summary
Citation analysis may severely underestimate the impact of clinical research as compared to basic research
Nees Jan van Eck, Ludo Waltman, Anthony F. J. van Raan, Robert J. M. Klautz, Wilco C. Peul
TL;DR
Popular bibliometric indicators have important limitations for performance assessment. The study defines scientific fields and examines citation practices, finding that clinical intervention research may be underestimated while basic and diagnostic research may be overestimated.
Problem
Popular bibliometric indicators, including the h-index, have important limitations for performance assessment.
Method
The study defines scientific fields using a widely used source for medical-literature coverage.
Results
Clinical intervention research may be underestimated, while basic and diagnostic research may be overestimated.
Takeaways & Limitations
More accurately normalized bibliometric indicators are urgently needed to account for differences between research areas within the same field.
Takeaways & Limitations
Differences in citation practices can bias bibliometric indicators.
Abstract
from arXiv · showhide
Background: Citation analysis has become an important tool for research performance assessment in the medical sciences. However, different areas of medical research may have considerably different citation practices, even within the same medical field. Because of this, it is unclear to what extent citation-based bibliometric indicators allow for valid comparisons between research units active in different areas of medical research. Methodology: A visualization methodology is introduced that reveals differences in citation practices between medical research areas. The methodology extracts terms from the titles and abstracts of a large collection of publications and uses these terms to visualize the structure of a medical field and to indicate how research areas within this field differ from each other in their average citation impact. Results: Visualizations are provided for 32 medical fields, defined based on journal subject categories in the Web of Science database. The analysis focuses on three fields. In each of these fields, there turn out to be large differences in citation practices between research areas. Low-impact research areas tend to focus on clinical intervention research, while high-impact research areas are often more oriented on basic and diagnostic research. Conclusions: Popular bibliometric indicators, such as the h-index and the impact factor, do not correct for differences in citation practices between medical fields. These indicators therefore cannot be used to make accurate between-field comparisons. More sophisticated bibliometric indicators do correct for field differences but still fail to take into account within-field heterogeneity in citation practices. As a consequence, the citation impact of clinical intervention research may be substantially underestimated in comparison with basic and diagnostic research.
1. Introduction
Citation-based assessment is widely used in medical research, but citation practices vary substantially between and within fields, limiting valid comparisons. The paper examines how these differences can bias comparisons between clinical intervention research and basic or diagnostic research.
- The h-index and impact factor fail to account for large differences in citation practices between scientific fields.
- Field-normalized indicators correct for differences between fields but often overlook heterogeneity between research areas within the same field.
- Within-field citation heterogeneity is especially apparent in medical fields, where clinical intervention, basic, and diagnostic research areas differ substantially.
- Clinical intervention research may have its citation impact substantially underestimated relative to basic and diagnostic research.
- The paper presents an empirical analysis using visualization to show how citation practices differ between research areas within medical fields.
2. Methodology
The methodology defines medical fields using Web of Science categories and maps research areas from publication terms. It combines term co-occurrence with normalized citation scores to visualize both field structure and citation impact.
- Natural language processing extracts noun phrases from publication titles and abstracts, then selects 2000 characteristic terms for each field.
- General or infrequently occurring noun phrases are filtered out to reduce distortion and improve robustness, using field-dependent occurrence thresholds.
- Term co-occurrence frequencies are supplied to VOS mapping, which places strongly related terms close together in a two-dimensional map.
- Map distances are interpreted as relationships between terms, while the horizontal and vertical axes have no substantive meaning.
- Each publication’s citations are normalized by the average citations for publications from the same year, and term colors represent average normalized citation scores.
3. Results
Term maps across three medical fields distinguish research areas and reveal systematic citation-impact differences. Clinical intervention areas generally have lower impact, whereas basic and diagnostic areas tend to have above-average impact, with some bridges between them.
- Term maps for Cardiac & cardiovascular systems, Clinical neurology, and Surgery show clear distinctions between different research areas.
- In Cardiac & cardiovascular systems and Clinical neurology, clinical intervention areas are mainly on the left, while basic and diagnostic areas are mainly on the right.
- In the Surgery map, clinical areas occupy the left, middle, and upper regions, while more basic research appears in the lower-right.
- Bridges between basic and clinical areas appear in the maps and seem to represent translational research.
- Within medical fields, some research areas receive two or three times more citations per publication than others.
- Low-impact areas tend to focus on clinical research, especially surgical interventions, while basic and diagnostic areas usually have above-average citation impact.
4. Discussion and conclusion
The analysis finds substantial within-field differences in citation practices, showing that common bibliometric indicators can misrepresent comparisons between medical research areas. Clinical intervention research tends to be underestimated relative to basic and diagnostic research, while the methodology provides a detailed but bounded visualization of these patterns.
- Popular indicators such as the h-index and impact factor do not correct for citation-practice differences and may lead to invalid field comparisons.
- Large citation-practice differences occur even within a single medical field, complicating comparisons between research areas.
- Low-impact research areas tend to be clinically oriented, whereas basic and diagnostic areas usually have citation impact above the field average.
- Two- or even threefold citation-impact differences between research areas within one medical field are not uncommon.
- The methodology does not draw explicit research-area boundaries, so it cannot provide exact citation-impact differences between areas such as basic and clinical research.
- Interpretation is limited by ambiguous terms, possible gaps in Web of Science coverage, and dependence on its field definitions.
- The authors conclude that more accurately normalized indicators should account for differences both between fields and between research areas within fields.
- Field-normalized indicators correct for between-field citation differences but fail to correct for heterogeneity among research areas within the same field.