Source-linked AI summary
How do you define and measure research productivity?
Giovanni Abramo, Ciriaco Andrea D'Angelo
TL;DR
Research productivity is commonly measured as publications per researcher, but this omits output value and field differences. The paper operationalizes productivity through Fractional Scientific Strength and argues that popular indicators and rankings have fundamental limits.
Problem
Bibliometrics commonly defines research productivity as publications per researcher, despite concerns that this overlooks output value and differences across research fields.
Method
The paper operationalizes research productivity economically through Fractional Scientific Strength (FSS), measuring it across researchers and organizational levels.
Results
Popular bibliometric indicators and rankings have two fundamental limits: they do not normalize output value to input or classify scientists by research field.
Takeaways & Limitations
The authors call for refining FSS measurement and avoiding institutional rankings based on invalid indicators in policy and research administration.
Takeaways & Limitations
Percentile ranks have limited suitability for comparing or aggregating productivity across multi-field units because they compress performance differences and lack equal intervals.
Abstract
from arXiv · showhide
Productivity is the quintessential indicator of efficiency in any production system. It seems it has become a norm in bibliometrics to define research productivity as the number of publications per researcher, distinguishing it from impact. In this work we operationalize the economic concept of productivity for the specific context of research activity and show the limits of the commonly accepted definition. We propose then a measurable form of research productivity through the indicator "Fractional Scientific Strength (FSS)", in keeping with the microeconomic theory of production. We present the methodology for measure of FSS at various levels of analysis: individual, field, discipline, department, institution, region and nation. Finally, we compare the ranking lists of Italian universities by the two definitions of research productivity.
Introduction
The paper argues that publication counts per researcher inadequately represent research productivity because research outputs differ in value and production involves multiple, difficult-to-measure inputs. It therefore operationalizes productivity through Fractional Scientific Strength (FSS), a proxy designed for measurement across multiple analytical levels.
- Problem: Bibliometricians commonly define research productivity as publications per researcher while treating citations as impact, but publication outputs have different values.This publication-count definition is therefore insufficient from a microeconomic productivity perspective.
- Contribution: Fractional Scientific Strength (FSS) is proposed as the paper’s best proxy for average yearly labor productivity at individual, field, discipline, department, organization, regional, and national levels.The methodology is presented for evaluating research performance across these unit levels.
- Conceptualization: Research is a production process using human, tangible, and intangible inputs to generate tangible and intangible forms of new knowledge.Publications, patents, presentations, and databases are among the codified outputs used to approximate new knowledge.
- Measurement requirements: Productivity measurement must account for unequal co-author contributions and differences in publication intensity and citation behavior across fields.The paper recommends fractional contributions and within-field comparisons to reduce ranking distortions.
- Limitations: Production factors beyond labor are difficult to identify and value, making total-factor productivity difficult to measure.Examples include accumulated knowledge and scientific instruments shared among units.
Labor productivity in a specific field
Field-level FSS measures research productivity using publication citations, fractional researcher contributions, and research-staff salary as an input. These measures support field-strength assessment and can be aggregated across multi-field units using either individual-researcher or field-level performance.
- Field-level FSS: Field-level FSS combines research-staff salary, publication counts, citations relative to same-year subject-category averages, and fractional researcher contributions.The salary is measured for the university’s research staff in the SDS during the observation period.
- Field-level FSS: FSS-based rankings for each SDS can be expressed as percentiles or as ratios to the average FSSS of productive universities.The reference average includes only universities with productivity above zero in the SDS.
- Field-level FSS: Field-level productivity measures identify strengths and weaknesses that can inform research policies and strategies.
- Multi-field aggregation: Multi-field units can aggregate productivity using either individual-researcher performance or the performance of the SDSs represented in the unit.The appropriate method depends on the objective of the measurement.
- Limits of percentile aggregation: 70 is the average rank percentile for a unit whose two researchers both rank third in separate SDSs with 10 researchers each.The example illustrates that percentile averaging can obscure differences in the gaps between researchers’ productivities and field sizes.
- Multi-field aggregation: The individual-based method emphasizes average researcher performance, whereas the field-based method emphasizes the overall product of researchers within each field.The two methods produce quite similar performance results, although the authors note that a comparative in-depth analysis is future work.
Discussion and conclusions
The discussion argues that popular bibliometric indicators often fail to measure research productivity because they do not normalize output to inputs or classify scientists by field. It calls for further refinement of Fractional Scientific Strength (FSS) and emphasizes productivity’s importance for research policy and management.
- Core limitations: The h-index ignores citations below h and citations above h for h-core works, while also failing to field-normalize citations or account for co-authorship.These shortcomings make the h-index and most variants inappropriate from a microeconomic perspective.
- Core limitations: The new crown indicator measures average standardized citations rather than unit productivity, so a unit with double another’s MNCS can have half its productivity.The passage also notes that input data are not correspondingly divided according to fields of allocation because field-level researcher and expenditure data are lacking.
- Related indicators: The research impact quotient is a related individual-level productivity indicator that divides the square root of total research impact by production time.Its total research impact normalizes external citations by cited-paper co-authors and citing-paper bibliographic references, while eliminating self-citations.
- Core limitations: Popular bibliometric indicators and rankings have two fundamental limits: they lack output-to-input normalization and classification of scientists by research field.The paper argues that without normalization there can be no measure of productivity, while field classification is necessary for multi-field research units.
- Implications: The authors call for further refinement of FSS in real-use contexts and caution against distributing institutional performance ranking lists.They argue that productivity is the most important or only indicator informing policy, strategy, and operational decisions in many evaluation contexts.