PaperlayerView exact revision on arXiv

Source-linked AI summary

The measurement of italian universities' research productivity by a non parametric-bibliometric methodology

Giovanni Abramo, Ciriaco Andrea D'Angelo, Fabio Pugini

arXiv:1812.08506v1cs.DL

TL;DR

Italian public research assessment still lacked effective, robust, low-cost, broadly agreed methods. This paper develops a DEA-based bibliometric methodology using disciplinary-area assessment and author-linked publications. It reports addressing distortions associated with university resource mixes, disciplinary differences, and database journal coverage.

  • Problem

    Italian public research assessment was still far from effective, robust, low-cost, and universally agreed-upon results.

    Source: §1 · p. 2

  • Method

    The methodology applies DEA to disciplinary areas and aggregates author-linked publication measures by area and university into a global efficiency index.

    Source: §1 · p. 4

  • Results

    The authors report that disciplinary-area assessment and author-level publication attribution address distortions linked to university resource mixes, area prolificity, and journal coverage.

    Source: §1 · p. 4, §6 · p. 18

  • Takeaways & Limitations

    The resulting approach supports a more consistent and robust comparison of university research efficiency across heterogeneous disciplinary compositions.

    Source: §1 · p. 4

    Ranking variation may reflect production factors omitted from the model or localization and scope economies, while the residual is not calculable.

    Source: §5 · p. 16

Abstract

from arXiv · show

This paper presents a methodology for measuring the technical efficiency of research activities. It is based on the application of data envelopment analysis to bibliometric data on the Italian university system. For that purpose, different input values (research personnel by level and extra funding) and output values (quantity, quality and level of contribution to actual scientific publications) are considered. Our study aims at overcoming some of the limitations connected to the methodologies that have so far been proposed in the literature, in particular by surveying the scientific production of universities by authors' name.

1. Introduction

Assessment of publicly funded research had become central to research-system governance, yet Italian public research assessment still lacked effective, robust, low-cost, broadly agreed methods. The paper proposes a bibliometric non-parametric approach for Italian universities, combining disciplinary-area efficiency assessment with author-linked publication data.

1. Introduction

  • Bibliometric and peer-review approaches are the two widespread evaluation methodologies, each with advantages and disadvantages involving cost, execution time, limitations, and objectivity.Prior Italian examples included peer review, bibliometric indicators, and DEA-based productivity assessment.Source: §1 · pp. 2–3
  • Italian public research assessment was still far from effective, robust, low-cost, and universally agreed-upon results.This assessment context sits within a broader demand for evaluation shaped by funding reforms, organizational change, and calls for evidence of publicly funded activities’ socioeconomic impact.Source: §1 · p. 2
  • The study develops a bibliometric non-parametric methodology for measuring public-sector research performance and applies it to Italian universities.It also conducts sensitivity tests by altering independent variables.Source: §1 · pp. 3–4
  • The approach assesses efficiency within disciplinary areas, then combines area scores into a university-level index to address differences in university composition, area prolificity, and journal coverage.Publications are linked to individual authors before being aggregated by disciplinary area and university affiliation.Source: §1 · p. 4
  • The author-based publication archive contrasts with prior approaches that assessed an area using portfolios of publications classified as falling within that area.The authors say the archive improves confidence in the significance of the model’s input and output measures.Source: §1 · pp. 4–5

2. Methods of assessment of research activities

Peer review and bibliometric assessment each have important trade-offs. Peer review is constrained by subjective judgments and cost, while bibliometric measures face coverage, attribution, and aggregation problems.

2.1 Peer-review methodologies

  • Peer review is subjective at expert selection, product assessment, and research units’ selection of products for review.The assessment may also be affected by conflicts of interest, scholar reputation, overlooked qualitative aspects, non-universal appraisal mechanisms, time, and cost.Source: §2.1 · pp. 5–6

2.2 Methodologies based on bibliometric techniques

  • Bibliometric indicators use publications and citations as proxies for research output, potentially overlooking other codified and non-codified forms of dissemination.The paper argues that publications remain a robust indicator of new knowledge production in Italy, where protected codification of research results is negligible.Source: §2.2 · pp. 6–7
  • SCI coverage is not exhaustive, and its journal representativeness is higher in technical-scientific fields than in the humanities.The SCI covers about 4,800 international journals, with marginal representation of humanities journals.Source: §2.2 · p. 7
  • Bibliometric attribution can be inaccurate: errors may reach 30% and arise from bad database entries, inconsistent affiliations, and author-affiliation mismatches.Prior Italian university studies used a top-down approach that assigned publications to organizations through the address field, solving only part of the author-disambiguation problem.Source: §2.2 · pp. 7–8
  • Aggregating bibliometric measures at university level can distort rankings when universities have different distributions of resources across disciplines.Disciplinary differences in publication rates and journal coverage compound the problem.Source: §2.2 · p. 8

3. Parametric and non-parametric techniques for the assessment of total

The section contrasts parametric and non-parametric approaches to measuring research productivity, then presents DEA as a flexible frontier method with specific assumptions and interpretive limits.

productivity

  • Partial productivity indexes normalize outputs by selected inputs, but research organizations’ multiple inputs and outputs make total productivity difficult to capture with weighted partial measures.Parametric methods require an a priori production function and do not identify benchmark best practices.Source: productivity · pp. 8–9
  • Non-parametric methods compare observed production units to construct an efficient frontier and assess other units by their distance from it.They can assess multiple inputs and outputs without predefined weights or a specified production function.Source: productivity · p. 9
  • DEA requires homogeneous decision-making units, a convex production set, and free disposability of resources.These assumptions define which production units and combinations can inform the frontier.Source: productivity · p. 10
  • The output-oriented DEA model measures the maximum equiproportional output increase possible with available inputs; VRS separates technical efficiency from scale effects.Comparing CRS and VRS models on the same data permits extraction of scale efficiency.Source: productivity · p. 10
  • DEA is deterministic, so deviations from the frontier count as inefficiency, and its efficiency measures depend on the selected variables.Unmeasured factors, including research-versus-teaching time, instruments, non-employed staff, and non-publication outputs, may distort university comparisons.Source: productivity · pp. 10–11

4. Survey model

The survey applies bibliometric DEA to Italian university disciplinary areas using research-staff and funding inputs and publication-based outputs. It builds a researcher-linked publication dataset and assigns outputs to authors’ disciplinary areas.

Survey scope

  • The survey covers Italian universities with at least four employed research resources in nine science areas and 205 disciplinary sectors, during 2001–2003.The threshold addresses possible productivity distortion when input values are near zero.Source: §4 · p. 11

DEA model

  • The study applies DEA to bibliometric data for each university disciplinary area, using objective metrics and allowing variable returns to scale.The model’s inputs and outputs are averages of 2001–2003 values, with inputs recorded one year before the output year.Source: §4 · pp. 11–13
  • Inputs distinguish full professors, associate professors, and research fellows, and use PRIN grants as a proxy for additional research funding.Ordinary funds are excluded because they are assumed homogeneous per capita; PRIN funds account for about 15% of universities’ additional funds.Source: §4 · pp. 12–13
  • Outputs measure publication quantity, author contribution, and scientific strength weighted by journal impact factor.PU counts publications with at least one author from the university-area; PC weights publications by that area’s share of authors.Source: §4 · p. 13

Publication database

  • The ORP database links SCI publications to researchers, universities, and disciplinary areas using a rule-based disambiguation algorithm.The survey covers papers and reviews by scientific staff at 77 universities; it disambiguated 53,420 publications algorithmically, required manual disambiguation for 1,593, and discarded 9,298.Source: §4 · pp. 13–14

Output aggregation

  • Output aggregation adds each author’s publications to the disciplinary area to which that author belongs.This author-based approach differs from assigning publications to areas through journal categories.Source: §4 · p. 14

5. Results of the analysis and observations

The results show substantial variation across disciplinary areas and university rankings, including sensitivity to model inputs and aggregation choices. A university-wide index combines area-level efficiency scores, while the authors caution that unmodeled factors may affect rankings.

5. Results of the analysis and observations

  • Average efficiency and score variability differed substantially across UDAs, though journal coverage and SCI representativeness may distort those comparisons.The authors note that journals in some border disciplines may be indexed in other Thomson databases rather than SCI.Source: §5 · p. 15
  • Average PTE scores differed by 17%, 35%, and 56% between efficient universities and the first, second, and third percentiles of inefficient universities.UDA 8 was excluded from this calculation because its technical-efficiency data were anomalous.Source: §5 · p. 15
  • The authors warn against university-level aggregation when it ignores inconsistency among DMUs, favoring assessment at the individual disciplinary-area level.They link an anomalous UDA result to SCI representativeness.Source: §5 · p. 16
  • The global university index weights each UDA’s mean-normalized PTE score by its share of the university’s research personnel.The normalization is intended to offset variability in average efficiency across UDAs.Source: §5 · p. 16
  • The general and active-UDA rankings place the more efficient universities at the frontier.Source: §5 · p. 16
  • Rank variability across a university’s UDAs may reflect omitted production factors or localization and scope economies; the residual’s size is not calculable.The authors also suggest that differences in human-resource quality may contribute to the residual.Source: §5 · p. 16
  • Rankings varied markedly against a publications-per-employee model, while DEA scores measured the output expansion needed to reach the frontier.The comparison used different input and output types and allowed variable returns to scale.Source: §5 · pp. 15, 17
  • Removing additional funds caused strong ranking variability, especially in UDAs 1, 5, 6, and 9, and reduced the number of efficient DMUs.Output-count sensitivity was weaker because the quality, quantity, and contribution measures were strongly correlated.Source: §5 · p. 17

6. Conclusions

The paper develops a bibliometric non-parametric approach to research-performance assessment, combining DEA with disciplinary-area analysis and author-level publication data. The authors present it as a benchmarking and policy-support tool, while cautioning that results are sensitive to model inputs and outputs.

6. Conclusions

  • The DEA model incorporates different research inputs and output dimensions, while area-level assessment and author-level publication aggregation address heterogeneity and improve data accuracy.Outputs include publication quantity, quality, and contribution; author data are homogenized and disambiguated before aggregation by UDA and university.Source: §6 · p. 18
  • Results are sensitive to the model’s input and output values, so the methodology requires further investigation and caution in resource-allocation applications.Source: §6 · p. 18
  • The authors argue that university heterogeneity and differing strategic objectives call for differentiated rather than necessarily uniform assessment and resource-allocation approaches.Source: §6 · pp. 18–19
  • The authors describe the model as a benchmarking tool for research management and an instrument to support policy decisions.Source: §6 · p. 19
  • Proposed extensions include cost-based efficiency, citations for paper quality, time-series data, broader fields, finer disciplinary levels, and analysis of performance determinants.Source: §6 · p. 19
Loading 1812.08506v1…