Professional and Citizen Bibliometrics: Complementarities and ambivalences in the development and use of indicators

1. Introduction

• Although output indicators are widely used in research management and hiring, whether measurement and transparency alter research intellectually or only its social forms remains open.Research evaluation is increasingly anticipated in scientific practice and institutional management.Source: §1 · pp. 2, 4, §2.2 · p. 13
• Bibliometric indicators’ ambivalence is inherent to evaluation practices: they act as boundary objects, while best-practice compromises can conceal underlying analytical assumptions and epistemic differences.The paper treats indicators as having different implications across contexts.Source: §1 · p. 4
• Four actor groups bring distinct perspectives: producers develop and sell products, bibliometricians test indicators, managers assess resource competition, and scientists are evaluated.The paper presents public discourse as shaped by translations among these groups.Source: §1 · p. 5
• The paper asks under what conditions indicators are justifiable in practice, conducive to knowledge creation, and consequential for researchers’ visibility and incentives.Translations among the four groups may move debate from blanket rejection toward specifying valid conditions and legitimate purposes.Source: §1 · p. 6
• The main tension is between management’s potentially insufficiently transparent assessments and evaluees’ preference for qualitative evaluation, including perceptions of unfair treatment when they identify errors.The paper organizes its examples around tensions in data and indicators used across contexts.Source: §1 · p. 6

• Evaluations may be criticized for “erroneous assumptions” (e.g., Spaan, 2010), but evaluation necessarily depends on assumptions.Source: §1 · p. 6
• An assumption may seem justified from one perspective and erroneous from another, making divergent evaluations always possible.Source: §1 · pp. 6–7
• Clarifying technical assumptions and possible sources of error may contribute to a more informed discussion of research-evaluation limitations.Source: §1 · p. 7

2.1. Sources of bibliometric data

• WoS and Scopus are considered usable for evaluation in natural and life sciences but under-cover social sciences and humanities (Larivière, Archambault, Gingras, & Vignola-Gagné, 2006; Nederhof, 2006).s3.p4#376da4Source: §2.1 · p. 7
• Google Scholar’s underlying document set remains unclear, and studies warn of misrepresentation and distortion when it is used for evaluation (Delgado López-Cózar, Robinson-García, & Torres-Salinas, 2014; Jacso, 2012a; Harzing, 2012; Aguillo, 2011).s3.p33#592b43Source: §2.1 · pp. 8–9

2.2. Disambiguation

• Database shortcomings may be familiar to producers and bibliometricians but insufficiently known to research managers and the scientists being evaluated.s4.p7#9e26e8Source: §2.2 · p. 9
• Among thousands of “Singh” family-name authors, WoS distinguishes 45 with initial “A.” and lists 9,266 publications for “A. Singh”; Scopus’ Identifier yields 1,908 authors matching “Singh” as family name and “A.” as first initial.Source: §2.2 · p. 10
• 20,143 citations in 2012 were still attributed to Biochimica et Biophysica Acta’s original title, and citation noise for one journal can be comparable to another journal’s signal.s4.p35#fd851cSource: §2.2 · p. 12
• Publication and citation errors can be beyond evaluation studies’ control; evaluees may be unable to correct omissions in time, and cannot change document-type or field assignments.s4.p43#eb6d5bSource: §2.2 · p. 13
• Evaluees may need to collaborate with bibliometricians to unpack reports and make their problems visible to management.The consequences of evaluation can be consequential for scientists.Source: §2.2 · p. 13

2.3. Normalization

• Different normalization procedures can produce different results, while skewed citation distributions challenge use of the mean and non-parametric alternatives can be less intuitive.Reference-set choices also remain difficult: subject categories may lack analytical clarity and fit poorly for sub-fields or multidisciplinary work.Source: §2.3 · pp. 14–15
• Citation-network clustering may improve normalization over journal-based schemes, but reference-set delineation remains theoretically unsolved and pragmatically handled in evaluation.Different analytical frameworks can yield uncertain decompositions of the citation system.Source: §2.3 · p. 16
• Normalization cannot guarantee unambiguous field delineations or eliminate misclassification, and it does not account for institutional differences between fields.Interdisciplinary work may be especially vulnerable to disciplinary misclassification, and users may mistakenly assume normalization solves all cross-field comparability problems.Source: §2.3 · pp. 16–17

2.4. Citation windows

• Managers’ interest in recent performance and near-term expectations can conflict with the longer citation windows standard in evaluation.Short windows favor research-front contributions, yet a considerable percentage of papers have higher mid- or long-term than short-term citation rates.Source: §2.4 · pp. 17–18
• Baumgartner and Leydesdorff (2014) found that many papers have higher mid- or long-term than short-term citation rates and recommend distinguishing transient from sticky knowledge claims.Research-front papers must be positioned relative to competing teams’ papers.Source: §2.4 · p. 18
• Journals and scientists seeking higher journal impact factor scores may deliberately speed up their production cycle, as with PLoS ONE.Source: §2.4 · p. 18
• Three-to-five-year citation windows are a common bibliometric practice and a pragmatic compromise between short- and long-term citation, but they exclude recent publications from evaluation.This creates tension for managers interested in long-term impact and especially in recent work, and for analysts pressured to deliver analyses covering recent years.Source: §2.4 · p. 19
• The window used in an evaluation should be considered in relation to the timeframe of knowledge creation in the field, which may affect its justification for scientists being evaluated.The section raises this as an additional question rather than resolving it.Source: §2.4 · p. 19

3.1. Journal Impact Factor

• As of July 1, 2016, Azrout, van Spanje, & de Vreese’s paper had 3 citations and Bornmann & Leydesdorff’s had 10, although their journals’ JIFs implied tenfold higher expected impact for the latter.The example shows that applying a journal-level average to individual papers can produce misleading expectations.Source: §3.1 · pp. 20–21, Figure 2 · p. 20
• The JIF averages journal impact over the previous two years, predicts neither short- nor long-term impact of an individual paper, and can be skewed by a few highly cited papers.Using a journal aggregate to infer an individual paper’s impact is an ecological fallacy.Source: §3.1 · p. 21
• Despite technical objections, the JIF serves as a boundary object: publishers and editors treat it as journal-quality evidence, while researchers see publishing in a high-JIF journal as recognition.Managers and scientists may continue using it for pragmatic reasons, although professional bibliometrics regards individual-paper evaluation by JIF as ill-advised.Source: §3.1 · pp. 21–22
• SNIP remains a journal indicator, so using it to evaluate single papers risks the same ecological fallacy as JIF, while its complex production environment prevents independent replication.More sophisticated indicators and aggregate-level models may be difficult to reproduce or interpret, and their usefulness can vary across scientific fields.Source: §3.1 · pp. 22–23

3.2. The h-index and its derivates

• The h-index remains popular with managers and scientists despite criticism of its arbitrary definition, and improved variants have not gained acceptance comparable to the original.The persistence illustrates how bibliometricians’ findings do not readily diffuse to institutional users without translation and recontextualization.Source: §3.2 · p. 24
• Because the h-index is not normalized for field, publication year, or scientists’ professional age, it supports comparisons only among scientists similar on all three dimensions.Proposed alternatives include field-, time-, and age-normalized citation scores, such as counting top-10%-cited papers and dividing by years since first publication.Source: §3.2 · pp. 24–25
• H-index and related indicators are used to compare scientists across fields and academic ages, but field delineations are uncertain.Source: §3.2 · p. 25

3.3. Advanced bibliometric indicators

• H-index and total citation counts are transparent and usable with database access, whereas advanced indicators require specialized knowledge and data for larger research units.The mean normalized citation score can be explained to managers and scientists, but percentiles and citing-side indicators are less intuitive or harder to explain.Source: §3.3 · pp. 25–26
• Citing-side variants described by Waltman and van Eck (2013) can normalize across fields and time, but are difficult to understand, reproduce, and calculate, and can obscure raw citation counts.Using these advanced indicators can require a professional bibliometrician in each evaluation.Source: §3.3 · pp. 26–27
• Producer systems and abundant web-based data weaken trust in professional bibliometrics, while users may overlook that databases are black boxes containing data uncertainty and model decisions.Source: §3.3 · p. 27

3.4. Benchmarking scientists and institutions

• Valid individual-scientist reference groups remain difficult to construct because author disambiguation and field boundaries are uncertain, and voluntary author databases may be incomplete.Source: §3.4 · p. 28
• Institutional rankings offer reference values, but results from other providers can differ significantly because their databases or ranking methods differ.Source: §3.4 · pp. 28–29

3.5. The (re-)construction of university rankings

• Despite changing field definitions, rankings across years are highly and significantly correlated, yet individual rank changes may be affected more by model changes than overall correlations suggest.Universities focus on whether their ranks improve or worsen; Leiden Rankings recalculate historical indicators using the latest methodology.Source: §3.5 · p. 30
• CMU’s top-10% share was 18.7% in 2013, 15.5% for reconstructed 2013, and 14.3% in 2016; model effects explain more than 70% of change, citation data 28%.The passage attributes a 1.2% difference to citation data and a remaining 3.2% to the model.Source: Figure 3 · p. 30, §3.5 · p. 31
• CMU's position by share of papers in the top-10% class (all journals included) worsened from 24th among 500 universities in 2013 to 67th among 842 in 2016, when its score was 14.3%.Source: §3.5 · p. 31
• Indicators may objectify discourse quality, but reifying differences over time is error-prone because they can arise from data, models, or model effects on data.Source: §3.5 · p. 31
• Leiden Rankings producers discourage treating ranks as the most important information and encourage inspection of publications and citations, while managers seek rankings to boost institutional reputation and resources.Source: §3.5 · p. 32
• Rankings are becoming more important even as their value as single numbers is questioned for long-term knowledge production across disciplines and specialties.Source: §3.5 · p. 32

4. Conclusions

• Provider data can contain errors, while advanced indicators depend on parameter choices such as field delineation and journal-set selection.Examples include missing journal issues and incorrect publication metadata.Source: §4 · pp. 32–33
• Bibliometricians use different analytical approaches, including field- and time-normalized indicators, which can produce different results.Underlying data may be skewed, and choices such as whether to include recent citation data also vary.Source: §4 · p. 33
• Managers and scientists may have limited bibliometric knowledge, yet their disciplinary and institutional context is needed to interpret results and can make them influential in evaluations.Source: §4 · p. 33
• Bibliometric indicators are socially constructed through translations among stakeholders, making ambivalence in their use inherent rather than accidental.Source: §4 · pp. 32, 34
• Evaluation should improve measurement precision while keeping visible how indicators are constructed; taking construction for granted can turn evaluation into a black-boxed technical routine.Such evaluations can be harmful for the processes being evaluated.Source: §4 · p. 34
• Agreement among users can make indicators seem reliable while reports omit methodological reflections and possible errors, and consensus may conceal problems in their construction.Source: §4 · p. 34
• Evaluation reports should invite discussion through arguments and counterarguments about indicator construction.Source: §4 · p. 35
• Bibliometric evaluations may register past performance poorly but can help construct new perspectives and identify issues for further investigation.Source: §4 · pp. 34–35