PaperlayerView exact revision on arXiv

Source-linked AI summary

The Operationalization of "Fields" as WoS Subject Categories (WCs) in Evaluative Bibliometrics: The cases of "Library and Information Science" and "Science & Technology Studies"

Loet Leydesdorff, Lutz Bornmann

arXiv:1407.7849v3cs.DLcs.CY

TL;DR

Journal sets may not be sufficiently disciplinarily oriented to support normalization. The authors examine WC-based LIS and STS examples and conclude that normalization using these categories might seriously harm evaluation quality, with no reason to treat the fields as exceptions.

  • Problem

    The disciplinary orientation of journal sets is questionable when they are used to define normalization reference sets.

    Source: Conclusion · pp. 15–16

  • Method

    The authors examine two familiar examples: LIS, which has a WC, and STS, which does not.

    Source: Conclusion · p. 16

  • Results

    In both cases, normalization using these categories might seriously harm evaluation quality.

    Source: Conclusion · p. 16

  • Takeaways & Limitations

    The authors see no reason to treat LIS and STS as exceptions to the problem.

    Source: Conclusion · p. 16

    The authors defer suggesting valid normalization practices until reference-set definition problems are further solved.

    Source: Conclusion · p. 18

Abstract

from arXiv · show

Normalization of citation scores using reference sets based on Web-of-Science Subject Categories (WCs) has become an established ("best") practice in evaluative bibliometrics. For example, the Times Higher Education World University Rankings are, among other things, based on this operationalization. However, WCs were developed decades ago for the purpose of information retrieval and evolved incrementally with the database; the classification is machine-based and partially manually corrected. Using the WC "information science & library science" and the WCs attributed to journals in the field of "science and technology studies," we show that WCs do not provide sufficient analytical clarity to carry bibliometric normalization in evaluation practices because of "indexer effects." Can the compliance with "best practices" be replaced with an ambition to develop "best possible practices"? New research questions can then be envisaged.

Introduction

The introduction frames the need for meaningful reference sets in field-normalized citation comparisons and proposes an empirical examination of whether WoS Subject Categories classify fields with sufficient precision.

Introduction

  • Field normalization is needed because publication and citation practices differ across fields and evaluated units often combine disciplines.The impact factor cannot support such comparisons because it varies systematically among fields, whose research fronts and communication channels also differ.Source: Introduction · p. 2
  • The introduction identifies journal classification as an unresolved problem: journals may not be disciplinarily organized, and subject sets are inherently fuzzy.Citation-matrix classification has not solved the problem of unambiguously defining journal-based reference sets.Source: Introduction · p. 3
  • Indexer-added categories can link otherwise unrelated journals, while overlapping or similar categories blur the analytical distinctions useful for evaluation.The authors note that these assignments may still serve information retrieval.Source: Introduction · pp. 4–5
  • Journal categories became accepted “best practice” for normalization, and this study focuses on WCs because they are widely used in bibliometric practice.More than one WC may be attributed to a journal, with categories often treated as proportional shares.Source: Introduction · pp. 4–5
  • Imprecise or potentially erroneous classifications are expected when WCs are used to represent fields (Haustein, 2012, p. 101).Source: Introduction · p. 7
  • The authors propose an empirical exploration of Web of Science Subject Categories’ quality.Source: Introduction · p. 7

Empirical examples: LIS and STS

The LIS and STS examples show that WC assignments and citation-based journal groupings do not align cleanly, raising doubts about using these categories for normalization.

Empirical examples: LIS and STS

Science and technology studies

STS spans multiple disciplines and is difficult to delineate through journal categories. The WoS categories assigned to relevant journals overlap only marginally, making them unsuitable for evaluating an STS unit with WC-based normalization.

Science and technology studies

  • STS is described as a meta-stable specialty that continually translates between interdisciplinary work and its mother disciplines.Those disciplines include sociology of science and technology, history and philosophy of science, and business and management.Source: Science and technology studies · p. 13
  • Quantitatively oriented STS studies had become more numerous than qualitative STS studies.Figure 3 covers STS journal articles published between 1965 and 2012.Source: Science and technology studies · p. 13, Figure 3 · p. 14
  • WoS categories assigned to journals most relevant to STS overlap only marginally, making evaluation of an STS unit using WCs impossible.The expected concentration in “Social science, interdisciplinary” is not borne out by the journal-category table.Source: Science and technology studies · pp. 14–15, Table 1 · p. 15
  • With no common denominator in the journal categories, evaluation systems may reinforce incentives for interdisciplinary units to return to disciplinary perspectives.The passage connects such incentives to evaluees’ sensitivity to evaluation systems (Dahler-Larsen, 2012).Source: Science and technology studies · p. 15

Conclusion

Using the LIS category and the categories assigned to STS journals, the paper argues that WC-based normalization can harm evaluation quality. It urges caution while reference sets remain difficult to define and points toward developing “best possible practices.”

Conclusion

  • WCs are increasingly used to select reference sets, although the authors question whether journal sets are sufficiently disciplinary and say collapsing them can aggravate confusion.The conclusion frames this as a problem in defining the units used for normalization.Source: Conclusion · pp. 15–16
  • WC-based normalization using the examined categories might seriously harm evaluation quality in both the LIS and STS cases.The authors add that there is no reason to treat these fields as exceptions.Source: Conclusion · p. 16
  • WCs developed incrementally for information retrieval, whereas newer journal classifications were designed explicitly for bibliometric analysis.The paper notes that WCs have been adapted piecemeal over time.Source: Conclusion · p. 16
  • Alternatives include professionally developed index terms, citing papers as reference sets, paper-level citation clusters, and normalization using universal distribution properties.The paper mentions MeSH and fractional citation counting as examples related to these approaches.Source: Conclusion · pp. 16–18
  • Algorithmic citation fields are difficult to name and validate, require papers to have citations or references for classification, and the Leiden system cannot be accessed or replicated outside its context of use.Journal names serve as a secondary key for assigning non-cited papers to resembling clusters.Source: Conclusion · p. 17
  • The authors urge caution about valid normalizations until reference-set definition is further solved, and frame “best possible practices” as opening new research perspectives.The proposed shift is from compliance with “best practices” toward exploring better possible practices.Source: Conclusion · p. 18

Related papers

Loading 1407.7849v3…