PaperlayerView exact revision on arXiv

Source-linked AI summary

What Do Citation Counts Measure? An Updated Review of Studies on Citations in Scientific Documents Published between 2006 and 2018

Iman Tahamtan, Lutz Bornmann

arXiv:1906.04588v2cs.DL

TL;DR

Conventional citation analysis cannot identify a work’s specific contribution to the citing work. This paper updates an earlier review through a narrative review of 41 studies and concludes that large citation databases could improve citation indexing and information retrieval, while citation errors remain a limitation.

  • Problem

    Conventional citation analysis cannot identify the specific contribution of a given work to the citing work.

  • Method

    The paper presents a narrative review updating Bornmann and Daniel (2008) and covering 41 studies published between 2006 and 2018.

  • Results

    Large databases such as WoS or Scopus could be used to develop and improve citation indexing and information retrieval.

  • Takeaways & Limitations

    Citation analysis should use richer citation information to better understand the relationship between citing and cited documents.

  • Takeaways & Limitations

    Citation errors seem common among cited references and citation contexts, which might influence results.

Abstract

from arXiv · show

The purpose of this paper is to update the review of Bornmann and Daniel (2008) presenting a narrative review of studies on citations in scientific documents. The current review covers 41 studies published between 2006 and 2018. Bornmann and Daniel (2008) focused on earlier years. The current review describes the (new) studies on citation content and context analyses as well as the studies that explore the citation motivation of scholars through surveys or interviews. One focus in this paper is on the technical developments in the last decade, such as the richer meta-data available and machine-readable formats of scientific papers. These developments have resulted in citation context analyses of large datasets in comprehensive studies (which was not possible previously). Many studies in recent years have used computational and machine learning techniques to determine citation functions and polarities, some of which have attempted to overcome the methodological weaknesses of previous studies. The automated recognition of citation functions seems to have the potential to greatly enhance citation indices and information retrieval capabilities. Our review of the empirical studies demonstrates that a paper may be cited for very different scientific and non-scientific reasons. This result accords with the finding by Bornmann and Daniel (2008). The current review also shows that to better understand the relationship between citing and cited documents, a variety of features should be analyzed, primarily the citation context, the semantics and linguistic patterns in citations, citation locations within the citing document, and citation polarity (negative, neutral, positive).

Article Highlights

Recent computational and machine-learning methods enable large-scale citation context analyses and automated citation-function recognition. The review finds that citations reflect diverse scientific and non-scientific reasons, so citation evaluation should incorporate citation content and context.

  • Computational and machine-learning techniques have enabled comprehensive citation context and content analyses of large datasets.
  • Automated recognition of citation functions has the potential to enhance information-retrieval capabilities.
  • Papers are cited for different scientific and non-scientific reasons, and only a small percentage of citations are influential.
  • Evaluative bibliometrics could produce more meaningful results by incorporating insights from citation context and content analyses.

1 Introduction

Traditional citation counts treat citations equally, although citations serve different functions and may reflect varied relationships between citing and cited works. Citation content and context analyses extend quantitative counting by examining how cited work is used, using features such as semantics, linguistic patterns, location, and polarity.

  • Citation counts are widely used to indicate scientific impact and overall research quality, but purely quantitative analysis has been criticized.
  • Citations may serve different reasons and functions, including extensive discussion, arbitrary citation, or perfunctory citation.
  • Conventional citation analysis cannot identify the specific contribution of a cited work to the citing work.
  • Citation content and context analyses complement citation counts by considering qualitative and quantitative factors, including how a work is cited.
  • Citation context can characterize a work’s influence through citation position, surrounding semantics and words, citation functions, linguistic patterns, and polarity.
  • Machine-readable formats and automated processing have made it easier to study citations in larger datasets with new approaches.

2 Methods: search for the literature

The authors searched English-language publications from 2006 to 2018 in major bibliographic databases, screened the retrieved records, and supplemented the search through reference-list browsing. The review ultimately included 41 relevant studies.

  • The literature search covered original English-language papers from 2006 to 2018 across WoS, Scopus, and PubMed.
  • The search strategy targeted citation classification, context, content analysis, citation functions, in-text citations, and citing motivations.
  • 188 papers were retrieved: 124 from Scopus, 55 from WoS, and 9 from PubMed.
  • 57 duplicate studies were removed, leaving 131 papers for title, abstract, and full-text screening.
  • Overall, 41 studies were included in the review.

3 Empirical results of studies on citations

The reviewed studies used citation content and context analyses, automated processing, manual analysis, and surveys or interviews to examine citation functions, influence, motives, and linguistic patterns. Across studies, citations varied by function, polarity, location, discipline, and perceived influence, with many references serving limited or non-influential roles.

  • The review grouped studies into citation content or context analyses, citer-motivation surveys or interviews, and reviews of previous studies.
  • Automated analysis: Automated classifiers achieved reported accuracies of 67% for citation-function determination and 94% for recognizing discoveries.
  • Citation influence: 14.6% of citations were labeled important because they used or extended the cited work, while 10.3% of 3134 cited references were influential.
  • Citation influence: The number of times a paper was cited within the citing paper was the most predictive feature of influential papers.
  • Linguistic patterns: Citation wording and function varied with document location, with verbs and hedging patterns differing across paper sections.
  • Citation functions: Citation functions varied substantially, with studies identifying uses such as “use”, “criticism”, “substantiation”, “basis”, “extend”, and “based on”.
  • Citer motivations: Survey and interview studies found that citation decisions reflected relevance and importance as well as publisher, editor, co-author, and outlet factors.

3.1 Content and context analyses of citations to characterize the cited works

Citation content and context analyses use text surrounding references to characterize cited works, citation processes, and relationships between citing and cited documents. Studies vary in context size, processing method, and the functions or meanings assigned to citations.

  • Context windows: 22% of 3500 citation contexts contained two or more sentences, motivating studies of optimal context-window size.
  • Context windows: Three-sentence contexts outperformed one-sentence contexts for information retrieval, while four sentences improved citation-purpose and sentiment classification over smaller windows.
  • Citation content analysis: Citation content analysis reads surrounding text to characterize cited documents and explain why they were cited.
  • Citation context analysis: Citation context analysis characterizes the citing authors’ citation processes, including citation functions, meanings, and locations within documents.
  • Processing methods: Manual analyses can be time-consuming and subjective, whereas automated classification schemes may produce unreliable citation-function results.
  • Citation functions: A single citation may have multiple functions, and citation content can differ across scholarly communities, as shown for Karl Weick’s work.

3.1.1 Automated data processing

Automated processing studies classify citation functions, polarity, linguistic cues, and influence using citation-context features. Across studies, citation functions and influence can be detected with varying accuracy, while most citations appear non-influential.

  • Citation function and/or polarity: More than 60% of citations were neutral, and the classification scheme identified citation functions with at least 75% accuracy across four categories.The categories were weak, contrast, positive, and neutral.
  • Citation function and/or polarity: 83% accuracy was achieved when a classification scheme determined citation sentiment.The scheme distinguished positive, neutral, and negative sentiments.
  • Network analysis: Network and co-occurrence analyses connected citation contexts with clusters of topics and with the substantive content of citing papers.Networks revealed bibliometric-topic clusters and associations between citing-paper content and the content of highly cited or delayed-impact works.

3.1.2 Manual data processing

Manual citation-content studies primarily examined how highly cited works were used, finding substantial variation in concepts, functions, relevance, and citation polarity across fields and regions.

  • Scope and approach: Manual case studies analyzed citation contexts of highly cited classical papers and books, extending traditional citation-content analysis across organization, biomedical, and information or computer sciences.These studies relied more strongly on manual data analysis than automated processing.
  • Organization studies: Regional differences meant that the same highly cited work could be cited for different reasons, despite similar citation frequencies across geographical regions.For Karl Weick’s book, only 3.4% of citations were classified as critical.
  • Organization studies: 496 citation contexts from 301 citing papers identified “storage bins” as the most frequent concept citing Walsh and Ungson’s organizational-memory paper.Other frequent concepts included general references to organizational memory and its use, misuse, and abuse.
  • Relevance and concepts: 221 citations, or 46.6%, to Bourdieu’s work came from “capital”, “habitus”, and “field”, while at least 50% of papers addressed these concepts only in a limited manner.“Social capital” was the most frequently cited capital form at 47.2%.
  • Citation functions: Citation functions differed between natural sciences and social sciences and humanities, although “evidence” and “related studies” were among the most frequent functions.In NS, “evidence” accounted for 20% and “related studies” 16.9%; in SSH, the corresponding values were 22.3% and 21.5%.
  • Citation polarity: 97.13% of citations were “confirmative” and 2.83% were “negational”, with humanities showing more negational citations than social sciences.History had 9.1% negational citations, while perfunctory citations were especially frequent in history and economics.

3.2 Citer motivation surveys or interviews

Surveys and interviews show that citation motivations are multidimensional and shaped by authors’ purposes, disciplinary practices, social networks, publication outlets, and audiences.

  • Approach: Surveys and interviews complement citation-content analyses by asking citing authors to explain the functions and motivations behind their references.Interviews may address cases where the motivation is not explicit in the citing text.
  • Disciplinary differences: Harwood identified eleven citation functions, with “position” most frequent in sociology at 46.2% and “signposting” most frequent in computer science at 26%.“Engaging” occurred at 17.2% in sociology but only 0.88% in computer science.
  • Multiple functions: Over half of citations in both sociology and computer science were reported to have more than one function.Common functions in both fields included supporting, credit, and building.
  • Publication context: Citation behavior was influenced by outlet specialization, co-contributors, citation policies, favored paradigms, audience location, space restrictions, and publication speed.Less specialized outlets encouraged general references that made papers understandable to broader audiences.
  • Interpretation and caveats: Citation counts were viewed as indicators of trust and authority but not as the sole measure of quality, because citing reasons were varied and socially influenced.Knowing cited authors was a popular reason for citing, but it was not always decisive; normative reasons accounted for almost 19% of reported reasons.
  • Reported motivations: Supporting one’s own argument and citing up-to-date papers were each reported by 95.4% of respondents as reasons for citing.Students also reported citing to criticize other works, with 80% endorsing that reason.

4 Summarizing the empirical results

The reviewed studies show that citation functions and motivations are diverse, while computational approaches increasingly analyze citation context, polarity, location, and linguistic features at scale. However, classifier results remain difficult to compare because studies use different schemes, methods, and datasets.

  • Computational approaches: Technical developments enabled comprehensive citation studies using large datasets, automated processing, and machine-learning techniques.Machine-readable papers and richer metadata supported analyses that were previously impractical.
  • Citation functions: Citation-function classification schemes use different category sets, although several functions share meanings across studies.The review reports schemes with small or large numbers of categories and partial overlap in function definitions.
  • Citation motivations: Citation motivations are multidimensional, encompassing scientific and non-scientific reasons, including personal or professional relationships with cited authors.Survey and interview studies also identified varied reasons for citing and trusting authors.
  • Computational approaches: Classifier accuracy varied with the selected features, and abstract similarity produced contrary findings across studies.Citation frequency in the citing article was among the features found more important in some studies.
  • Limitations: Results across classifier studies are not comparable because their classification schemes, methodologies, and datasets differ, and no standard scheme has been established.The review also notes that detecting citation functions is difficult because citing authors do not always state their purposes explicitly.
  • Citation context and polarity: A fuller account of citation relationships requires analyzing context, semantics and linguistic cues, citation locations, and polarity together.Polarity analyses distinguish negative, neutral, and positive attitudes; humanities had more negational citations than social sciences in one study, 4.17% versus 1.7%.
  • Citation functions: “Use” was a frequent citation function, accounting for 17.7% of citations, while the proposed classifier recognized it with 60% accuracy.“Use” appeared frequently in methods and introduction sections.

5 Conclusions

This review updates earlier citation research by synthesizing 41 studies published from 2006 to 2018, including citation-context analyses and studies of scholars’ citation motivations. The evidence indicates that citations have diverse scientific and non-scientific reasons, while citation contexts can improve retrieval and citation indexing.

  • The review updates Bornmann and Daniel’s earlier work by covering 41 studies published between 2006 and 2018.
  • The reviewed studies examine citation content and context, citation motivations through surveys or interviews, and functional relationships between citing and cited documents.
  • Automated citation-function recognition could enhance citation indices and information retrieval by identifying functional categories and relationships between cited and citing documents.
  • Citation-context information improved information retrieval, document clustering, and retrieval of relevant references across several reviewed studies.
  • Citation errors, including incorrect references, faulty reporting, and incorrect attribution, remain common and may influence citation-context study results.
  • Faulty citations were attributed mainly to authors not reading or not fully comprehending original papers, although one study reported contrary findings.
Loading 1906.04588v2…