PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring academic influence: Not all citations are equal

Xiaodan Zhu, Peter Turney, Daniel Lemire, André Vellino

arXiv:1501.06587v1cs.DLcs.CLcs.LG

TL;DR

Citation counts treat all references equally even though the paper seeks to distinguish those with central academic influence. The authors use author-labeled paper–reference pairs and supervised feature selection, finding a four-feature model and better average precision for influence-weighted researcher identification.

  • Problem

    Citation indexes treat citations as equally significant, despite the need to identify references with central academic influence on a citing paper.

  • Method

    The authors ask citing-paper authors to label references and use supervised machine learning with greedy feature selection across citation features.

  • Results

    The best model uses four features and achieves an F-measure of about 42%.

  • Takeaways & Limitations

    Influence-weighted citations produce better average precision than conventional h-index citations when identifying ACL Fellows.

  • Takeaways & Limitations

    Ranking references by degrees of influence is left for future work because finer-grained annotation may be harder for humans and difficult to interpret.

Abstract

from arXiv · show

The importance of a research article is routinely measured by counting how many times it has been cited. However, treating all citations with equal weight ignores the wide variety of functions that citations perform. We want to automatically identify the subset of references in a bibliography that have a central academic influence on the citing paper. For this purpose, we examine the effectiveness of a variety of features for determining the academic influence of a citation. By asking authors to identify the key references in their own work, we created a data set in which citations were labeled according to their academic influence. Using automatic feature selection with supervised machine learning, we found a model for predicting academic influence that achieves good performance on this data set using only four features. The best features, among those we evaluated, were those based on the number of times a reference is mentioned in the body of a citing paper. The performance of these features inspired us to design an influence-primed h-index (the hip-index). Unlike the conventional h-index, it weights citations by how many times a reference is mentioned. According to our experiments, the hip-index is a better indicator of researcher performance than the conventional h-index.

Introduction

Citation counts are widely used to measure academic impact, but equal weighting overlooks whether references substantially influence a citing paper. This paper proposes identifying influential references with author-labeled data and supervised learning, then applying influence-weighted citations to researcher assessment.

  • Motivation: Equal-weight citation indexes treat every citation as equally significant despite citations serving diverse functions.The paper contrasts citation frequency and existing indexes with the goal of measuring how a researcher is cited.
  • Motivation: 40% of references were perfunctory and 18.9% provided an essential component for the citing paper in cited prior studies.Moravcsik and Murugesan also found a third of references redundant, while Teufel et al. reported that 62.7% lacked an attributable specific function.
  • Approach: The model evaluates citation features including linguistic context, citation location, reference frequency, and semantic similarity to the citing paper.Its aim is to predict labels for paper–reference pairs rather than references in isolation.
  • Evaluation: The authors compare conventional h-index values with influence-weighted values when identifying computational-linguistics ACL Fellows.The weighted citations achieved better average precision in this experiment.
  • Approach: The study uses authors’ judgments to label references as influential or non-influential because citing authors are considered best positioned to identify proximate influences.The authors acknowledge that these judgments can include false positives and false negatives.

Related work

Related work has classified citation functions through linguistic cues, rules, and machine learning, while other impact measures weight citations by venue, author prestige, or citation frequency. This paper differs by using citing authors’ labels to identify influential references.

  • Citation functions: Citation context analysis categorizes citations according to linguistic relations such as support, criticism, agreement, or background.CiTO provides a machine-readable ontology for representing many citation relations.
  • Automatic classification: Earlier systems used manually selected rules or linguistic features to classify citation types.Garzone and Mercer used more than 200 rules, while Teufel et al. used 892 cue phrases and reported average F-measures of 68% and 71% for two category schemes.
  • Automatic classification: A biomedical citation classifier using unigrams and bigrams reported an average F-measure of 76.5%, with SVM models generally outperforming MNB models.The features were ranked using mutual information.
  • Annotation: The paper’s methodology asks citing-paper authors to identify influential references, unlike related systems that rely on independent annotations.The authors argue that independent annotators may have difficulty determining citation influence.
  • Citation weighting: Other citation-weighting proposals use citing-journal prestige, journal reference counts, or author prestige rather than the citation’s role in the citing paper.Hou, Li, and Niu instead proposed weighting a citation by how often the reference appears within the citing paper.

Features for supervised learning

The paper frames influential-reference detection as supervised classification of paper–reference pairs. It represents each pair with diverse feature classes and evaluates which features best support prediction.

  • Task formulation: The task is binary classification of each reference as influential or non-influential within a given research paper.The model predicts labels for paper–reference pairs because a reference’s influence can differ across citing papers.
  • Learning setup: Supervised learning trains on manually labeled paper–reference feature vectors and predicts labels for testing pairs.The experiments use a support vector machine.
  • Feature design: The paper’s main methodological contribution is evaluating a wide range of features to determine which representations most effectively predict influence.It identifies feature quality as central to successful prediction.
  • Feature design: The study evaluates count-based, similarity-based, context-based, position-based, and miscellaneous feature classes.The authors emphasize feature selection because not all intuitively attractive features are useful.

1. Count-based features

Count-based features measure how often and where a reference appears in the citing paper, while similarity features compare the cited title with the citing text. The design also considers practical access constraints.

  • Count-based features: countsInPaper whole counts reference occurrences across the paper, while intro, related, and core variants restrict counting to specified sections.Core sections exclude the introduction, related work, acknowledgments, conclusion, and future-work sections.
  • Count-based features: countsInPaper secNum records the number of different sections containing a reference, treating cross-section presence as a signal of significance.This distinguishes broad distribution from high frequency concentrated in one section.
  • Similarity-based features: Similarity features estimate influence from semantic overlap between the citing paper and cited paper, using the cited paper’s title as a surrogate for unavailable full text.The first five similarity features compare the cited title with different parts of the citing paper.
  • Practical constraints: The feature design assumes access to the citing text but not necessarily the cited paper’s full text, allowing features to be computed efficiently from limited document access.Missing reference titles may be easier to locate than complete cited-paper text.

2. Similarity-based features 2.1. sim titleTitle 2.2. sim titleCore 2.3. sim titleIntro 2.4. sim titleConcl 2.5. sim titleAbstr

The paper uses cosine similarities to compare reference titles and citation contexts with sections of the citing paper. Citation contexts are also examined as signals of influence beyond semantic similarity.

  • Title-based features: Title-based features compare a reference title with the citing paper’s title, abstract, introduction, conclusion, and core sections.These features treat major citing-paper sections as summaries that can capture semantic similarity between the citer and reference.
  • Similarity computation: Cosine similarity represents each text as a word-frequency vector after stemming with Porter’s algorithm.Stop words were retained because removing them did not improve model performance.
  • Context-based similarity: Citation-context features compare citation contexts with the citing paper’s title, abstract, introduction, and conclusion.When a reference appears multiple times, the similarities are averaged across its contexts.
  • Context-based similarity: The citation context may also signal influence directly, such as when it says that a cited work inspired the citing authors.For these features, citation context is defined as a ten-word window around the citation, with repeated citations averaged.

3. Context-based features

Context-based features characterize how a citation appears and what evaluative, semantic, sentiment, or emotional language surrounds it. The feature set combines manually constructed lexical categories with General Inquirer, WordNet, and emotion-lexicon signals.

  • Context-meta features: Context-meta features indicate whether authors are mentioned, whether a citation appears alone, and whether it appears first among grouped citations.The author-mention example is “the work of Smith et al. [4].”
  • Lexical features: Twelve lexical features classify citation contexts by relevance, recency, extremity, comparison, semantic differential categories, sentiment, and emotion.The manually created lists target relevance, newness, extremity, and comparison; other features use General Inquirer and emotion resources.
  • Lexical features: General Inquirer features count positive, strong, and active words because such language is hypothesized to indicate influential citations.The corresponding labels are evaluative, potency, and activity categories.
  • Lexical features: WordNet extends the General Inquirer labels from 11,788 to 114,271 words, while emotion features count polarity- and emotion-associated words.Context values range from zero to ten, and repeated citation occurrences are averaged; WordNet features retain only adjective or adverb senses.
  • Feature motivation: Citation position may be predictive because earlier citations are intuitively treated as more important.This motivates position-related context features alongside the lexical and metadata features.

4. Position-based features 4.1. posInSent begin 4.2. posInSent end

Position-based features encode where citations occur within sentences and across the citing paper. Sentence-level positions are represented as binary or percentage features, while paper-level positions are normalized to the paper’s length.

  • Sentence position: Sentence-level features indicate whether a citation appears at the beginning or end of a sentence.For repeated references, the features are percentages across citation occurrences.
  • Paper position: Paper-level features measure the mean, standard variance, first, and last positions of sentences citing a reference.These positions are normalized against the total number of sentences, ranging from 0 at the beginning to 1 at the end.
  • Scope boundary: The feature set does not include section-aware location features, although methodology-section citations might be more influential than related-work citations.The authors identify such features as possible but do not consider them.

5. Miscellaneous features 5.1. aux citeCount 5.2. aux selfCite 5.3. aux yearDiff

Miscellaneous features capture a reference’s external citation count, self-citation status, and age relative to the citing paper. Feature values are normalized to support supervised learning, while binary features remain unchanged.

  • Miscellaneous features: The aux citeCount feature records how often each reference was cited in the general literature.This count concerns citations received outside the specific citing paper.
  • Miscellaneous features: The aux selfCite feature is a binary indicator of whether the citer and reference share at least one author.The study examines whether self-citation correlates positively or negatively with academic influence.
  • Miscellaneous features: The aux yearDiff feature is the non-negative difference between the citing paper’s publication date and the reference’s publication date.It tests whether older cited papers are more likely to be academically influential.
  • Normalization: Non-binary features are normalized to [0, 1], while binary features retain values 0 and 1.Normalization is contextual for paper–reference pairs and divides each value by the maximum for that feature in the citing paper; a zero maximum yields zero.

Experiments with features

The experiments use author-labeled references to evaluate citation features and build classifiers of academic influence. In-paper mention counts correlate most strongly with influence, and a four-feature model achieves the best reported performance.

  • Dataset: Authors labeled references as influential or non-influential to create a gold-standard dataset for supervised learning.The dataset was based on authors identifying references that were highly influential or inspirational for their papers.
  • Feature correlations: In-paper count features—countsInPaper whole and countsInPaper secNum—were the most correlated individual features with academic influence.countsInPaper whole counts reference mentions in a paper, while countsInPaper secNum counts the sections in which it appears.
  • Feature correlations: Citation counts had a positive but moderate correlation with influence, ranking below at least half of the tested features.The authors note that highly cited papers may be cited for reasons other than academic influence.
  • Feature correlations: References cited one or four to seven years earlier were positively correlated with influence, whereas papers from other age ranges were poorly or negatively correlated.The publication-year difference was discretized into ranges for the correlation analysis.
  • Classification results: The four-feature model achieved an F-measure of about 42% using two count-based features, sim titleCore, and aux selfCite.Greedy selection began with countsInPaper whole; adding more features beyond four did not improve performance.
  • Classification results: The peak F-measure for countsInPaper whole alone was 0.37 at a threshold corresponding to 13% of references predicted influential.The cross-validated single-feature model reached 0.35, compared with 0.37 when training on the full dataset without independent testing data.

Experiments with in-paper citation counts

The paper extends citation counting by weighting references according to how often they are mentioned in the citing paper, then evaluates the resulting influence-primed measures against conventional counts. Across ranking and ACL Fellow identification experiments, influence-primed measures differ meaningfully from conventional ones and achieve higher average precision in the reported Fellow-identification task.

  • Methods: In-paper citation counts are used to modify citation networks through edge filtering or weighting, with the experiments focusing on weighted edges.Filtering removes edges below an in-paper citation threshold; weighting converts the count of mentions into an edge weight.
  • Methods: The influence-primed citation count weights each citation edge by c^2 rather than 1, where c is the number of body-text mentions.This gives greater weight to references mentioned more often in the citing paper.
  • Methods: The influence-primed h-index, or hip-index, applies the same c^2 edge weighting and thresholds influence-primed citation counts rather than conventional citation counts.It is defined as the largest h for which at least h papers have influence-primed citation counts of at least h.
  • Paper and author rankings: For the top 100 papers, conventional and influence-primed citation counts correlate highly, but their correlation declines lower in the ranking.The two methods agree on the most highly ranked papers and disagree more among less cited papers.
  • ACL Fellow identification: Influence-primed counting differs from conventional counting non-trivially for author rankings, while the ACL Fellow experiment shows higher average precision for the hip-index.The influence-primed model is better for seven tested values of N, never worse through N = 17, and achieves AveP 14% versus 10% for the conventional model.

Future work and limitations

The paper identifies limits in its data, feature design, citation counting, and evaluation scope, and outlines validation and extension paths. It also notes unresolved challenges in detecting manipulated or difficult-to-interpret citation influence.

  • Validation: More extensive and different datasets are needed to validate the results, and author annotations were not tested for reliability.The authors suggest multiple annotators, repeated annotation after a delay, and comparison with human-level performance.
  • Feature scope: The feature set excludes full-text information from cited works and omits venue prestige, author prestige, and author-relationship features.The study intentionally uses only the cited paper’s title rather than recovering its full text.
  • Adversarial citation behavior: 20% of authors in a cited survey reported coercion to add references, making influential-citation identification potentially adversarial.The paper suggests that concentrations of non-influential citations from particular journals could indicate unethical behavior.
  • Evaluation scope: The study’s scientist-identification approach is simpler than state-of-the-art methods, which might benefit from influential-citation information.This limits the scope of the reported identification experiment rather than the proposed citation features themselves.
  • Counting mentions: Counting citation occurrences may require coreference resolution when references are mentioned without repeated formal citation markers.Some authors and editors cite a reference once but refer to it several additional times nominally or pronominally.
  • Annotation scope: The authors avoid ranking fine-grained influence levels because distinguishing and interpreting differences among weak references may be difficult even for human annotators.The present work uses a simpler influential versus non-influential distinction.

Conclusions

The conclusions favor weighting citations by their in-paper frequency and combining citation counts with semantic relatedness. The authors propose these weighted measures for assessing influence, tracking follow-up work, and recommending papers.

  • Conclusions: countsInPaper whole is one of the best predictors of a reference’s academic influence.The paper also considers countsInPaper secNum, which counts the sections in which a paper is cited.
  • Conclusions: Weighting citations by in-paper citation features would provide more robust results when assessing research papers or researchers.The authors also recommend weighted measures for tracking follow-up work and recommending research papers.
  • Conclusions: Combining countsInPaper with semantic relatedness produces a superior classifier, while self-citations provide only slight correlation with academic influence.The classifier can nevertheless benefit from self-citations when they are combined with other features.

Notes

The notes identify external resources, dataset and software links, experimental settings, and the ACL Fellows reference used in the experiments.

  • Notes: The General Inquirer and WordNet lexicons are identified as resources used or referenced by the paper.The notes provide download or access locations for both lexicons.
  • Notes: The notes provide links for citation-counting resources, influential-reference resources, the dataset, OpenNLP, and LIBSVM.They also record the LIBSVM parameters used in the experiments.
  • Notes: The ACL Fellows experiment used a list of seventeen fellows at the time, although the list has since grown.The note points readers to the ACL Fellows listing.
Loading 1501.06587v1…