PaperlayerView exact revision on arXiv

Source-linked AI summary

Predicting citation counts based on deep neural network learning techniques

Ali Abrishami, Sadegh Aliakbary

arXiv:1809.04365v3cs.DLcs.LGcs.SI

TL;DR

Predicting long-term citations is challenging because papers can follow very different citation trajectories. This paper trains a recurrent neural network on early citation patterns alone, and reports stronger prediction accuracy than state-of-the-art methods, including for highly cited papers.

  • Problem

    Predicting long-term citations is difficult because papers may gain attention late or lose citations over time, making a single simple rule inadequate.

    Source: §1 · p. 2

  • Method

    A customized recurrent neural network uses early citation counts to predict later yearly and total citations, without other paper or author features.

    Source: §1 · pp. 2–3

  • Results

    The proposed method outperformed state-of-the-art methods in prediction accuracy and in predicting highly cited papers.

    Source: §6 · p. 19

  • Takeaways & Limitations

    The authors present citation-pattern-only inputs as a simple, general setup applicable in other domains.

    Source: §1 · p. 2

    AVR and GMM require storing the full training dataset, and noisy training data may reduce their accuracy.

    Source: §5.4 · p. 17

Abstract

from arXiv · show

With the growing number of published scientific papers world-wide, the need to evaluation and quality assessment methods for research papers is increasing. Scientific fields such as scientometrics, informetrics and bibliometrics establish quantified analysis methods and measurements for scientific papers. In this area, an important problem is to predict the future influence of a published paper. Particularly, early discrimination between influential papers and insignificant papers may find important applications. In this regard, one of the most important metrics is the number of citations to the paper, since this metric is widely utilized in the evaluation of scientific publications and moreover, it serves as the basis for many other metrics such as h-index. In this paper, we propose a novel method for predicting long-term citations of a paper based on the number of its citations in the first few years after publication. In order to train a citations prediction model, we employed artificial neural networks which is a powerful machine learning tool with recently growing applications in many domains including image and text processing. The empirical experiments show that our proposed method out-performs state-of-the-art methods with respect to the prediction accuracy in both yearly and total prediction of the number of citations.

1. Introduction

The paper uses early citation patterns to predict a paper’s longer-term citation count, addressing the difficulty of forecasting diverse citation trajectories. It applies a customized recurrent neural network and reports effectiveness against existing baselines.

  • Citation counts are used to assess paper impact and underpin metrics including the h-index, impact factor, and i-10 index (Garfield, 1998; Moed, 2005; Oppenheim, 1995; Hirsch, 2005; Garfield, 2006; Wildgaard et al., 2014; Moed et al., 2012).Source: §1 · p. 2
  • The method uses citations from the first three to five years to predict later counts, such as citations from years five to fifteen, without author, journal, or paper-text features.Source: §1 · p. 2
  • Citation trajectories can include papers that attract attention only after years and papers whose citations decline, making a single rule or simple model inadequate (Ke et al., 2015).Source: §1 · p. 2
  • The customized recurrent neural network learns citation sequences, and the authors report that experiments show its effectiveness over state-of-the-art baselines.Source: §1 · pp. 2–3

2. Related Work

Prior impact-prediction research draws on varied data sources and targets, including citation networks, paper and author attributes, and short-term citations. This paper focuses on predicting long-term citations from short-term citation counts alone.

  • The literature also predicts highly cited papers, researcher h-index, and journal impact factor, in addition to paper citation counts (McNamara et al., 2013; Newman, 2014; Sarig¨ol et al., 2014; Ayaz et al., 2018; Dong et al., 2016; Acuna et al., 2012; Ketcham, 2007; Wu et al., 2008; Rocha-e Silva, 2016).Source: §2 · p. 3
  • Some prior approaches use scientific-paper graphs, including co-authorship networks, citation links, and paper-word networks (Sarig¨ol et al., 2014; Pobiedina and Ichise, 2016; Daud et al., 2017; B¨ut¨un et al., 2017; McNamara et al., 2013; Klimek et al., 2016).Source: §2 · p. 3
  • Other studies use information available at publication, such as paper content, venue, author properties, subject, and references (Dong et al., 2016; Mazloumian, 2012; Yan et al., 2012, 2011; Yu et al., 2014; Castillo et al., 2007; Bornmann et al., 2014).Source: §2 · pp. 3–4
  • Short-term citation counts have been used to predict long-term citations; Wang et al. (2013) fit a human-designed formula, while Cao et al. (2016) use similar papers’ citation patterns.Source: §2 · p. 4
  • The information-source categories overlap; some prior methods combine short-term citation counts with author properties.Source: §2 · p. 4
  • This paper uses only short-term citation counts to predict long-term citations, a setup the authors say remains applicable when input data are limited.Source: §2 · p. 4

3. Methodology

The methodology predicts later citation counts from a paper’s early citation history alone, framing the task as regression and evaluating both yearly and cumulative predictions. It uses a recurrent sequence-to-sequence network and compares it with three baselines.

3.1. Problem Statement

  • The task uses citations from years 0 through k to predict yearly counts through year n and their total over years k+1 through n.Accuracy is assessed for both individual-year predictions and the total predicted citations.Source: §3.1 · p. 5

3.2. Citation Count Prediction

  • A many-to-many recurrent sequence-to-sequence network maps early citation counts to later counts, feeding prior decoder predictions forward during sampling.Training uses actual later-year counts as targets; sampling feeds predicted counts into subsequent decoder steps.Source: §3.2 · pp. 6–8

3.3. Implementation Details

  • A 512-unit SimpleRNN with ReLU and 0.2 dropout uses RMSProp at 10^-5 and batches of 256 papers.Training runs for 100 epochs and tunes more than half a million parameters.Source: §3.3 · pp. 8–9

3.4. Baseline Methods

  • The baselines are MEY’s early-count average and Cao et al. (2016)’s same-journal nearest-paper AVR and GMM centroid methods.For AVR and GMM, cross-validation on Nature papers selected L = 20 among values below 100.Source: §3.4 · pp. 9–10

4. Data

The dataset draws citation histories from five prestigious journals for papers published from 1980 through 2002. Papers from 1980–1997 form the training set, with 1998–2002 papers reserved for testing.

4. Data

  • The Web of Science dataset covers Nature, Science, NEJM, Cell, and PNAS papers published from 1980 through 2002, with citation records through 2017.The dataset includes 14 years of citations for each paper.Source: §4 · p. 10
  • Papers published in 1980–1997 form the training set, and papers from 1998–2002 form the test set.Source: §4 · p. 10
  • Each paper record includes identifiers, journal, publication year, and yearly citation counts c0–c14; Table 4 shows five sampled papers’ citation-history features.Source: §4 · p. 10, Table 4 · p. 11

5. Results

NNCP outperforms the baselines when at least three years of citation history are available, using RMSE and R2 to assess predictions. Results also show a data-size qualification for highly cited papers and contrast NNCP with simpler baselines.

5.1. Measurement Criteria

  • RMSE measures variation between predicted and actual values, with lower scores preferred; R2 measures their correlation, with higher scores preferred.R2 = 0 indicates no correlation and R2 = 1 a perfect correlation.Source: §5.1 · p. 11

5.2. Evaluation Results

  • At k=5, NNCP beats all baselines on RMSE and R2 for yearly and total predictions across journals; it also wins at k=3, where all methods are less accurate.The figures present the k=5 and k=3 evaluations, respectively.Source: §5.2 · p. 13, Figure 4 · p. 14, Figure 5 · p. 15
  • NNCP most often gives the best predictions for highly cited papers at k=3 and k=5, but is not first for Cell and PNAS, which have fewer samples.The k=5 and k=3 figures show the fraction of samples for which each method achieved the best prediction.Source: §5.2 · p. 16, Figure 6 · p. 16, Figure 7 · p. 17

5.3. Sensitivity Analysis

  • NNCP outperforms baselines when at least three years of citation history are available, predicting citations from years 7–14 in the largest training and test sets used.The sensitivity experiment varies k from 0 to 7.Source: §5.3 · pp. 16–17, Figure 8 · p. 18

5.4. Discussion

  • MEY can be effective for highly cited papers in small datasets such as Cell and PNAS; AVR and GMM require storing the full dataset, while NNCP is reported faster and more accurate.AVR and GMM use lazy learning; NNCP learns a model from the training data.Source: §5.4 · pp. 17, 19

6. Conclusion

The conclusion presents an artificial-neural-network method that learns citation patterns and reports stronger prediction accuracy and highly cited-paper prediction than state-of-the-art methods. It also identifies deep-learning design challenges and proposes future extensions.

6. Conclusion

  • Comprehensive evaluations show the proposed method outperforms state-of-the-art citation-count methods in prediction accuracy and ability to predict highly cited papers.Source: §6 · p. 19
  • The method uses RNNs and sequence-to-sequence models to learn citation-pattern sequences from papers’ early publication years.Source: §6 · p. 19
  • Applying deep learning requires mapping citation prediction to a learning problem, designing experiments, selecting evaluation criteria and architecture, and tuning hyperparameters.Source: §6 · p. 19
  • Future work will add input features and deeper hidden layers, and investigate predicting highly cited papers and authors and estimating authors’ future h-index.Source: §6 · p. 19
Loading 1809.04365v3…