PaperlayerView exact revision on arXiv

Source-linked AI summary

SPECTER: Document-level Representation Learning using Citation-informed Transformers

Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, Daniel S. Weld

arXiv:2004.07180v4cs.CL

TL;DR

Existing pretrained language models focus on token- and sentence-level objectives and do not use inter-document relatedness, limiting their document-level representations. SPECTER pretrains a Transformer with citation-graph supervision and applies the resulting embeddings without task-specific fine-tuning. It substantially outperforms competitive baselines across document-level tasks, while SCIDOCS provides a seven-task benchmark.

  • Problem

    Pretrained language models primarily use intra-document objectives and lack inter-document information needed for strong document-level scientific-paper representations.

  • Method

    SPECTER encodes titles and abstracts with a Transformer and pretrains it using citation-based triplet loss to learn document-level embeddings.

  • Results

    SPECTER substantially outperforms competitive baselines across document-level tasks, including topic classification, citation prediction, and recommendation.

  • Takeaways & Limitations

    SCIDOCS introduces a seven-task evaluation suite for scientific document embeddings, and SPECTER embeddings can be used downstream without task-specific fine-tuning.

  • Takeaways & Limitations

    SPECTER uses only titles and abstracts; incorporating full paper text could provide a more complete representation but requires addressing access and model input-length constraints.

Abstract

from arXiv · show

Representation learning is a critical ingredient for natural language processing systems. Recent Transformer language models like BERT learn powerful textual representations, but these models are targeted towards token- and sentence-level training objectives and do not leverage information on inter-document relatedness, which limits their document-level representation power. For applications on scientific documents, such as classification and recommendation, the embeddings power strong performance on end tasks. We propose SPECTER, a new method to generate document-level embedding of scientific documents based on pretraining a Transformer language model on a powerful signal of document-level relatedness: the citation graph. Unlike existing pretrained language models, SPECTER can be easily applied to downstream applications without task-specific fine-tuning. Additionally, to encourage further research on document-level models, we introduce SciDocs, a new evaluation benchmark consisting of seven document-level tasks ranging from citation prediction, to document classification and recommendation. We show that SPECTER outperforms a variety of competitive baselines on the benchmark.

1 Introduction

SPECTER addresses the limited document-level representation power of pretrained language models by incorporating inter-document citation context. The paper also introduces SCIDOCS, a seven-task benchmark for evaluating scientific document embeddings.

  • Pretrained language models primarily represent words or sentences, while whole-document embeddings and inter-document signals remain underexplored.
  • Off-the-shelf pretrained models do not produce accurate representations for document-level tasks such as topic classification or recommendation.
  • SPECTER incorporates citation context into a Transformer and uses citation-based triplet-loss pretraining for general-purpose scientific document representations.
  • At inference, SPECTER requires no citation information, allowing embeddings for newly published papers that have not yet been cited.
  • SPECTER substantially outperforms state-of-the-art baselines across topic classification, citation prediction, and recommendation tasks.
  • SCIDOCS provides seven document-level tasks and releases datasets, training data, a trained embedding model, and code.

2 Model

SPECTER encodes paper titles and abstracts with a Transformer, then pretrains the representation using citation-informed triplet loss. Citation positives and harder related negatives shape embeddings that can be used without citation data at inference.

  • SPECTER aims to learn task-independent academic-paper representations by combining Transformer encoding with citation-based inter-document supervision.
  • Document Representation: The paper representation is the Transformer’s pooled [CLS] output from the concatenated title and abstract.
  • Citation-Based Pretraining Objective: Citation-based triplets contain a query paper, a cited positive paper, and an uncited negative paper, encouraging cited papers to have closer representations.
  • Citation-Based Pretraining Objective: The training objective uses L2 distance in a triplet margin loss with an empirically chosen margin of m = 1.
  • Selecting Negative Distractors: Hard negatives are uncited papers cited by a paper cited by the query, and including them produces more accurate embeddings than random negatives alone.
  • Inference: At inference, SPECTER uses only the input paper’s title and abstract, so it can embed papers without citation information.

3 SCIDOCS Evaluation Framework

SCIDOCS is a larger, diverse evaluation framework for scientific paper embeddings, covering citation, user-activity, classification, and recommendation tasks. Its datasets evaluate whether embeddings recover multiple forms of document relatedness without task-specific fine-tuning.

  • SCIDOCS addresses prior evaluations’ small datasets, limited task coverage, and already-high AUC scores by providing a larger and more diverse benchmark.
  • The framework spans citation prediction, user-activity prediction, document classification, and paper recommendation.
  • Document Classification: Classification tasks evaluate scientific papers using disease classes and predefined Microsoft Academic Graph topics.
  • Citation Prediction: Citation evaluation ranks directly cited or highly co-cited papers above uncited or other candidate papers using MAP and nDCG.
  • User Activity: User-activity tasks test whether embeddings rank frequently co-viewed and co-read papers as more similar than randomly selected papers.
  • Recommendation: The recommendation task uses 22K scholarly-search clickthrough events and adds embedding distance as a feature in an existing recommendation system.

4 Experiments

The experiments compare SPECTER with textual, citation-mining, and graph-learning baselines across document-level tasks. Training uses citation-linked paper triples, SciBERT initialization, and task-specific downstream models.

  • Training Data: The model trains on query papers, cited positive papers, and negative papers sampled from the Semantic Scholar corpus.The training subset contains about 146K query papers and uses up to five triples per query.
  • Training and Implementation: SPECTER is initialized from SciBERT and continues training all model parameters on the citation-based objective.Hyperparameters receive minimal tuning on validation data, while baselines are extensively tuned.
  • Task-Specific Model Details: Classification uses embedding vectors as the only features in a linear SVM, with C tuned on a held-out validation set.
  • Task-Specific Model Details: Recommendation uses a feedforward ranking network combining embedding similarity with citation, title, author, and publication-date features.
  • SPECTER is compared with textual, citation-mining, and graph-learning baselines, including SIF, SciBERT, Sent-BERT, Citeomatic, and SGC.

5 Results

SPECTER improves performance across classification, user-activity prediction, citation-related tasks, and recommendation, while citation-based pretraining is critical to these results. Its online recommender also improves clickthrough over the production baseline.

  • 80.0 average performance across all metrics and tasks is a 3.1-point absolute improvement over the next-best baseline.
  • Document Classification: 86.4 F1 on MeSH and 82.0 F1 on MAG improve over the best baselines by 2.3 and 1.5 points, respectively.
  • User Activity: 83.8 MAP on co-view and 84.5 MAP on co-read improve over Citeomatic by 2.7 and 4.0 points, respectively.
  • Citation and Co-Citation: SPECTER achieves 94.8 nDCG on co-citation, improving over SGC by 2.3 points and over Citeomatic by 2.0 nDCG.SPECTER slightly outperforms Citeomatic on direct citation prediction, while substantially outperforming it on co-citations.
  • Recommendation: 53.9 nDCG makes SPECTER the best model on recommendation, although score differences are smaller because the ranker also uses other informative features.
  • Online Study: 46.5% higher clickthrough in 4,113 clicks shows an online advantage over the existing production recommender.
  • Ablation: Removing citation-based pretraining and using vanilla SciBERT decreases performance on all tasks.

6 Analysis

SPECTER’s analysis examines input metadata, hard negatives, embedding geometry, and task-specific fine-tuning. Fixed SPECTER representations generally outperform alternatives while requiring less task-specific adaptation.

  • Ablation Study: Removing abstracts substantially decreases performance, while adding authors hurts performance across the evaluated tasks.Author names may be sparse in the corpus, making them difficult for the model to use effectively.
  • Ablation Study: Venue information slightly decreases performance overall, except on document classification, where venues correlate with paper topics.The authors note that venue metadata was not normalized, which could affect results.
  • Ablation Study: Hard negatives are important: using only easy negatives reduces performance on all tasks.The hard negatives are constructed from citations of citations, providing a more challenging training signal.
  • Visualization: SPECTER embeddings produce more compact topical clusters than SciBERT in the t-SNE visualization.The projected embeddings also show related topics, including Engineering, Mathematics, and Computer Science.
  • Visualization: SPECTER’s projected embeddings achieve homogeneity 0.41 and completeness 0.72, versus SciBERT’s 0.19 and 0.63.These values indicate improved topic separation on the two-dimensional projection.
  • Comparison with Task Specific Fine-Tuning: SPECTER fixed representations outperform SciBERT fine-tuned on task-specific data and its multitask combination without additional task-specific fine-tuning.Further task-specific fine-tuning of SPECTER did not yield additional improvements.

7 Related Work

Related work spans pretrained language models, document and graph representation learning, and network-structured methods. SPECTER combines pretrained language modeling with citation signals and is evaluated against these approaches using document-level benchmarks.

  • Pretrained Language Models: Pretrained language models have succeeded at sentence- and token-level tasks, while document-level representation learning remains relatively under-explored.Existing language-model objectives primarily use intra-document context rather than inter-document relatedness.
  • Document Representation Learning: Prior document representations include word-vector extensions, convolutional methods, variational autoencoders, and sentence-embedding approaches.These approaches represent documents or sentences but generally do not incorporate inter-document relatedness.
  • Network-Structured Methods: Methods combining textual features with network structure typically do not leverage recent pretrained contextual representations or generalize to unseen documents.SPECTER instead uses paper title and abstract text with citation-based signals.
  • Network-Structured Methods: SPECTER combines pretrained language models with graph-based citation signals and outperforms graph-based approaches in the authors’ experiments.Its input remains only the paper title and abstract, distinguishing it from citation-context recommendation methods.
  • Evaluation Resources: SCIDOCS addresses limited public recommendation data by releasing an evaluation benchmark with an anonymized clickthrough dataset.The framework covers citation prediction, user activity prediction, document classification, and paper recommendation.

8 Conclusions and Future Work

The paper presents SPECTER, a citation-pretrained Transformer for scientific-paper representations, and introduces SCIDOCS for evaluating document-level embeddings. It reports improvements over diverse baselines while identifying several directions for future work.

  • Conclusions: SPECTER learns scientific-paper representations with a Transformer language model pretrained on citations.The citation signal supplies inter-document supervision for representation learning.
  • Conclusions: The authors report substantial improvements over a wide variety of baselines.The conclusion characterizes these improvements as evidence of SPECTER’s effectiveness.
  • Conclusions: SCIDOCS is an evaluation suite with seven document-level tasks and released datasets intended to support further research.The suite accompanies the model contribution as a benchmark for scientific-paper embeddings.
  • Future Work: Future work includes initializing SPECTER from newer Transformer models, improving multitask use of relatedness signals, and exploring additional bibliometric graphs or citation inputs.The authors also identify full-paper modeling as a broader future direction elsewhere in the paper.

A Appendix A - Baseline Details

The appendix describes baseline representations used for comparison, including random vectors, Doc2Vec, and weighted sums of pretrained word vectors. The word-vector baseline uses tuned weighting options and scientific-domain fastText embeddings.

  • Random: The random baseline represents each document with a random zero-mean 25-dimensional vector.This provides a simple non-semantic reference representation.
  • Doc2Vec: Doc2Vec is included as an earlier neural document-representation baseline trained on the paper training subset.The implementation uses Gensim and a hyperparameter grid selected from prior suggestions.
  • Doc2Vec: The Doc2Vec search evaluates 27 models across sample and epoch settings, with 300-dimensional vectors and specified training parameters.The reported grid spans sample values 0, 10^-6, and 10^-5 and epoch values 50, 100, and 200.
  • Word-Vector Baseline: The bag-of-words baseline sums pretrained word vectors using weighting options tuned on a validation set.The authors train 300-dimensional fastText embeddings on approximately 3.1B scientific-paper tokens.

3. Fasttext-Sum

The paper compares SIF, ELMo, Citeomatic, SGC, and SciBERT as document-representation baselines. These methods derive representations from weighted word vectors, contextualized token states, citation-graph training, graph propagation, or Transformer pooling.

  • ELMo: ELMo creates document embeddings by averaging token representations across all three LSTM layers of a 768-dimensional pretrained model.
  • Citeomatic: Citeomatic learns 75-dimensional L2-normalized weighted sums of title and abstract embeddings from citation-graph triplets with dynamic negative sampling.
  • SGC: SGC applies citation-graph propagation to SIF node features, multiplying the normalized adjacency by the feature matrix k times for k from 1 through 8.The resulting baseline is 300 dimensional, and k was chosen with a validation set.
  • SciBERT: SciBERT uses the final-layer [CLS] hidden state as the aggregate document representation in a controlled comparison isolating SPECTER’s citation-based fine-tuning objective.Averaging all token representations slightly decreased performance relative to the [CLS] pooled token.

8. SciBERT

The paper uses SciBERT as a controlled Transformer baseline and compares it with Sentence BERT’s sentence-oriented document construction. The SciBERT representation is taken from the final [CLS] token because averaging token states performed slightly worse.

  • 8. SciBERT: Averaging all token representations produced a slight performance decrease compared with using the [CLS] pooled token.
  • Sentence BERT: Sentence BERT encodes the title and each abstract sentence separately, then sums the embeddings and L2 normalizes the result.The underlying model was pretrained for sentence embeddings rather than paragraph embeddings.
Loading 2004.07180v4…