Source-linked AI summary
Content-Based Citation Recommendation
Chandra Bhagavatula, Sergey Feldman, Russell Power, Waleed Ammar
TL;DR
Citation recommendation must remain useful when query-document metadata or citation lists are unavailable. The paper embeds document text to select nearby candidates and reranks them with a discriminative model, achieving state-of-the-art results without metadata while releasing a large dataset and web tool.
Problem
Citation recommendation needs to support literature review when metadata or citation lists for query documents are unavailable, including during peer review or early research.
Method
The method embeds document text, selects nearest-neighbor candidates, and reranks them with a model trained to distinguish observed from unobserved citations.
Results
The method achieves state-of-the-art results on two citation-recommendation datasets using textual content without metadata.
Takeaways & Limitations
The paper provides a content-based citation recommender robust to missing metadata, plus an online system and a seven-million-article dataset for future research.
Takeaways & Limitations
Candidate selection requires a pool substantially larger than a document’s citation count but substantially smaller than the corpus.
Abstract
from arXiv · showhide
We present a content-based method for recommending citations in an academic paper draft. We embed a given query document into a vector space, then use its nearest neighbors as candidates, and rerank the candidates using a discriminative model trained to distinguish between observed and unobserved citations. Unlike previous work, our method does not require metadata such as author names which can be missing, e.g., during the peer review process. Without using metadata, our method outperforms the best reported results on PubMed and DBLP datasets with relative improvements of over 18% in F1@20 and over 22% in MRR. We show empirically that, although adding metadata improves the performance on standard metrics, it favors self-citations which are less useful in a citation recommendation setup. We release an online portal (http://labs.semanticscholar.org/citeomatic/) for citation recommendation based on our method, and a new dataset OpenCorpus of 7 million research articles to facilitate future research on this task.
1 Introduction
Citation recommendation addresses the difficulty of comprehensive literature review when query-document metadata or citation lists are unavailable. The paper proposes a content-based, scalable method and reports strong results without metadata, alongside a large dataset and web tool.
- Rapid scientific-literature growth makes comprehensive literature review challenging, motivating citation recommendation for improving review quality and efficiency.
- Existing systems may rely on author names, publication venues, or partial citation lists that are unavailable during peer review or early research.
- The method embeds document text, selects nearest-neighbor candidates, and reranks them with a model distinguishing observed from unobserved citations.
- Using only paper text, the method outperforms reported results on two datasets despite missing query-document metadata.
- The paper contributes a robust content-based method, large improvements without metadata, a seven-million-paper dataset, and a scalable web-based literature-review tool.
2 Overview
The system formulates citation recommendation as ranking and divides inference into fast candidate selection followed by feature-rich reranking. This reduces the need to score every document while retaining a manageable pool for detailed scoring.
- Citation recommendation ranks corpus documents that should be referenced in a query document above other documents using standard retrieval metrics.
- The two-phase pipeline combines fast, recall-oriented candidate selection with feature-rich, precision-oriented reranking.
- Candidate selection embeds the corpus once, projects each query into the same space, and identifies nearby documents for further analysis.
- The candidate pool should be much larger than a document’s citation count but much smaller than the corpus to balance reference coverage and computational cost.
- Reranking scores each candidate-query pair with a neural classifier, sorts candidates by estimated citation probability, and returns the top recommendations.
3 Phase 1: Candidate Selection (NNSelect)
NNSelect embeds documents from textual content and retrieves nearby candidates, augmented with citations of those neighbors. Its embedding model is trained to bring cited documents closer than uncited documents.
- NNSelect selects nearest neighbors of a query embedding and adds outgoing citations from those neighbors to expand the candidate pool.
- The phase outputs candidates scored by cosine similarity between query and document embeddings.
- The document embedding model represents textual fields with bag-of-words features and projects documents into dense embeddings.
- Each field representation is normalized, then fields are combined by a weighted average; experiments use titles and abstracts.
- Training uses query, cited-document, and uncited-document triplets so cited pairs receive higher cosine similarity than uncited pairs.
- Negative examples include random uncited documents, uncited nearest neighbors, and documents cited by positive citations but not directly by the query.
4 Phase 2: Reranking Candidates (NNRank)
NNRank reranks candidate citation pairs with a neural model that combines field similarities and additional document features to estimate citation probabilities.
- Model purpose: NNRank estimates the probability that candidate document d_i should be cited in query document d_q.The model takes document-pair features as input and uses a sigmoid output.
- Input features: The model combines cosine similarities across document fields, numeric features, and summed weights of words shared by the documents.Fields include title, abstract, authors, venue, and keyphrases when available.
- Input features: The input includes the query-candidate embedding similarity and the candidate's log corpus in-citation count.The embedding similarity is computed as cos-sim(e_dq, e_di).
- Network architecture: Two ELU layers and one sigmoid layer form the FeedForward network used to produce the citation score.The output sigmoid estimates the probability that d_q cites d_i.
- Training and inference: During training, NNRank parameters are learned with the triplet loss after redefining similarity as the model's sigmoid output.At test time, candidates with the highest s(d_q, d_i) scores are recommended.
5 Experiments
Experiments compare citation-recommendation baselines with three variants of the proposed method, test feature and sampling choices, and analyze metadata bias. The content-based variants outperform prior methods without metadata, while metadata produces the best overall standard-metric results but favors self-citations.
- Main results: The experiments evaluate BM25, ClusCite, NNSelect, NNSelect + NNRank, and NNSelect + NNRank + metadata using F1@20 and MRR.The results include mean and standard deviation over five trials where available.
- Main results: NNSelect alone outperforms both baselines by ranking candidates according to cosine similarity in the text-derived embedding space.Although designed for candidate selection, the embedding space also supports direct citation recommendation.
- Main results: NNSelect + NNRank significantly outperforms previous methods without encoding venues, authors, or keyphrases.The reranking model discriminates between observed and unobserved citation pairs for candidates selected by NNSelect.
- Main results: Over 20% relative improvement in F1@20 and over 25% in MRR is achieved on both DBLP and PubMed versus ClusCite.The metadata variant gives the best overall results on the reported datasets.
- Controlled experiments: Adding negative nearest neighbors improves PubMed F1@20 from 0.306 to 0.329 and MRR from 0.705 to 0.771.These negatives focus training on harder cases where the model is more likely to make mistakes.
- Controlled experiments: Intersection features and numerical features are especially valuable, while recurrent and convolutional text encoders are slower without significant F1@20 or MRR gains.Intersection features improve PubMed F1@20 from 0.296 to 0.329 and MRR from 0.653 to 0.771; numerical features improve them from 0.314 to 0.329 and 0.735 to 0.771.
- Metadata analysis: Metadata lowers mean and maximum prediction ranks because the metadata-trained model tends to favor documents authored by a query author, indicating self-citation bias.This pattern holds as the number of predictions varies from 1 to 100.
6 Related Work
Related work includes local and global citation recommendation, metadata- and graph-based methods, and text-based document representations. The paper addresses concerns about missing metadata, transductive representations, scalability, dataset pruning, reproducibility, and cross-dataset variation through content-based modeling and OpenCorpus.
- Citation recommendation settings: Citation recommendation systems are categorized as local systems using a few sentences and global systems using the entire scholarly article.The paper positions its approach among global citation recommendation methods.
- Metadata dependence: The proposed content-based method remains usable when query metadata such as authors, venues, keyphrases, or seed citations is unavailable.This contrasts with prior systems that depend on metadata or an existing citation list.
- Scalability: Graph-based citation models can be expensive at scale, whereas the paper uses a neural network trained with batched stochastic gradient descent for large datasets.ClusCite training complexity is cubic in the number of graph edges.
- Document representation: Word-based document representations allow new documents to be embedded from their text, unlike transductive representations learned separately for training documents.Some prior methods also disadvantage newly published candidates by requiring incoming citations.
- Evaluation resources: OpenCorpus addresses concerns about strongly pruned evaluations, limited re-implementation details, and performance variation across datasets.The paper also releases code, extensive system details, and experiments across multiple datasets.
- Hybrid representations: Combining graph and text representations may improve text-based representations when graph representations can be created at test time.This connection is identified as a direction in related work rather than as a component of the proposed method.
7 Conclusion
The paper presents a content-based citation recommendation method that remains effective when query-document metadata is unavailable, reports state-of-the-art results, and releases an online system and a seven-million-article dataset. Reproducibility details describe selective hyperparameter tuning, model settings, and training choices across datasets.
- 7 Conclusion: The proposed content-based citation recommendation method remains robust when metadata is missing from query documents.The authors highlight use during early literature search and peer review, where metadata may be unavailable.
- 7 Conclusion: The method achieves state-of-the-art results on two citation recommendation datasets without using metadata available to the baseline method.
- 7 Conclusion: The authors publicly release an online citation recommendation system and introduce OpenCorpus, a dataset containing seven million scientific articles.
- 7 Conclusion: Hyperparameter tuning used 75 Hyperopt trials, short five-epoch runs, and full training of the ten best models on DBLP and PubMed.NNSelect was optimized for Recall@20, while NNRank was optimized for F1@20 on development data.
- 7 Conclusion: OpenCorpus hyperparameters were selected through informal hand-tuning, with additional untuned parameters reported for reproducibility.The implementation notes also define the margin multiplier, Siamese embeddings, and pretrained word embeddings used in the experiments.
B Margin Loss Details
The margin-loss formulation uses a margin multiplier and negative-type-specific margins, with a citation-frequency boosting function that slightly favors frequently cited candidates.
- B Margin Loss Details: The margin multiplier γ and the negative-dependent α(d−) determine the margins used in the triplet loss.
- B Margin Loss Details: A boosting function varies the margin according to the type of negative document.
- B Margin Loss Details: The boosting function uses citation frequency and slightly favors candidates cited more frequently in the corpus.Its constants were set without optimization.
C Nearest Neighbors for Training Details
Training negatives are selected by combining embedding-space nearest neighbors with textual Jaccard similarity, retaining candidates that are sufficiently textually dissimilar to true citations.
- C Nearest Neighbors for Training Details: The heuristic seeks non-citation samples that are dissimilar in the original text but similar in embedding space.
- C Nearest Neighbors for Training Details: The procedure computes Jaccard similarities between each training query and all its true citations.Similarity uses concatenated title and abstract text.
- C Nearest Neighbors for Training Details: It sets a threshold at the bottom fifth percentile of true-citation Jaccard similarities.This threshold lies below the 5% least textually similar true citations.
- C Nearest Neighbors for Training Details: The method fetches nearest neighbors from the previous epoch’s Annoy index and computes their textual Jaccard similarity with the query.
- C Nearest Neighbors for Training Details: Nearest neighbors are retained when their Jaccard similarity is below the fifth-percentile threshold.For an example threshold of 0.3, candidates below 0.3 are retained.
D BM25 Details
The BM25 baseline uses key terms extracted from query-document titles and abstracts to construct queries, while OpenCorpus key phrases are extracted and ranked automatically.
- D BM25 Details: BM25 is used as an information-retrieval baseline for citation recommendation.The implementation is evaluated on DBLP, PubMed, and OpenCorpus.
- D BM25 Details: For each query document, key terms are extracted from the title and abstract and concatenated into a BM25 query string.
- D BM25 Details: The OpenCorpus key-phrase pipeline first extracts noun phrases as candidate key phrases.
- D BM25 Details: Candidate key phrases are ranked using corpus-level and document-level features and a model trained on author-provided key phrases.
F Detailed Experimental Results
NNRank is compared with prior citation-recommendation systems on DBLP and PubMed, where it consistently outperforms ClusCite and other baselines across all reported metrics. The section also documents dataset-specific tuning and OpenCorpus hyperparameters.
- Comparison with prior work: NNRank consistently outperforms ClusCite and other baselines on every reported metric across both DBLP and PubMed.ClusCite is identified as the previous best system on these datasets.
- Hyperparameter tuning: DBLP and PubMed receive separate hyperparameter-tuning tables, including fixed 300-dimensional pretrained-vector settings.The DBLP table fixes dense dimension for pretrained vectors, and the PubMed table does so for pretrained GloVe vectors.
- Hyperparameter settings: Additional tables list hand-specified per-dataset parameters and the hyperparameters used for OpenCorpus.The per-dataset parameter table identifies LazyAdamOptimizer and Nadam as framework-provided components.
- Comparison with prior work: The comparison includes ClusCite, BM25, and L2-LR as the principal reported baselines.ClusCite uses a heterogeneous graph, while L2-LR uses meta-path-based linear features.