PaperlayerView exact revision on arXiv

Source-linked AI summary

A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networks

Chanwoo Jeong, Sion Jang, Hyuna Shin, Eunjeong Park, Sungchul Choi

arXiv:1903.06464v1cs.CLcs.IRcs.LG

TL;DR

The paper addresses the lack of organized benchmarks for context-aware citation recommendation, where surrounding text and paper metadata are used to select references. It builds reprocessed datasets and a BERT–GCN model, reporting state-of-the-art performance and significant improvements over an existing model, while comparisons rely on stated frequency assumptions.

  • Problem

    Context-aware citation recommendation lacks well-organized benchmarking datasets that jointly support citation context, metadata, and consistent evaluation.

  • Method

    The study reprocesses AAN and PeerRead into citation-context datasets and combines BERT-based context encoding with GCN-based citation-network representation.

  • Results

    The proposed model delivers significant improvement in MAP, MRR, and Recall@K over the existing model and is reported as state of the art.

  • Takeaways & Limitations

    FullTextPeerRead provides updated papers with organized citation contexts and metadata for context-aware citation recommendation research.

  • Takeaways & Limitations

    The baseline comparison assumes a citation frequency of one for CACR because the original paper does not report its frequency, and the proposed comparison underperforms after Recall@10.

Abstract

from arXiv · show

With the tremendous growth in the number of scientific papers being published, searching for references while writing a scientific paper is a time-consuming process. A technique that could add a reference citation at the appropriate place in a sentence will be beneficial. In this perspective, context-aware citation recommendation has been researched upon for around two decades. Many researchers have utilized the text data called the context sentence, which surrounds the citation tag, and the metadata of the target paper to find the appropriate cited research. However, the lack of well-organized benchmarking datasets and no model that can attain high performance has made the research difficult. In this paper, we propose a deep learning based model and well-organized dataset for context-aware paper citation recommendation. Our model comprises a document encoder and a context encoder, which uses Graph Convolutional Networks (GCN) layer and Bidirectional Encoder Representations from Transformers (BERT), which is a pre-trained model of textual data. By modifying the related PeerRead dataset, we propose a new dataset called FullTextPeerRead containing context sentences to cited references and paper metadata. To the best of our knowledge, This dataset is the first well-organized dataset for context-aware paper recommendation. The results indicate that the proposed model with the proposed datasets can attain state-of-the-art performance and achieve a more than 28% improvement in mean average precision (MAP) and recall@k.

1 Introduction

Context-aware citation recommendation selects suitable papers for citation placeholders using surrounding text and paper metadata. The study addresses fragmented benchmarking and proposes reproducible datasets and a BERT–GCN model for the task.

  • Task definition: Context-aware citation recommendation selects a suitable scientific document for a citation placeholder using surrounding sentences and paper metadata.Metadata may include authors, titles, citation information, and journal or conference names.
  • Research gap: Existing datasets do not consistently provide the context sentences and metadata needed for benchmarking.AAN lacks preprocessed sentences and metadata, DBLP provides only bibliographic information, and CiteseerX lacks metadata alongside context and citation information.
  • Research gap: The absence of a shared benchmarking dataset has prevented related studies from using the same evaluation basis.
  • Contributions: The study constructs reproducible AAN and FullTextPeerRead datasets and a model combining BERT for textual context with GCN for graph-based metadata.The authors also investigate factors affecting task performance through experiments.

2 Proposed Dataset

The proposed datasets reprocess AAN and PeerRead to combine citation context with bibliometric information. Context is extracted from converted papers and linked to cited-paper metadata, with manual cleaning applied to noisy records.

  • Dataset overview: AAN and PeerRead were reprocessed to create datasets containing citation context together with bibliometric information.FullTextPeerRead expands the PeerRead dataset, while AAN was revised for the task.
  • Data acquisition: The acquisition pipeline converts LaTeX-based papers to HTML, detects citation symbols with regular expressions, and stores adjacent sentences with reference-paper metadata.
  • Dataset statistics: The AAN dataset cannot be disclosed under its policy, while the authors state that it may be opened after receiving a grant.
  • Data acquisition: The collected data required manual post-processing because inconsistent LaTeX formats produced noisy records and could leave placeholder text that enabled overfitting.
  • Dataset statistics: Dataset statistics distinguish base papers, cited papers, citation contexts, and total papers after excluding unusable or noisy PDFs.The extracted datasets are smaller than the original AAN or PeerRead collections because some PDFs could not be processed reliably.

3 A BERT-GCN Model for Context-Aware Citation Recommendation

The proposed model combines BERT-based textual context encoding with GCN-based citation-graph encoding, then uses their concatenated representations for citation recommendation. Its graph component uses VGAE to learn paper representations and reconstruct citation relationships.

  • Model overview: BERT encodes citation-context text, while GCN represents citation relationships and extracts graph-based paper embeddings.The two encoders are pretrained separately before their outputs are concatenated.
  • Model overview: The concatenated encoder embeddings pass through a feedforward network and softmax output layer, trained with cross entropy.
  • Relation to CACR: Compared with CACR, the proposed citation encoder uses citation information through GCN rather than relying on author, venue, and abstract metadata.
  • Citation encoder: The citation encoder uses GCN-based VGAE with citation relationships as input to learn latent representations of papers.VGAE performs unsupervised learning for citation-link prediction.
  • Graph Convolutional Network layer: The GCN inference model abstracts citation-network information through two propagation layers using identity features X and adjacency matrix A.A is an N by N matrix, where N is the number of input papers.
  • Variational Graph AutoEncoder: VGAE’s generative layer predicts an adjacency matrix from latent paper vectors using their inner products, reducing the difference from the actual adjacency matrix.The latent variables zi and zj correspond to paper representations used for adjacency generation.

4.1 Experiments overview

The experiments compare the proposed model with CACR on AAN and FullTextPeerRead using information-retrieval metrics. They examine overall performance, the contributions of BERT and GCN, practical left-context recommendation, text length, and citation frequency.

  • Overall comparison: The proposed model is compared with CACR, an existing state-of-the-art context-aware citation recommendation model.
  • Overall comparison: Experiments use AAN and FullTextPeerRead and evaluate Mean Average Precision, Mean Reciprocal Rank, and Recall@K.
  • Component analysis: A component analysis investigates performance differences between models using BERT for textual data and GCN for graph data.
  • Practical context: The study tests whether papers can be recommended using only the text written before a citation symbol.This setting reflects recommendation during ongoing paper writing without the entire document.
  • Robustness analyses: Additional analyses examine whether citation-context length introduces noise and how performance varies for rarely cited papers.

4.2 Experiments setting

The experiments use two datasets with different publication cutoffs and evaluate recommendation quality from citation contexts of specified lengths. Performance is measured with MAP, MRR, and Recall Top@K across multiple recommendation-list sizes.

  • Datasets: AAN contains papers published before 2014, whereas FullTextPeerRead contains papers published before 2018.
  • Datasets: After context and metadata processing, AAN yielded 6,500 papers and FullTextPeerRead yielded 4,898 papers.
  • Context construction: The datasets were divided into two parts, with contexts built around citation placeholders using left-side or two-sided text windows.The supplied passage specifies a 100-text-length left context and a paired context with 100 text length on each side.
  • Evaluation metrics: MAP, MRR, and Recall Top@K were used as information-retrieval evaluation metrics.MAP was measured with K=30, while Recall was evaluated at K = 5, 10, 30, 50, 80, and 100.
  • Implementation: The BERT configuration used 12 attention heads, 12 encoder layers, 30 epochs, batch size 16, maximum sequence length 128, and hidden size 768.Optimization used Adam with learning rate 2e-5.
  • Implementation: The GCN configuration used 200 epochs, a first hidden dimension matching document size, a second hidden dimension of 768, full-batch gradient descent, and Adam with learning rate 0.01.

4.3 Experiments results

The experiments compare the proposed model with CACR and examine how citation frequency, context length, and GCN affect recommendation performance. Results report strong gains over the existing model, while showing that longer contexts eventually have reduced impact.

  • Baseline comparison: Approximately three times higher MAP, MRR, and Recall@K performance was reported for the proposed models compared with the SOTA CACR model.The comparison used papers with at least five citations and fifty words on both sides of the citation symbol.
  • Baseline comparison: The reproduced CACR comparison assumes citation frequency one because the original paper did not specify its frequency setting.Under this comparison, the proposed model outperformed CACR for MAP, MRR, and Recall@10 but underperformed beyond Recall@10.
  • Impact of BERT and GCN: Adding GCN improved performance in all tested cases, with a greater impact when the input contained fewer context words.BERT-left uses half the context-sentence input of BERT, and its GCN-enhanced variant showed a larger recall difference.
  • Context length and direction: Using only left-side context reduced performance by approximately 0.03 compared with using the full context on both sides of the citation marker.The experiment therefore reports an advantage for incorporating both sides of the citation context.
  • Context length and direction: When context length reached 100 or more words, comparative context length had less effect on model performance.The results indicate that context remains relevant, but its impact is reduced beyond a certain length.
  • Citation frequency: Performance improved as citation frequency increased from one to three and five citations.The authors note that uncited papers are excluded from learning and may remain sparse during testing.

4.4 Recommendation Examples

Recommendation examples show that GCN can improve the relevance of cited-paper recommendations beyond textual similarity alone. The model also uses citation information from related papers and the current article.

  • Recommendation examples: GCN produced higher relevance similarity for cited-paper recommendations than textual data alone in the reported examples.The examples identify this as one of the biggest effects of adding GCN.
  • Recommendation examples: The graph representation incorporates citation relationships involving the citing paper, previous papers by the author, and cited papers in the current article.The authors describe this citation information as helpful for improving context-aware citation recommendation.
  • Recommendation examples: Table 4 compares the ground-truth citations with the model’s top-five recommended citation lists.The table presents concrete recommendation examples from the proposed model.

5 Conclusion

The study addresses outdated datasets and unclear context detection by releasing FullTextPeerRead with updated papers and organized context-metadata extraction. Its BERT- and VGAE-based framework improves context-aware citation recommendation performance over existing models.

  • The proposed model significantly improves MAP, MRR, and Recall@K over existing models by combining BERT context encoding with VGAE-based citation encoding.VGAE uses a GCN layer to process the paper citation network, while regularization combines paper-network and context representations.
  • FullTextPeerRead addresses outdated datasets and the lack of clear context detection in context-aware citation recommendation research.
  • The dataset includes papers through 2017 and supports readily and accurately extracting context metadata in an organized format.
Loading 1903.06464v1…