CiteBART: Learning to Generate Citations for Local Citation Recommendation

• Citation recommendation suggests papers for a specified placeholder and can provide additional reading; CiteBART fills citation slots rather than generating retrieval-based summaries with citations.Source: §1 · pp. 1–2
• CiteBART-Base trains on masked local contexts, while CiteBART-Global adds the citing paper’s title and abstract to learn their relation to local citation contexts.At inference, CiteBART-Global predicts from these enhanced contexts without target-paper global information.Source: §1 · pp. 2–3
• CiteBART is an end-to-end system that masks citation tokens in local contexts to learn to reconstruct them, unlike prior LCR systems using Transformers indirectly.It does not use cited papers’ titles and abstracts to make recommendations.Source: §1 · pp. 2–3
• CiteBART-Global achieves state-of-the-art LCR performance except on small FullTextPeerRead, with significant effects on larger Refseer and ArXiv benchmarks.CiteBART-Base remains a strong baseline.Source: §1 · p. 3
• The paper analyzes cross-dataset generalization and categorizes hallucinated citations, reporting macro hallucination rates for them.Source: §1 · p. 3
• The ablation study identifies local citation contexts as central to learning and finds the Global training scheme effective over Base.Source: §1 · p. 3

• Encoder-decoder pre-training combines bidirectional encoder learning with autoregressive generation, while BERT uses masked language modeling and SpanBERT masks contiguous spans.PMI-Masking selects word n-grams using pointwise mutual information.Source: §2 · p. 3
• Citation-impact prediction has moved from metadata and abstracts toward whole-paper content, including SChuBERT [van Dongen et al., 2020] and FGCCP.Source: §2 · pp. 3–4
• Earlier citation recommendation used meta-paths with authorship metadata [Yu et al., 2012] or LDA-Bayes with metadata features [Tanner and Charniak, 2015].Source: §2 · p. 4
• Global citation-aware representations include SciBERT [Beltagy et al., 2019] and SPECTER [Cohan et al., 2020], which supports paper classification and global citation recommendation.Legal-document provision prediction also pairs source text with target citations during pre-training [Luo et al., 2023].Source: §2 · p. 4
• The four LCR benchmarks are FullTextPeerRead, ACL-200, Refseer, and ArXiv; GPT-2 and BERT have also been compared for predicting citation placeholders, though their task differs.GPT-2 outperformed BERT in that task, but the results are not directly comparable to CiteBART.Source: §2 · p. 4
• LCR approaches include BERT-GCN [Jeong et al., 2020], DualEnh [Medi´c and Snajder, 2020], HAtten [Gu et al., 2022], and SymTax [Goyal et al., 2024].SymTax adds an Enricher and reranks using taxonomy relationships; it outperforms HAtten on benchmarks.Source: §2 · p. 4

3 Methodology

• CiteBART continually pre-trains BART to reconstruct masked ground-truth author-date citations from citation contexts, without requiring architectural modifications.The model uses BART’s ability to predict multiple tokens from a single mask to generate complex citations.Source: §3 · p. 4, §3.1 · p. 5

3.1 Custom BART Pre-training for LCR

• CiteBART-Base uses local context alone, while CiteBART-Global appends the citing paper’s title and abstract to connect local context with global information.The Global scheme uses the </s> token as a separator and allows backpropagation through the appended global information.Source: §3.1 · p. 5

3.2 Dataset Preprocessing

• Experiments use ACL-200, PeerRead, RefSeer, and Arxiv benchmarks, whose contexts place a target citation in the middle.The context sizes are measured in characters, so words at the starts and ends may be incomplete.Source: §3.2 · p. 5, Table 2 · p. 5
• Preprocessing replaces TARGETCIT markers with <mask> and removes non-target citations so the model focuses on predicting target citations.The preprocessed dataset statistics include training and test partition sizes for each benchmark.Source: §3.2 · p. 5, Table 3 · p. 6
• ACL-200 and RefSeer cases with inconsistent author names, citation ordering, incorrect citations, or empty author names were removed for consistency.The preprocessing also addressed conflicts between citation tokens and target metadata.Source: §3.2 · p. 5

3.3 Metric Definitions

• Evaluation uses Recall@10, Exact Match, and MRR; Recall@10 checks whether the sole target appears in the top 10, while Exact Match checks the first prediction.MRR averages the reciprocal rank of the correctly recommended citation, using k=10 in these experiments.Source: §3.3 · pp. 5–6

4.1 Results

• CiteBART-Global outperforms prior approaches on LCR benchmarks except PeerRead, while CiteBART-Base remains a strong baseline.Its advantage over SymTax and HAtten is largest on Refseer, which has more training contexts than ArXiv.Source: §4.1 · p. 6, Table 4 · p. 7

4.2 Qualitative Analysis

• On the same PeerRead context, adding citing-paper global information moves the ground-truth citation from rank 18 to first; cross-dataset transfer also succeeds when another context cites that reference.An ACL-200-trained model misses a PeerRead ground truth absent from its training data, while an Arxiv-trained model retrieves a reference shared across datasets from a different context.Source: §4.2 · p. 7

4.3 Ablation Study

• Adding the cited paper’s title and abstract during pre-training lowers performance when that information is unavailable at inference.The authors suggest that training on both citing- and cited-paper information may confuse predictions under this inference setting.Source: §4.3 · p. 8
• Recall@10 falls from 0.739 to 0.588 without local context; removing the title or abstract has no statistically significant effect.The ablation uses the ACL-200 Global dataset.Source: §4.3 · p. 8, Table 6 · p. 9

4.4 Taxonomy and Measurement of Hallucinated Citations

• The hallucination taxonomy includes incorrect-year, partially correct-author, year-correct, wrong-format, and other errors; proposed metrics target categorized hallucination types.CiteBART hallucinations typically involve entity errors or fabricated citations.Source: §4.4 · pp. 9–10
• MaHR-partial accounts for almost half of top-3 hallucinations and declines at top-5 and top-10; conditional hallucination metrics differ significantly from MaHR (p < 0.001).Arxiv Global best mitigates hallucinations when a ground-truth citation appears in the top-k list.Source: Table 7 · p. 10, §4.4 · p. 10

4.5 Qualitative Analysis on Hallucinations

• Additional examples include a top-10 list without hallucinations, examples of different hallucination types, and a cross-dataset generalization case.Contexts and abstracts are abbreviated in the examples.Source: §4.5 · p. 10

4.6 LLMs in LCR

• Recall@10 for Jiang et al.’s BM25-prefetched LLM approach was 0.134 on ArXiv and 0.152 on RefSeer; evaluating 100 candidates separately requires 100 prompts per example.Presenting all candidates together often exceeded prompt limits or yielded invalid outputs, and the authors describe prompt design and efficiency as bottlenecks.Source: §4.6 · p. 11

5 Discussion and Conclusion

• CiteBART-Global achieves state-of-the-art performance on LCR benchmarks except FullTextPeerRead, where the dataset is too small to show the advantage of generative pre-training.CiteBART performs LCR end-to-end, unlike pre-fetch-and-rerank pipelines that require cited-paper titles and abstracts at inference.Source: §5 · p. 11
• CiteBART can be fine-tuned for downstream tasks, and the authors propose continual pre-training on new recommendation datasets rather than pre-training from scratch.The authors hypothesize that citation-specific learning signals may benefit downstream tasks involving citations and scientific papers.Source: §5 · p. 11
• BART permits masking parenthetical author-date citation tokens, whereas RoBERTa cannot add citation tokens through its MLM and makes citation-constrained prediction less straightforward.The authors note that BART is prone to hallucination while its capabilities significantly enhance LCR performance.Source: §5 · p. 11
• Almost half of top-3 hallucinations are partially correct, and partially correct author names may identify relevant authors for additional reading.The taxonomy includes examples of hallucinated publication years and author names, fabricated author lists, and citation-format differences.Source: §5 · pp. 11, 13, Table 9 · p. 12
• When the ground truth appears in the top-k list, hallucination tendency in other predictions drops significantly, with Arxiv Global most advantageous.The authors report this as a finding from their hallucination analysis.Source: §5 · p. 13
• Adding both citing- and cited-paper titles and abstracts during continual pre-training did not improve results; missing cited-paper information at test time complicates learning the citation link.The authors plan to investigate this all-including configuration and more sophisticated masking strategies.Source: §5 · p. 13
• The authors propose exploring citation-specific hallucination reduction and combining citing-paper information, local context, and cited-paper information through more sophisticated masking.They also plan to investigate custom mask-filling and retrieval-token recognition in generative information retrieval.Source: §5 · p. 13

Limitations

• CiteBART assumes a local context with a citation placeholder is already citation-worthy.The task is to predict candidates for a placeholder, not to determine where a citation should be placed.Source: Limitations · p. 13
• CiteBART requires pre-training on a specific dataset’s paper pool, so it may omit works or authors outside that corpus, while generated unseen-paper citations can hallucinate.The authors state that hallucination remains a problem even when fabricated citations capture ground-truth author names.Source: Limitations · p. 13
• The current benchmarks provide metadata only for the middle citation in each context, so evaluating multi-citation contexts requires a dataset containing multiple citations and their metadata.The authors say supporting such contexts would require only minor model and code changes, but dataset construction is the more important requirement.Source: Limitations · p. 13
• Because CiteBART learns from local context and citing papers, its recommendations may be biased toward citing papers.The authors leave learning from the citing paper, local context, and cited paper together for future exploration.Source: Limitations · p. 13

Ethics Statement

• CiteBART supports paper writing but does not replace researchers’ thoughtful selection of appropriate local-context references.Source: Ethics Statement · p. 13

A Token Limits

• Base datasets use 200–400 token limits, with 400 tokens selected as optimal to preserve context without excessive padding.ACL-200 had 607 contexts above the limit; these were shortened to 400 tokens because they were a small proportion and the discarded tokens were negligible.Source: A Token Limits · p. 16
• Table 10 presents the maximum token limits used for the preprocessed base and global datasets.Source: Table 10 · p. 16, A Token Limits · p. 16
• Global datasets use a 350-token limit, with local contexts capped at 100 tokens and 50 tokens on each side of the mask.Abstracts are limited to 200 tokens because most fit within that length.Source: A Token Limits · p. 16

B Training and Evaluation Times

• Global pre-training takes 2 hours for Peerread and 6 hours for ACL-200, versus up to 8–9 days for Arxiv and Refseer.Base training takes 8 hours for Peerread and 20 hours for ACL-200, versus up to 14–15 days for Arxiv and Refseer.Source: B Training and Evaluation Times · pp. 16–17
• Evaluation takes up to 2 days on larger datasets, compared with 20 minutes for Peerread and 2 hours for ACL-200.The authors attribute the evaluation burden to generating top-10 predictions and note limited hardware resources.Source: B Training and Evaluation Times · p. 17
• Gu et al. [2022] reported results on only a 10K test-set subsection because of long evaluation times.Source: B Training and Evaluation Times · p. 17

C Exact Match Scores

• Previous studies did not report EM scores, which the authors consider useful for assessing whether the correct citation is generated on the first attempt.Source: C Exact Match Scores · p. 17
• CiteBART directly predicts the correct citation for a substantial portion of the benchmark datasets, according to the reported EM results.Source: C Exact Match Scores · p. 17, Table 11 · p. 17
• Table 11 reports CiteBART’s exact match (EM) scores on LCR benchmarks.Source: Table 11 · p. 17, C Exact Match Scores · p. 17

D Qualitative Analysis on Large Language Models’ Performances in LCR

• Prompts provide 200 citation tokens and masked-context examples, then ask Llama-2-70b-chat to select a citation for a new context.The examples include the prompt, the LLM answer, and the ground-truth masked citation.Source: D Qualitative Analysis on Large… · p. 17
• Figure 2 shows the base approach responding “Shwartz et al., 2016” in correct Part (a), but filling Part (b)’s mask with “Bahdanau et al., 2016” instead of expected “Bluche, 2016”.Source: D Qualitative Analysis on Large… · p. 17
• Figure 3 shows a correct global-dataset prediction using the citing paper’s title and abstract, alongside an incorrect but plausible prediction.The authors conclude from the observed behavior that LLMs need custom pre-training for citation tokens to perform well in local citation recommendation.Source: D Qualitative Analysis on Large… · p. 17, Figure 3 · p. 19
• In further trials, LLMs sometimes ignore the provided citation list, ask for confirmation, and then change their answer even after confirmation.The authors characterize these behaviors as hallucinations.Source: D Qualitative Analysis on Large… · p. 17