PaperlayerView exact revision on arXiv

Source-linked AI summary

Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings

Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, Georg Rehm

arXiv:2202.06671v2cs.CL

TL;DR

Discrete citation relations can miss similar uncited papers and create conflicting positive and negative samples. SciNCL uses controlled nearest-neighbor sampling over citation graph embeddings, and achieves 81.8 on SCIDOCS, 1.8 points above SPECTER.

  • Problem

    Discrete citation signals can treat a paper pair as both positive and negative, reflect non-semantic citation motivations, and miss similar papers without direct citations.

    Source: §3.2 · p. 4

  • Method

    SciNCL uses citation embeddings to sample controlled nearest-neighbor positives and negatives for contrastive training, with a margin to avoid collisions.

    Source: §2 · p. 2, §3.1 · p. 4

  • Results

    81.8 on SCIDOCS is 1.8 points above SPECTER, and SciNCL’s improvement is also observed when excluding SCIDOCS papers during training.

    Source: §8 · p. 9

  • Takeaways & Limitations

    SciNCL outperforms baselines with 1% of training triplets, a general-domain language model, or training only model bias terms.

    Source: §1 · p. 2

    SciNCL requires additional computation for graph embedding training, KNN search, and hyperparameter optimization, partly offset by sample-efficient language-model training.

    Source: §9 · p. 9

Abstract

from arXiv · show

Learning scientific document representations can be substantially improved through contrastive learning objectives, where the challenge lies in creating positive and negative training samples that encode the desired similarity semantics. Prior work relies on discrete citation relations to generate contrast samples. However, discrete citations enforce a hard cut-off to similarity. This is counter-intuitive to similarity-based learning, and ignores that scientific papers can be very similar despite lacking a direct citation - a core problem of finding related research. Instead, we use controlled nearest neighbor sampling over citation graph embeddings for contrastive learning. This control allows us to learn continuous similarity, to sample hard-to-learn negatives and positives, and also to avoid collisions between negative and positive samples by controlling the sampling margin between them. The resulting method SciNCL outperforms the state-of-the-art on the SciDocs benchmark. Furthermore, we demonstrate that it can train (or tune) models sample-efficiently, and that it can be combined with recent training-efficient methods. Perhaps surprisingly, even training a general-domain language model this way outperforms baselines pretrained in-domain.

1 Introduction

Scientific document representations need similarity signals beyond discrete citations, which can miss related uncited papers and create conflicting positive and negative labels. SciNCL samples controlled neighbors in citation-graph embedding space to address these problems.

1 Introduction

  • Large pretrained language models perform well after task fine-tuning, but their sentence and document embeddings can be poorer than GloVe because their spaces are anisotropic.The passage describes anisotropy as a space that is poorly defined in some areas.Source: §1 · p. 1
  • SPECTER uses citations by the query as a discrete signal, treating cited papers as positives and papers not cited by it as negatives.SPECTER applies this sampling to contrastively fine-tune SciBERT.Source: §1 · p. 1
  • A single citation direction can make a paper pair both positive and negative, while noisy citations and uncited related papers undermine a discrete similarity cutoff.The passage notes citations may reflect politeness or policy rather than semantic similarity.Source: §1 · p. 2
  • SciNCL generates contrastive samples from citation embeddings, which encode the full citation graph and provide a continuous, undirected similarity signal.This signal supports selecting difficult-to-learn positive and negative samples.Source: §1 · p. 2
  • 9 of 12 SCIDOCS metrics and the average favor SciNCL over SPECTER; 1% of triplets, a general-domain model, or bias-only training each suffice to outperform baselines.The introduction also reports that SciNCL outperforms the state of the art on average.Source: §1 · p. 2

2 Related Work

Prior work uses contrastive objectives and citation signals to learn document representations, but discrete citation-based sampling can produce conflicting labels. SciNCL instead uses citation embeddings to select samples and transfer citation information through sample selection.

Contrastive learning requirements

  • SciNCL uses citation graph embeddings to add citation-derived information to contrastive views, with hard positives and negatives separated by a sampling-induced margin to avoid collisions.Hard positives are close to the query but not trivially close; hard negatives approach positives without overlapping them.Source: §2 · p. 2

Triplet Mining

  • Prior triplet-mining approaches include text-based nearest-neighbor hard negatives, while SciNCL additionally integrates citation information for scientific papers.The passage characterizes citation information as valuable in this context.Source: §2 · pp. 2–3

Scientific document representations

  • Scientific document representation work includes domain-pretrained transformers and hybrid methods combining text with citation features for recommendation or citation tasks.The related work names SciBERT, BioBERT, and SciGPT2 among domain-pretrained models.Source: §2 · p. 3

Citation-based document representations

  • Unlike SciNCL, SPECTER and Citeomatic use discrete citation relations that do not account for citation-based similarity pitfalls; Citeomatic also samples neighbors from prior-epoch document embeddings.Both approaches use papers cited by the query as positive samples.Source: §2 · p. 3

Cross-Modal Transfer

  • SciNCL transfers knowledge across modalities through contrastive sample selection, which the authors report is superior to direct transfer.The comparison is reported in Appendix F.9.Source: §2 · p. 3

3 Methodology

SciNCL learns citation-informed scientific document representations with a triplet objective and citation-graph neighborhoods. Its sampling choices control positive and negative difficulty while separating the two classes to reduce collisions.

3 Methodology

  • The method forms query, positive, and negative document-vector triplets, using citation embeddings to locate the neighborhood from which contrast samples are drawn.The methodology defines document vectors, citation-based similarity, and neighborhood sampling as its three components.Source: §3 · p. 3, §3.2 · p. 4

3.1 Contrastive Learning Objective

  • SciNCL represents papers as vectors from title-and-abstract inputs and continues training SciBERT with a self-supervised triplet margin loss.The objective uses a slack term of 1 and the L2 norm as its distance function.Source: §3.1 · pp. 3–4

3.2 Citation Neighborhood Sampling

  • Citation embeddings provide a continuous similarity scale for sampling positive and negative papers from a query paper’s nearest-neighbor neighborhood.This replaces discrete citations, which can assign conflicting learning signals across citation directions and may not reflect semantic similarity.Source: §3.2 · p. 4

3.3 Positives and Negatives Sampling

  • Positives come from a close query neighborhood but remain far enough away to avoid gradient collapse; hard negatives approach positives without crossing the tunable sampling margin.Easy negatives are farther from the query than hard negatives.Source: §3.3 · p. 4

3.4 Sampling Strategies

  • KNN sampling selects ranges of citation-space neighbors, while random sampling draws from the corpus and filtered random sampling excludes retrieved neighbors.The KNN parameter k controls sample difficulty and the positions and widths of the positive and negative bands.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate document embeddings on SCIDOCS and compare training datasets that reproduce SPECTER’s leakage and exclude mapped SCIDOCS papers. The setup reuses SPECTER’s training scale and includes established embedding baselines and an oracle comparison.

4.1 Evaluation Dataset

  • SCIDOCS evaluates embeddings on classification, user activity prediction, citation prediction, and recommendation tasks.The benchmark tasks use embeddings without explicit task fine-tuning.Source: §4.1 · p. 5

4.2 Training Datasets

  • 40.5% of SCIDOCS papers occur in SPECTER’s training data, so the authors train both a leakage-replicating set and a subset excluding mapped SCIDOCS papers.The no-leakage subset leaves only 1.4% of SCIDOCS papers potentially able to leak into training.Source: §4.2 · p. 5

4.3 Model Training and Implementation

  • SciNCL uses SciBERT and triplet loss for two epochs on two RTX 6000 GPUs, while citation graph embeddings train on a 60-core CPU.KNN indexing and retrieval take less than 30 minutes.Source: §4.3 · p. 6

4.4 Baseline Methods

  • Baselines include general and scientific language models, graph and citation methods, and SPECTER; Oracle SciDocs estimates an upper bound using gold task labels.Oracle SciDocs uses 106K training triplets and balances triplets across tasks.Source: §4.4 · p. 6

5 Overall Results

SciNCL improves average SCIDOCS performance over SPECTER, with most gains on user-activity and citation tasks. Its results are mixed across classification and recommendation tasks.

5 Overall Results

  • Results vary by task: SciNCL trails SPECTER on MAG and recommender P@1, leads on MeSH and user-activity and citation tasks, and is on par on recommender nDCG.SciNCL scores 81.4 F1 on MAG versus SPECTER’s 82.0 F1, 88.7 F1 on MeSH (+2.3 versus SPECTER), and 19.3 P@1 versus SPECTER’s 20.0.Source: §5 · p. 6
  • Without leakage, SciNCL outperforms SPECTER on 11 of 12 metrics and ties on the remaining metric.The authors suggest this indicates low corpus dependency for SciNCL’s hyperparameters, which were optimized only on the corpus with leakage.Source: §5 · p. 6
  • General-domain BERT scores 63.4, above SciBERT at 59.6 and BioBERT at 58.8; LLMs without citations or contrastive objectives generally score poorly.The authors say this emphasizes embedding anisotropy and the advantage of combining text and citation information.Source: §5 · pp. 6–7
  • 81.8 average score across SCIDOCS metrics; SciNCL beats SPECTER, the next-best baseline, by 1.8 points with replicated SPECTER training data.Without leakage, SciNCL’s average score is 79.4, a 1.7-point improvement over SPECTER.Source: §5 · p. 6, Table 1 · p. 7
  • Most gains occur on user-activity and citation tasks; Oracle SciDocs reaches 83.0 average despite using SCIDOCS test and validation data for triplet selection.The reported SciNCL average is 81.8.Source: §5 · p. 7

6 Impact of Sample Difficulty

The experiments tune positive and negative KNN sampling using 10% of the replicated SPECTER training data; the authors report that subset’s optimizations correlate with the full dataset. Performance depends on sample difficulty and the margin that separates positive from negative samples.

6 Impact of Sample Difficulty

  • The sampling settings were optimized by partial grid search on 10% of replicated SPECTER training data, with validation scores averaged over three seeds.The authors report that optimization on this subset correlates with the entire dataset.Source: §6 · p. 7

6.1 Positive Samples

  • Validation performance peaks at k+=25, stays relatively stable below 100, and declines as k+ increases beyond 100.The authors suggest that very large k+ narrows the sampling-induced margin, increasing collisions with negative samples and contrastive label noise.Source: §6.1 · pp. 7–8, Figure 2 · p. 7
  • The standard deviation σ also varies with k+, because positives and negatives interact through the triplet margin loss.The authors suggest sampling jointly by query distance and positive-negative distance to account for this interaction.Source: §6.1 · p. 8

6.2 Hard Negative Samples

  • At k−_hard=4000 performance plateaus on the test set; k−_hard=3000 scores 81.7, while larger values begin to reduce performance.The validation experiments use k+=25; performance declines when k−_hard exceeds 4000.Source: §6.2 · p. 8, Figure 3 · p. 8

6.3 Easy Negative Samples

  • Filtered random easy-negative sampling slightly improves validation performance over pure random sampling, but their rounded average test scores are identical.Without filtering, random seeds have a larger effect; k-means clustering reduces performance.Source: §6.3 · p. 8

6.4 Collisions

  • Smaller sample-induced margins produce more positive-negative collisions; the number reaches zero when k−_hard > 1000.The authors report that large k+ or small k−_hard also lowers performance toward SPECTER’s level.Source: §6.4 · p. 8, Figure 4 · p. 8

7 Ablation Analysis

The ablations vary language-model initialization, training-triplet quantity, and parameter-efficient training. SciNCL retains scores above the SPECTER baseline with general-domain initialization, fewer triplets, and BitFit.

7.1 Initial Language Models

  • With BERT-Base, BERT-Large, and BioBERT initialization, scores decline by 0.6, 0.4, and 0.4, respectively, but all remain above SPECTER.For recommendation, where SPECTER is superior to SciNCL, BioBERT outperforms SPECTER.Source: §7.1 · p. 9, Table 2 · p. 9

7.2 Data and Computing Efficiency

  • With 1% of the original triplets (6,840), SciNCL scores 80.8: 1.0 point below its 100% setting and 0.8 points above SPECTER.Using 10% of the triplets yields 81.1, a 0.6-point decline.Source: §7.2 · p. 9, Table 2 · p. 9
  • BitFit, which trains only bias terms (0.1% of the original parameters), yields a score of 81.2, down 0.5 points.The authors state that this could allow SciNCL to be trained on the same hardware with larger general-domain language models.Source: §7.2 · p. 9, Table 2 · p. 9

8 Conclusion

SciNCL improves scientific document representations by using citation graph embeddings to select informative contrastive samples. It exceeds SPECTER on SCIDOCS while requiring far fewer training triplets, and general-domain language models can also perform competitively with this approach.

8 Conclusion

  • 81.8 on SCIDOCS, a 1.8-point improvement over SPECTER, is achieved by tuning sample difficulty and avoiding positive–negative collisions.The improvement also holds when SCIDOCS papers are excluded during training; oracle triplets yield 83.0.Source: §8 · p. 9
  • A subset of triplets suffices to outperform SPECTER, while the remaining 99% provide only 1.0 additional points, increasing the score from 80.8 to 81.8.This sample efficiency requires reasonable sample-generation effort, including graph-embedding training.Source: §8 · p. 9
  • General-domain language models can achieve comparable performance and even outperform SPECTER when trained this way.The authors conclude that controlling sample difficulty and avoiding collisions is more effective than in-domain pretraining in scenarios where training an LLM from scratch is infeasible.Source: §8 · p. 9

9 Limitations

SciNCL’s main limitation is additional compute, especially for graph-embedding training; the evaluation also leaves graph-model effects and smaller graph settings unexplored. The paper describes its S2ORC mapping procedure and reports substantial overlap between SPECTER training papers and SCIDOCS test and validation data.

9 Limitations

  • Additional compute for graph-embedding training, KNN search, and tuning k+ and k− is SciNCL’s major limitation.Sample-efficient language-model training offsets some, but not all, of this effort; graph-embedding training accounts for most of the extra compute.Source: §9 · pp. 9–10

A Mapping to S2ORC

  • Mapping papers to S2ORC relies on PDF hashes and exact-title matches, with remaining metadata collected through the Semantic Scholar API.Failed mappings can reflect papers unavailable through the API or absent from the S2ORC citation graph.Source: A Mapping to S2ORC · p. 14

B SPECTER-SciDocs Leakage

  • 40.5% of SPECTER’s training papers overlapped with SCIDOCS test and validation papers, affecting unsupervised metadata but not gold labels (Cohan et al., 2020).The overlap comprises 25.4% of SCIDOCS test papers and 25.5% of validation papers.Source: B SPECTER-SciDocs Leakage · p. 14

C Dataset Creation

The datasets are built from the S2ORC citation graph, with separate graph and paper-selection procedures for reproducing SPECTER’s setting with and without SCIDOCS leakage. The construction specifies which papers and triplets are used for training each setting.

C Dataset Creation

  • The source is S2ORC version 20200705v1, whose full citation graph contains 52.6M nodes and 467M edges.Table 4 summarizes statistics for the two datasets and their overlap with SPECTER and SCIDOCS.Source: C Dataset Creation · p. 14, Table 4 · p. 15
  • For the replicated SPECTER setting, the graph excludes SCIDOCS papers not used by SPECTER, leaving 52.5M nodes and 463M edges.This graph is used to train the citation graph embeddings.Source: C Dataset Creation · p. 14
  • SciNCL triplet selection uses 227,869 of 248,007 SPECTER papers and 131,644 of 136,820 query papers, with random papers added to align triplet counts.Source: C Dataset Creation · p. 14
  • The leakage-free graph excludes successfully mapped SCIDOCS papers and contains 52.3 nodes and 447M edges after filtering.The graph is used to train the citation embedding model.Source: C Dataset Creation · p. 14
  • On the filtered graph, SPECTER-style sampling uses 136,820 queries to generate 684,100 triplets; SciNCL then selects triplets from the same paper set and queries.The SPECTER-style triplets include five cited positives, two citation-of-citation hard negatives, and three random nodes per query; the SciNCL FAISS index contains only those 680,967 unique papers.Source: C Dataset Creation · p. 14

D Graph Embedding Evaluation

On link prediction, BigGraph embeddings performed better with dot product than cosine similarity, and larger dimensions correlated with better performance. The evaluation also notes a corpus-mapping constraint and a compute limit on embedding size.

D Graph Embedding Evaluation

  • Embeddings were trained with dimensions 128, 512, or 768 and cosine similarity or dot product, then evaluated on link prediction using a 99%/1% train-test split.All variations used 20 epochs, margin m=0.15, and learning rate λ=0.1; SCIDOCS evaluation was not possible because its papers could not be mapped to S2ORC.Source: D Graph Embedding Evaluation · p. 15
  • Dot product substantially outperformed cosine similarity, and larger embeddings correlated positively with link-prediction performance across MRR, Hits@1, Hits@10, and AUC.The evaluation reports these qualitative trends without supplying the table’s individual values in the provided passages.Source: D Graph Embedding Evaluation · p. 15, Table 5 · p. 15
  • Dimension 768 was the largest embedding size permitted by available compute resources, with disk space as the limiting factor.Source: D Graph Embedding Evaluation · p. 15

E Baseline Details

  • Baseline weights came from Hugging Face Hub for BERT, BioBERT, CiteBERT, and DeCLUTR; Universal Sentence Encoder came from TensorFlow Hub.Oracle SciDocs triplets were undersampled for balanced task distributions, and the no-leakage SPECTER variant used the SciNCL implementation.Source: E Baseline Details · p. 15

F Negative Results

The explored sampling and model modifications generally failed to improve on the preferred setup, with outcomes varying by strategy and, for KNN, by task. The experiments identify several practical constraints on alternative sampling and training choices.

F.1 Undirected Citations

  • Making citations undirected yielded an 81.7 average score (-0.1 points), so this variant was discarded for the final experiments.Source: §F.1 · p. 15

F.2 KNN with interval large than c

  • Using a larger KNN interval produced comparable results but increased randomness, making optimization more difficult than sampling an interval equal to the requested sample count.Source: §F.2 · p. 15

F.3 K-Means Cluster for Easy Negatives / F.7 Sorted Random / F.8 Mask Language Modeling

  • K-means easy negatives reduced performance; sorted-random sampling ranks n candidates by distance, while combining mask-language modeling with contrastive loss decreased SCIDOCS performance.The sorted-random strategy selects the closest or furthest c′ candidates after ordering them by distance to the query.Source: §F.3 · p. 16, §F.7 · p. 16, §F.8 · p. 16

F.4 Sampling with Similarity Threshold / F.5 Hard Negatives with Similarity Threshold

  • Similarity-threshold sampling was inferior to KNN; hard negatives yielded 81.7 (-0.1 points), with validation scores decreasing as negatives moved closer to the query.Figure 5 presents validation results for different similarity thresholds.Source: §F.5 · p. 16, Figure 5 · p. 16, §F.4 · p. 16

F.6 Positives with Similarity Threshold

  • With positive similarity sampling at t+ < 0.5, more than 40% of query papers had no neighbors, and changing the query set was omitted for comparability to SPECTER.Source: §F.6 · p. 16

F.9 Student-Teacher Learning

  • L_ST overfit and scored 64.7 on SCIDOCS, slightly above SciBERT but below SciNCL with triplet loss; L_Joint scored 80.5 without overfitting, yet trailed triplet loss alone.The joint objective was discarded partly because computing cosine similarities added complexity.Source: §F.9 · p. 17

F.10 SPECTER & Bidirectional Citations

  • Changing SPECTER sampling from unidirectional to bidirectional citations raised its SCIDOCS average from 79.0 to 79.4 (+0.4 points).The unidirectional model replicated the original sampling strategy using the paper’s training data.Source: §F.10 · p. 17

G Task-specific Results / H Examples

Loading 2202.06671v2…