Source-linked AI summary
Asymmetric Within-Document Predictive Learning for Scientific Document Representation
You Zuo, Éric de la Clergerie, Benoît Sagot
TL;DR
Scientific document representation commonly relies on citation-derived supervision, motivating a citation-free alternative based on paper discourse structure. SciJEPA predicts later-section representations from earlier ones, and experiments show that SIGReg improves predictive training while its benefits depend on task geometry and regularization strength.
Problem
Citation-derived supervision is effective but delayed for new papers, variable across fields and article types, and not limited to semantic similarity.
Method
SciJEPA learns citation-free representations through asymmetric within-document prediction from title+abstract to method and from method to conclusion.
Results
Plain SciJEPA is weaker than controlled contrastive learning, while SIGReg improves performance across tasks and reaches 44.06 MAP on high-influence citation with weight 0.0025.
Takeaways & Limitations
Within-document predictive learning is a promising citation-free complement to citation-supervised representation when embedding geometry is carefully controlled.
Takeaways & Limitations
Evaluation includes citation-aligned benchmarks, so citation-supervised encoders are tested under signals close to their training objective.
Abstract
from arXiv · showhide
We study predictive pretraining for scientific document representation using the discourse structure of papers. We propose SciJEPA, a citation-free framework that learns through asymmetric within-document prediction: title and abstract representations are used to predict method representations, and method representations are used to predict conclusion representations. Experiments on RELISH, high-influence citation, SciDocs, and cite prediction show that plain predictive training is viable but weaker than a controlled contrastive baseline using the same section pairs. Adding Sliced Isotropic Gaussian Regularization (SIGReg) substantially improves performance and narrows this gap. The effect of regularization is task-dependent: moderate SIGReg helps fine-grained ranking, while stronger regularization can weaken local alignment. We further show that different encoding branches support different retrieval regimes. These results position within-document predictive learning as a promising citation-free complement for scientific document representation, provided that embedding geometry is carefully controlled.
1 Introduction
SciJEPA addresses the limits of citation-derived supervision by learning scientific document representations from discourse-structured, asymmetric within-document prediction. Plain prediction is viable but weaker than controlled contrastive learning, while SIGReg improves performance and narrows the gap.
- Citation links are effective supervision but are delayed for new papers, vary across fields and article types, and serve rhetorical functions beyond semantic similarity.
- SciJEPA predicts method representations from title-and-abstract representations, then predicts conclusion representations from method representations.The framework uses the paper’s structured progression from problem statement to methodology and outcomes.
- The predictive objective operates in representation space without citation links, negative examples, or token-by-token reconstruction.It is designed to learn broader discourse and semantic dependencies rather than exact later-section wording.
- SIGReg substantially improves SciJEPA and narrows its performance gap with the controlled contrastive baseline.Plain predictive training remains weaker than contrastive learning under the matched evaluation setup.
2 Related Work
Prior scientific document encoders rely largely on domain adaptation or cross-document supervision, while other work exploits document structure and predictive learning. SciJEPA combines section structure with citation-free latent prediction and addresses the geometry challenges of non-contrastive learning.
- SciBERT, SPECTER, and SPECTER2 represent scientific documents using domain-adapted language modeling or citation-based cross-document supervision.SPECTER and SPECTER2 are strong on citation-derived benchmarks because their training signals align closely with those evaluations.
- SciJEPA applies citation-free predictive learning to structured scientific-paper sections rather than using citations as supervision.
- Hierarchical, facet-based, and citation-intent methods show that sections can support long-document modeling and aspect-specific scientific representations.Examples include hierarchical attention, section-aware contrastive learning, and background-, method-, and result-oriented representations.
- JEPA predicts a target view’s latent representation from a context view instead of reconstructing inputs or contrasting positive and negative pairs.The paradigm has been applied in image, video, and more recently language learning.
3 Methodology
SciJEPA encodes directional relationships between paper sections with an online–target predictive architecture and a residual predictor. SIGReg controls global embedding geometry, while a matched InfoNCE baseline tests the contribution of the predictive objective.
- 3.1 Asymmetric Within-document Prediction: SciJEPA predicts methodological content from title-and-abstract context and outcome-oriented content from method context, preserving the direction of discourse dependencies.
- 3.2 Predictive Architecture and Regularization: The online encoder predicts a stop-gradient target representation, while the target encoder is updated by exponential moving average.The slowly changing target branch provides a stable reference and reduces representational-collapse risk.
- 3.2 Predictive Architecture and Regularization: The predictor applies a residual transformation because context and target sections have different discourse roles and should not be identical.
- 3.2 Predictive Architecture and Regularization: SIGReg regularizes normalized online representations by matching random one-dimensional projections toward a Gaussian reference.It discourages collapse and concentration of variance in dominant directions, which the predictive loss alone does not control.
- 3.3 Controlled Contrastive Baseline: The controlled contrastive baseline uses the same section pairs and backbone but replaces latent prediction with InfoNCE and in-batch negatives.Its cosine-similarity loss uses a temperature and negatives drawn from other documents.
4 Experimental Setup
Experiments pretrain controlled models on a decontaminated S2ORC-ArXiv corpus and evaluate them across retrieval tasks with different relevance signals and candidate granularities. The setup also compares off-the-shelf encoders and diagnoses embedding geometry.
- 4.1 Pretraining Data: After filtering and decontamination, pretraining retains 1.14M unique documents and samples approximately 5M section-pair examples with replacement.The 5M figure counts sampled training examples, not unique documents.
- 4.1 Pretraining Data: Title+abstract, method, and conclusion/results sections are extracted by heading-keyword matching with positional fallbacks; papers without methods are discarded.Papers without conclusions contribute only the title+abstract →method task.
- 4.2 Model and Training Details: Models use SciBERT with mean pooling, while the predictor is a two-layer residual MLP and the target encoder follows an EMA schedule.SIGReg, when enabled, uses 512 random projection slices before the predictor.
- 4.2 Model and Training Details: Training uses AdamW for one fixed stream of 5M examples, with effective batch size 4,096 and bf16 mixed precision on one NVIDIA H100 GPU.
- 4.4 Evaluation: Evaluation covers RELISH, high-influence citation, SciDocs, and cite prediction using NDCG, MAP, and triplet accuracy across fine-grained and coarse candidate settings.The controlled comparison matches corpus, backbone, extraction procedure, and section pairs; off-the-shelf comparisons include SciBERT, SPECTER, and SPECTER2.
- 4.4 Evaluation: Citation-aligned benchmarks favor citation-supervised encoders’ training signal, whereas SciJEPA tests transfer from citation-free discourse prediction to cross-document retrieval.
- 4.3 Geometry Diagnostics: Embedding geometry is evaluated with alignment, uniformity, and singular spectrum deviation on fixed cite-prediction samples.Uniformity and SSD use 6,000 embeddings, while alignment uses 2,000 query–positive pairs.
5 Results
SciJEPA improves over SciBERT but generally trails controlled contrastive learning, while SIGReg narrows this gap and its optimal strength varies by task. Different representation branches also favor different retrieval regimes.
- Plain SciJEPA improves substantially over SciBERT across all benchmarks but remains below the controlled CL baseline in most tasks.
- Adding SIGReg improves SciJEPA across all tasks and substantially narrows its gap to CL.
- 44.06 MAP: SciJEPA + 0.0025· SIGReg slightly outperforms CL on high-influence citation.
- SciJEPA remains competitive but does not surpass the strongest citation-supervised baselines, which retain an advantage on citation-derived tasks.
- The optimal SIGReg weight is task-dependent: stronger regularization helps RELISH and high-influence citation, whereas smaller values work better for cite prediction and SciDocs.
- The online encoder performs best on fine-grained ranking, whereas predictor outputs are more robust for cite prediction and average SciDocs retrieval; target encoders remain weakest.
6 Conclusion
SciJEPA is introduced as a citation-free framework that predicts between document sections. The results show that predictive learning is useful but depends on controlling embedding geometry and regularization strength.
- SciJEPA uses asymmetric within-document section prediction for citation-free scientific document representation.
- Plain predictive training remains weaker than a controlled contrastive baseline using the same section-pair supervision.
- SIGReg improves SciJEPA, but overly strong regularization can weaken local alignment and its effects vary by task.
- Different predictive-architecture branches support different retrieval regimes.
Références
The references include foundational and recent work on predictive architectures, scientific document representation, corpora, benchmarks, and citation-informed learning.
- The bibliography cites foundational JEPA and predictive-learning work, including LeCun et al. and related image and video architectures.
- It includes scientific language and document-representation research such as SciBERT, SPECTER, and citation-embedding methods.
- The references cover resources and evaluation benchmarks including S2ORC and SciRepEval.
A Section Extraction Details
The data pipeline extracts title-and-abstract, method, and conclusion/results segments from structured papers, applies filtering and fallbacks, and constructs one or two prediction tasks per paper.
- Segment construction: Each paper is represented using title+abstract, method, and conclusion/results segments extracted from structured section text.
- Filtering: Filtering removes papers with short abstracts or bodies, extreme section counts, or excessive malformed section titles.
- Method extraction: Method sections are identified by heading matches, with positional fallbacks selecting substantial sections after an introduction or the second substantial section.
- Conclusion extraction: Conclusion/results sections are identified after the method section, with the last substantial section used as a fallback when necessary.
- Task construction: Papers with both target sections yield title+abstract →method and method→conclusion/results tasks, while missing conclusions permit only the first task.
- Dataset statistics: The processed pool contains 1.14M unique retained documents and is sampled with replacement to produce 5,000,000 training examples.
B SIGReg Implementation Details
SIGReg regularizes normalized online representations by comparing random one-dimensional projection distributions with a standard Gaussian reference. The procedure discourages anisotropy rather than exactly enforcing uniformity on the sphere.
- Placement: SIGReg is applied to the online encoder representation before the predictor.
- Preprocessing: The method normalizes representations, then standardizes each dimension within the batch before regularization.
- Projection statistic: SIGReg samples random unit directions and compares empirical projection characteristic functions with a standard normal reference.
- Hyperparameters: The implementation uses K = 512 projection slices and L = 17 characteristic-function evaluation points over [0, 3].
- Interpretation: Because embeddings are ℓ2-normalized, the Gaussian reference is not an unconstrained Gaussian distribution in R^d.
- Interpretation: The regularizer discourages collapsed, highly anisotropic, or dimensionally concentrated embeddings rather than exactly enforcing spherical uniformity.
- Limitation: A future direction is to balance spherical spread with local alignment more directly.