Source-linked AI summary
Scalable Patch-Level Self-Supervised Learning
Maximilian Seitzer, Gabriele Trivigno, Antonín Vobecký, Seungeun Yi, Maxime Oquab, Huy V. Vo, Oriane Siméoni, Piotr Bojanowski
TL;DR
Scalable SSL methods often combine objectives and stabilization mechanisms, prompting the search for a principled alternative. JEM derives a student-teacher patch-level method from multi-view InfoMax, with alignment and information- and structure-preserving regularizers. It scales stably to 7B parameters and performs strongly on dense tasks, including a reported 7B panoptic result above DINOv3.
Problem
Scalable SSL methods often rely on complex combinations of objectives and stabilization mechanisms, motivating the search for a principled high-performing alternative.
Source: §1 · p. 1Method
Results
JEM trains stably from 300M to 7B parameters, performs strongly on segmentation while remaining competitive on classification, and exceeds DINOv3 on 7B panoptic segmentation.
Source: §1 · pp. 1–2, §5 · p. 9Takeaways & Limitations
The results suggest that multi-objective SSL complexity is not necessary for strong performance and establish principled patch-level objectives as a viable direction.
Source: §5 · p. 9The multi-view assumption's validity depends on the chosen views and downstream tasks, and its limitations are known.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Self-supervised learning (SSL) at scale produces powerful visual representations. However, most scalable SSL methods rely on ad hoc combinations of multiple objectives and stabilization mechanisms. Taking a step back, we ask if we can design a high-performing, yet principled SSL algorithm. Starting from the multi-view assumption, stipulating that task-relevant content is captured by the information common to different views, we construct an information-theoretic objective decomposing into interpretable terms. This derivation yields JEM, a student-teacher method that learns by aligning corresponding patch representations across views, explicitly regularized by information and structure preservation losses. JEM trains stably from 300M to 7B parameters, and, to our knowledge, is the first latent-space patch-level method demonstrated at 7B scale. Across all scales, JEM reaches strong performance on both global and dense probing tasks, on segmentation benchmarks consistently surpassing the DINOv2 algorithm, an influential foundation for today's strongest visual SSL methods. Notably, at 7B parameters, it exceeds the performance of DINOv3 on panoptic segmentation, despite being trained on $12\times$ less data without refinement stages. These results demonstrate that we can indeed design an SSL algorithm that learns strong representations, is principled and stable.
1 INTRODUCTION
The paper seeks a principled alternative to complex scalable SSL objectives, deriving JEM from the idea that task-relevant information is shared across views. JEM uses patch-level alignment and regularization, scales to 7B parameters, and performs strongly on dense prediction.
1 INTRODUCTION
- Scalable SSL methods combine image- and patch-level objectives, but their objectives may compose poorly and require additional stabilization mechanisms.The paper frames this complexity as motivation to reconsider principled SSL objectives.Source: §1 · p. 1
- JEM derives a student-teacher objective from the multi-view assumption, aligning patch representations and regularizing against global collapse and loss of spatial structure.The objective is motivated by retaining information shared across views.Source: §1 · pp. 1–2
- JEM scales stably to 7B parameters, surpasses DINOv2 on dense tasks, remains competitive on classification, and exceeds DINOv3 on 7B panoptic segmentation.The paper reports the DINOv3 comparison despite JEM using less data and no additional refinement steps.Source: §1 · pp. 1–2, Figure 1 · p. 2
2 RELATED WORK
Related work spans several SSL objective families and increasingly combines complementary objectives. The paper positions JEM as an information-theoretic, patch-only approach to the question of whether such combinations are necessary.
2 RELATED WORK
- Contrastive, redundancy-reduction, self-clustering, and masked-prediction methods differ in how they align views, prevent collapse, or predict missing content.Examples include pixel prediction in MAE, latent target prediction in I-JEPA, and cluster-assignment prediction in iBOT and CAPI.Source: §2 · p. 2
- DINOv2 combines global self-distillation, patch-level iBOT prediction, and KoLeo regularization, while DINOv3 adds Gram anchoring to address declining spatial locality during long training.I-JEPA and CAPI are described as single-family latent-prediction approaches.Source: §2 · p. 2
- JEM uses multi-view InfoMax with a variational approximation in a student-teacher setup, rather than contrastive objectives or separate critics.The paper also notes that discrete representation targets can act as an architectural bottleneck and that representation quality depends on inductive biases from architecture and view construction.Source: §2 · pp. 2–3
3 METHOD
JEM derives a student-teacher SSL objective from multi-view InfoMax and implements it with patch alignment and regularizers. Its view construction combines overlapping crops, masking, and photometric transformations.
3 METHOD
- JEM is presented as a principled, competitive joint-embedding SSL method with an alignment loss and regularizers derived from the multi-view InfoMax objective.The method section identifies view design as crucial for best performance.Source: §3 · p. 3, §3.2 · p. 4
3.1 THE MULTI-VIEW INFOMAX OBJECTIVE
- The multi-view assumption treats task-relevant information as shared across views, but whether it holds depends on view choices and downstream tasks.The paper uses it as an idealized principle motivating a representation-learning criterion.Source: §3.1 · p. 3
- The teacher produces targets from one view, while the student predicts them from a paired view through a predictive distribution.Views are generated using masking, cropping, and photometric distortions; the target is a collection of discrete variables in practice.Source: §3.1 · p. 3
- The InfoMax lower bound separates cross-view alignment from an anti-collapse term that preserves information in teacher targets.With an EMA teacher, the anti-collapse term is not directly maximized on the teacher side, motivating the subsequent regularizers.Source: §3.1 · p. 3
3.2.1 PATCH-WISE ALIGNMENT LOSS
- JEM aligns corresponding student and teacher patch-layer elements on overlapping views, with the mapping making alignment invariant to element indices.The method applies the alignment loss over overlapping student patch-layer pairs.Source: §3.2.1 · p. 4
3.2.2 PREVENTING GLOBAL COLLAPSE
- Global-collapse regularization maintains per-element information by maximizing marginal entropy while minimizing conditional entropy.The student acts as a proxy for the teacher through EMA updates; the figure summarizes the regularizer as maintaining per-patch mutual information.Source: §3.2.2 · p. 4, §3.1 · p. 4, Figure 2 · p. 5
3.2.3 PREVENTING STRUCTURAL COLLAPSE
- Structural-collapse regularization controls redundancy by matching student similarities to teacher-feature geometry across layers and patch pairs.This proxy is used because estimating total correlation across many discrete targets is intractable; matching early-layer similarities is intended to avoid redundant-vector collapse while allowing higher-level structure.Source: §3.2.3 · p. 5
3.3 VIEW CONSTRUCTION
- JEM treats cropping and masking as complementary ways to control shared information, aligns overlapping patches whether visible or masked, and can perform well without masks.Its implementation uses shifted, overlapping student and teacher crops on a common patch grid, with local crops, some masked student views, and independent photometric transformations.Source: §3.3 · p. 6
4 EXPERIMENTS
Experiments evaluate frozen-feature probes, ablate JEM’s components, compare it with other SSL methods, and test scaling. JEM performs strongly across dense tasks and trains stably up to 7B parameters.
4.1 IMPLEMENTATION DETAILS
- Experiments use ViT-L, ViT-g, and ViT-7B encoders and evaluate frozen backbones with classification, dense prediction, and depth probes.Training uses ImageNet-1k, ImageNet-22k, or A140M; probes include IN-1k accuracy, segmentation mIoU, instance mask AP, panoptic PQ, and depth RMSE.Source: §4.1 · pp. 6–7
4.2 ABLATIONS
- JEM tolerates removing masks, local crops, or shifted views individually, but removing independent photometric augmentations greatly degrades global and dense performance.The authors attribute this to losing invariance to pixel-level details, which supports learning semantic local features rather than shortcuts.Source: §4.2 · p. 7
- Removing multi-layer prediction, masked-patch prediction, or the EMA teacher changes IN-1k accuracy by at most 1.5 points and ADE20k mIoU by at most 2.1 points.Removing visible patches from the alignment loss costs 20 points on ADE20k.Source: §4.2 · p. 7
4.3 COMPARISON TO OTHER SSL ALGORITHMS
- JEM outperforms other methods on six of nine tasks, gains 4 PQ points on both panoptic tasks over DINOv2, and remains slightly behind DINOv2 on classification.The results span coarse semantic through fine-grained instance and panoptic segmentation.Source: Table 2 · p. 8, §4.3 · p. 9
4.2 ABLATIONS
- Removing alignment drops IN-1k accuracy from 82.8 to 63.1 and ADE20k mIoU from 48.1 to 22.4, while removing global regularization causes collapse.At ViT-L scale, structural regularization has a marginal effect on IN-1k, but becomes important at larger scales.Source: §4.2 · p. 8
- For ViT-g, disabling global regularization collapses teacher patch mutual information, while structural regularization preserves patch diversity and adds 2.7 ADE20k mIoU.Without structural regularization, spatial degradation begins after 250k iterations.Source: Figure 5 · p. 7, §4.2 · p. 8
4.4 SCALING JEM
- At 300M, 1B, and 7B parameters, JEM trains stably and outperforms DINOv2 at every model scale.The comparison uses A140M for 500k steps; DINOv3’s 7B reference uses twice the training duration, 12× more data, and refinement steps.Source: Figure 6 · p. 9, §4.4 · p. 9
5 CONCLUSION
JEM derives a principled patch-level method from a multi-view InfoMax objective and combines cross-view alignment with global-collapse and structural regularization. It trains stably up to 7B parameters and performs strongly on segmentation while remaining competitive on classification and depth estimation.
5 CONCLUSION
- JEM’s three loss terms align representations across views, prevent global collapse, and preserve structural integrity.The method is derived from a multi-view InfoMax objective motivated by the multi-view assumption.Source: §5 · p. 9
- JEM trains stably from 300M to 7B parameters, the first exclusively patch-level latent-space method demonstrated at this scale to the authors’ knowledge.Source: §5 · p. 9
- JEM achieves strong segmentation results while remaining competitive on classification and depth estimation.The authors suggest these results show that multi-objective SSL complexity is not necessary for strong performance and make principled patch-level objectives a promising foundation for scalable visual representation learning.Source: §5 · p. 9
A APPENDIX
The appendix collects additional analyses, experiments, implementation and evaluation details, derivations, and qualitative feature visualizations for JEM.
A APPENDIX
- The appendix adds analyses, implementation details, and experiments for JEM.Source: A APPENDIX · p. 16
- It discusses existing SSL methods through the multi-view InfoMax perspective and relates JEM to prior work.Source: A APPENDIX · p. 16
- It supplies benchmark scaling results, objective derivations, training and evaluation configurations, and feature visualizations.The appendix also details the structural collapse regularizer, view construction, training procedure, and training data.Source: A APPENDIX · p. 16
A.1 RELATIONSHIP TO EXISTING SSL PARADIGMS
This section interprets prior SSL methods through the multi-view InfoMax objective and compares their components with JEM’s alignment, entropy, structural, view-design, and layer-target choices.
A.1 RELATIONSHIP TO EXISTING SSL PARADIGMS
- The section relates JEM’s components to prior work on discrete-target entropy, dense losses across overlapping views, multi-layer targets, and structural regularization.Source: §A.1 · p. 16
- Self-clustering and redundancy-reduction methods combine cross-view alignment with entropy control, including target balancing, sharpening, or output-distribution regularization.Earlier clustering approaches also vary in whether they explicitly control marginal and conditional entropy.Source: §A.1 · pp. 16–17
- DINO-family methods can be interpreted as InfoMax instantiations: cross-entropy aligns views, while empirical mechanisms implicitly control collapse.JEM instead derives each objective component from the lower bound and simplifies DINOv2’s recipe.Source: §A.1 · p. 16
- Compared with DINOv2, JEM uses one head per supervised depth, explicit entropy regularization, and no prototype-freezing mechanism.DINOv2 applies separate DINO and iBOT losses and adds KoLeo regularization; JEM applies the same alignment and anti-collapse terms to all target elements.Source: §A.1 · pp. 16–17
- Patch-level assignments add positional collapse, where encoding grid position can raise pooled entropy; JEM conditions marginal entropy on position to address it.Structural collapse—patches encoding the same image-level content—motivates JEM’s structural regularizer.Source: §A.1 · p. 17
- JEM aligns partially overlapping views on a shared patch lattice, giving exact one-to-one patch correspondences and applying alignment to masked and visible overlaps.The authors report that interpolating targets between misaligned patch grids degrades dense features.Source: §A.1 · pp. 17–18
- At each supervised depth, a student layer predicts the EMA teacher’s categorical distribution at the same depth; alignment and anti-collapse terms apply at every depth.The final student layer also uses intermediate teacher layers as structural targets through their similarity structure.Source: §A.1 · p. 18
- JEM’s structural regularizer matches projected final-student patch similarities to early-to-middle teacher-layer similarities, from the start of training.Unlike DINOv3 gram anchoring, this does not force the final representation to share earlier-layer geometry; it must retain information needed to recover it.Source: §A.1 · p. 18
A.2 ADDITIONAL RESULTS
Figure 7 provides individual benchmark results underlying Figure 6’s task averages and also reports ImageNet-1k classification accuracy.
A.2 ADDITIONAL RESULTS
- Figure 7 reports individual benchmark results that underlie the task averages in Figure 6.Source: §A.2 · p. 18
- The individual-results figure also includes ImageNet-1k classification accuracy.Source: §A.2 · p. 18
- Figure 6’s task averages are complemented by benchmark-level results in Figure 7.Source: §A.2 · p. 18
A.3 IMPLEMENTATION DETAILS
This appendix section points to further details on the structural collapse regularizer, view construction, hyperparameters and training, and training data.
A.3 IMPLEMENTATION DETAILS
- The section gives additional details on the structural collapse regularizer and construction of the views.Source: §A.3 · p. 18
- It also covers hyperparameters and training details.Source: §A.3 · p. 18
- The section additionally describes the training data used.Source: §A.3 · p. 18
A.3.1 STRUCTURAL COLLAPSE REGULARIZER
The structural collapse regularizer compares pairwise patch structure across student and teacher views, using only geometrically corresponding patches and averaging across target layers and patch pairs.
A.3.1 STRUCTURAL COLLAPSE REGULARIZER
- The regularizer uses student and teacher patch indices that overlap geometrically, with a mapping from each student patch to its corresponding teacher patch.Source: §A.3.1 · p. 18
- Only geometrically corresponding patch pairs contribute, and the loss is averaged over target layers and patch pairs.Source: §A.3.1 · pp. 18–19
- ViT-L uses target layers [0, 8, 16], while ViT-g and ViT-7B use [0, 10, 20, 30]; the teacher-based version also permits applying the loss to masked patches.Source: §A.3.1 · p. 19
- A student-only version was also tested; the teacher version was chosen for masked-patch coverage, more stable EMA targets, and possible view-invariance benefits for similarity structure.Source: §A.3.1 · pp. 19–20
A.3.2 VIEW CONSTRUCTION
Student and teacher views are independently cropped from a shared image region with guaranteed overlap, then receive photometric augmentations; most student crops are additionally masked.
A.3.2 VIEW CONSTRUCTION
- The crop procedure ensures at least 10% overlap between student and teacher crops, which are sampled from a patch-grid-aligned super crop.Source: §A.3.2 · p. 20
- Each crop receives independent color jittering, grayscale conversion, Gaussian blur, and solarization, while all views share orientation after a common horizontal-flip choice.Source: §A.3.2 · p. 20
- Masks are sampled for 80% of student crops, covering 65% of global-crop patches and 50% of local-crop patches.Source: §A.3.2 · p. 20
A.3.3 TRAINING DETAILS
The training details describe JEM's weighted losses, its implementation basis, and configuration changes used for the no-EMA ablation.
A.3.3 TRAINING DETAILS
- JEM and the self-trained DINOv2 baseline use the DINOv3 open-source implementation and hyperparameters; the baseline omits Gram anchoring and high-resolution refinement stages.Source: §A.3.3 · p. 20
- For the no-EMA ablation, the teacher follows the student after each update, and regularizer weights are changed to move training out of initial collapse.Source: §A.3.3 · p. 20
- JEM combines visible- and masked-patch alignment with global-collapse and structural regularization losses.Source: §A.3.3 · p. 20
- The structural regularizer is applied only to final-layer representations, while all losses cover global and local crops.Source: §A.3.3 · p. 20
A.3.4 DATA
The appendix details JEM’s data construction and objective derivation, evaluation protocols and baselines, training configurations, and qualitative representation visualizations.
A.3.4 DATA
- A140M is retrieved from a roughly 750M-image pool deduplicated from an internal collection of 1.3 billion images, with benchmark-test contamination filtering, NSFW filtering, and facial blurring.Retrieval aligns images with public dataset train splits, including IN-1k, IN-22k, and ADE20K.Source: §A.3.4 · pp. 20–21
A.4 DERIVATIONS
- Under the joint-view model, the mutual-information lower bound becomes I(XT; T) minus the KL divergence between teacher and student predictions, and element-wise targets make alignment a sum over patch/layer elements.The element-wise decomposition assumes teacher and student distributions factorize over target elements.Source: §A.4 · pp. 21–22
A.5.1 BENCHMARK PROTOCOLS
- Evaluation uses frozen final-layer patch features and lightweight probes across classification, semantic, instance, panoptic, and depth tasks.Semantic segmentation uses kNN and linear probes; instance and panoptic segmentation use a patch-wise classifier and an MLP for patch-pair segment relations.Source: §A.5.1 · pp. 22–24
A.5.2 BASELINE DETAILS
- The appendix lists external checkpoints and estimated training-token counts, while noting that released-checkpoint training configurations cannot always be verified.Token estimates count full student patch grids before masking and exclude teacher and special tokens.Source: §A.5.2 · p. 25, Table 3 · p. 25
A.6 TRAINING CONFIGURATIONS
- Shared architecture and training hyperparameters are reported separately from JEM-specific settings, ablation settings, and DINOv2 baseline settings.Tables 5–8 cover these respective configuration categories.Source: Table 5 · p. 27, Table 6 · p. 27, Table 7 · p. 28
A.7 QUALITATIVE EXAMPLES
- Qualitative visualizations show per-image PCA projections and cosine similarity between final-layer patch representations and the center-patch representation of JEM ViT-g.The visualizations use 896 × 896 images mapped to 56 × 56 patch grids; the cosine-similarity figure marks the reference patch and uses a 0-to-1 color scale.Source: §A.7 · p. 28, Figure 9 · p. 29, Figure 10 · p. 29