PaperlayerView exact revision on arXiv

Source-linked AI summary

SiamJEPA: On the Role of Siamese Student Encoders in JEPA

Makoto Yamada

arXiv:2607.04044v2cs.CVstat.ML

TL;DR

The role of Siamese student encoders in JEPA-based latent prediction remains underexplored. SiamJEPA studies this design and finds that Siamese encoders regularize representations, accelerate convergence, and improve training efficiency relative to comparable approaches.

  • Problem

    The role of Siamese student encoders in JEPA-based latent predictive architectures remains insufficiently understood.

  • Method

    SiamJEPA uses two independently masked Siamese student views to predict latent representations with an EMA teacher, while continuously controlling the Siamese contribution.

  • Results

    Siamese encoders act as an effective regularizer, accelerate convergence, and achieve competitive ImageNet linear probing performance with significantly fewer training epochs than MAE.

  • Takeaways & Limitations

    Siamese student encoders provide an important inductive bias for more training-efficient JEPA representation learning.

  • Takeaways & Limitations

    The study prioritizes investigating Siamese encoders rather than maximizing benchmark performance, leaving extensive hyperparameter optimization and longer training for future work.

Abstract

from arXiv · show

Recently, Joint Embedding Predictive Architectures (JEPAs) have attracted significant attention in the computer vision and machine learning communities as a promising framework for self-supervised representation learning. Unlike masked autoencoders that reconstruct pixels, JEPA models learn representations by predicting latent embeddings of masked regions. Existing JEPA-based methods, such as I-JEPA and V-JEPA, typically employ a single encoder in the student network. In contrast, using Siamese encoders for student network is more naturally aligned with brain-inspired representation learning frameworks, yet their role in JEPA models remains largely unexplored. In this paper, we investigate the effect of Siamese student encoders in JEPA-based representation learning. To this end, we propose SiamJEPA, masked Siamese student encoders equipped with an exponential moving average (EMA) teacher network. SiamJEPA can also be viewed as a JEPA formulation of the brain-inspired representation learning model PhiNet. Through extensive experiments on ImageNet linear probing, we demonstrate that Siamese encoders act as an effective regularizer for the JEPA objective, improving representation separability and accelerating learning during the early stages of training. Furthermore, SiamJEPA consistently outperforms comparable single-encoder JEPA variants under limited training budgets and achieves higher linear probing accuracy than Masked Autoencoders (MAE) which requires longer training. Our findings reveal that Siamese student encoders are not merely an architectural choice but constitute an important inductive bias for predictive representation learning. These results provide new insights into the design of JEPA-based models and suggest that incorporating Siamese student architectures offers a simple yet effective approach for improving self-supervised representation learning.

1 Introduction

SiamJEPA investigates the unexplored role of Siamese student encoders in JEPAs by introducing masked latent prediction from independently masked views. It shows that Siamese encoders regularize representations, accelerate convergence, and improve training efficiency relative to reconstruction-based learning and single-encoder JEPA.

  • Background: JEPAs extend joint embedding methods by predicting masked latent representations from visible context, with I-JEPA [Assran et al., 2023] and V-JEPA [Bardes et al., 2024, Assran et al., 2025] as representative examples.Preventing representation collapse while preserving informative latent representations remains a central JEPA challenge.
  • Related Work: Prior Siamese masked-prediction methods, including SiamMAE [Gupta et al., 2023], CropMAE [Eymaël et al., 2024], and RSP [Jang et al., 2024], primarily reconstruct pixels or features rather than predict latent representations.This distinction motivates studying Siamese encoders specifically within JEPA frameworks.
  • Contributions: SiamJEPA is a masked latent-prediction architecture using two independently masked views to predict masked-token representations from unmasked counterparts without pixel reconstruction.It introduces a regularization parameter that continuously controls the contribution of Siamese student encoders.
  • Contributions: The paper provides the first systematic study of Siamese student encoders in JEPA, finding that they constrain the representation space and accelerate convergence versus conventional single-encoder JEPA.The study frames Siamese encoders as an effective regularizer for the JEPA objective.
  • Contributions: Siamese student encoders make latent prediction substantially more training-efficient than reconstruction-based self-supervised learning while achieving competitive ImageNet linear probing with fewer epochs than MAE.This establishes the paper’s main efficiency and performance finding.

2 Related Work

The paper situates JEPA among joint-embedding and masked-modeling approaches, defining it specifically as latent-target prediction through a predictor network. Prior work motivates SiamJEPA through non-contrastive learning, brain-inspired Siamese encoders, and latent prediction, while leaving image classification with Siamese masked architectures unresolved.

  • JEPA is defined here as an architecture that explicitly predicts target latent representations using a predictor network, rather than merely any joint-embedding method.
  • Contrastive methods such as SimCLR [Chen et al., 2020a] require many negative samples, whereas BYOL and SimSiam [Chen and He, 2021] learn without negatives.
  • Brain-inspired PhiNet [Ishikawa et al., 2025] uses Siamese encoders, temporal prediction, and exponential moving averages, improving collapse robustness over SimSiam and supporting continual learning.
  • MAE [He et al., 2022] reconstructs masked image patches, while JEPA predicts masked representations directly in latent space, extending from images to videos.
  • PhiNetv2 [Yamada et al., 2025] predicts future latent representations for video, but latent prediction for image representation learning and classification remains largely unexplored.

3 JEPA with Siamese encoders (SiamJEPA)

SiamJEPA extends I-JEPA with an additional shared-weight student encoder, independently masked views, and an EMA teacher, following the brain-inspired PhiNet architecture [Yamada et al., 2025]. It aligns global representations with KL divergence and predicts masked-token representations with NMSE, using a probabilistic latent-consistency mechanism inspired by future-frame prediction [Denton and Fergus, 2018].

  • Architecture: SiamJEPA adds a shared-weight student encoder to I-JEPA, applies masking independently to both branches, and updates the teacher with an EMA.Non-overlapping masking is used to prevent shortcut learning, with positional embeddings added before encoding.
  • Predictors: Two predictors separately align global Siamese representations and predict masked-token latents, following PhiNet [Ishikawa et al., 2025, Yamada et al., 2025].The masked-token predictor uses a probabilistic architecture whose posterior receives both branches while its prior uses only the first branch.
  • Probabilistic formulation: The probabilistic formulation adapts future-frame prediction [Denton and Fergus, 2018] to consistency between two augmented views, with latent uncertainty represented by Z.The model uses a two-layer-network parameterized posterior and prior, preceded by a projector head with batch normalization.
  • Training objectives: SiamJEPA optimizes global-view alignment with KL divergence and masked-token prediction with normalized mean squared error.The KL objective makes a single-view prior approximate a posterior inferred from both views, encouraging shared information while discarding view-specific variation.
  • Objective variants: Setting λ_KL = 0 makes SiamJEPA similar to single-encoder JEPA, whereas λ_KL = 10^-4 defines the reported JEPA-like variant.Stopping gradients through the prior branch was found in preliminary experiments to improve optimization stability and downstream performance.

4 Experiments · 4.1 Setup and implementation details

The experiments ablate SiamJEPA and compare it with MAE [He et al., 2022], context autoencoder, and I-JEPA [Assran et al., 2023] in a common framework. Ablations use fixed training settings, while final models are pretrained for 400 epochs to assess Siamese student encoders rather than claim state-of-the-art JEPA performance.

  • 4 Experiments: The experiments compare SiamJEPA with MAE [He et al., 2022], context autoencoder, and I-JEPA [Assran et al., 2023].The study focuses on ablation and validation of Siamese student encoder properties, not state-of-the-art JEPA performance.
  • 4 Experiments: Enforcing consistency between Siamese student encoders improves learned representations and ImageNet linear probing performance across SiamJEPA variants.Table 2 compares SiamJEPA variants with MAE and reports that consistency improves representation quality; SiamJEPA uses mean pooling, whereas MAE uses the CLS token.
  • 4 Experiments: The Siamese student encoder properties are intended to apply to other JEPA models, including V-JEPA [Bardes et al., 2024] and I-JEPA [Assran et al., 2023].The experiment validates whether these properties are useful within the JEPA architecture beyond SiamJEPA itself.
  • 4.1 Setup and implementation details: All ablations use an effective batch size of 8192, decoder depth 1, and base learning rate 1.5 × 10−4.The EMA momentum increases from 0.99 during epochs 1–200, to 0.999 during epochs 201–300, and to 0.9999 during epochs 301–400.
  • 4.1 Setup and implementation details: SiamJEPA models are evaluated using mean pooling from the intermediate tenth layer.This evaluation choice is used consistently across the SiamJEPA models in the ablation studies.
  • 4.1 Setup and implementation details: Final models are pretrained for 400 epochs after hyperparameters are selected from the ablation studies.Experiments run on NVIDIA V100, A100, or H100 GPUs.
  • 4.1 Setup and implementation details: The implementation is based on the official MAE codebase rather than the official I-JEPA implementation.This common framework makes reconstruction-based and JEPA-based methods comparable, although its training dynamics differ from reported I-JEPA dynamics.

4.2 Comparison to other SSL methods · 4.3 Effect of Siamese student encoder

SiamJEPA matches or exceeds selected SSL baselines with substantially fewer training epochs, although comparison with I-JEPA is confounded by different training durations. Ablations show that stronger KL regularization improves representation quality and convergence, while free-bit adjustments have only marginal effects.

  • 4.2 Comparison to other SSL methods: SiamJEPA outperforms MAE with less than one-quarter of its training epochs and achieves performance comparable to CAE, demonstrating improved training efficiency.The comparison includes MAE, CAE, and I-JEPA; Table 2 reports the baseline comparison.
  • 4.2 Comparison to other SSL methods: SiamJEPA has lower final linear probing performance than I-JEPA, but the comparison is not direct because I-JEPA trains for 600 epochs versus 400 for SiamJEPA.The authors identify the differing training setups as a limitation and leave longer training and broader hyperparameter optimization for future work.
  • 4.2 Comparison to other SSL methods: The study prioritizes investigating Siamese student encoders within JEPA rather than maximizing benchmark performance or outperforming existing JEPA methods.The authors suggest that longer training and more extensive hyperparameter optimization could further improve SiamJEPA, but defer that study to future work.
  • 4.3 Effect of Siamese student encoder: Increasing λ_KL consistently improves linear probing, with λ_KL = 0.01 or 0.03 outperforming λ_KL = 0.00001.Table 3 evaluates KL regularization weight and weight decay under block masking with a 0.75 mask ratio and 10th-layer mean pooling.
  • 4.3 Effect of Siamese student encoder: λ_KL = 0.01 reaches performance comparable to weaker regularization after 200 epochs, whereas the weaker setting requires approximately 400 epochs, indicating faster convergence.The KL term encourages consistency between the two masked Siamese views.
  • 4.3 Effect of Siamese student encoder: Small λ_KL produces KL divergence above 40, while large λ_KL keeps it near the 0.1 free-bit threshold; total-loss behavior remains similar, but larger λ_KL converges more slowly.These observations come from learning curves comparing small and large KL regularization weights.
  • 4.3 Effect of Siamese student encoder: A free-bit value of 0.05 consistently yields slightly better performance than other settings, but changing the threshold produces only marginal overall differences.The free-bit mechanism allows discrepancy between Siamese encoder outputs to help prevent representational collapse.

4.4 Effect of Weight decay

SiamJEPA is relatively robust to weight-decay choice, consistent with PhiNet observations [Ishikawa et al., 2025], but larger values are especially beneficial during longer training. With the default 0.05, linear-probing performance saturates around epoch 300, whereas stronger decay supports continued improvement.

  • 4.4 Effect of Weight decay: Larger weight decay improves SiamJEPA particularly during longer training, as shown by the weight-decay sweep in Table 3.The study varies the weight-decay coefficient and finds stronger decay increasingly beneficial over extended pre-training.
  • 4.4 Effect of Weight decay: SiamJEPA is relatively robust to weight-decay choice, consistent with observations for PhiNet [Ishikawa et al., 2025].
  • 4.4 Effect of Weight decay: With weight decay 0.05, linear-probing performance saturates around epoch 300, while stronger regularization improves generalization during extended pre-training.

4.5 Masking strategy · 4.6 Effect of learning rate

SiamJEPA compares random and block masking, finding that block masking is especially effective early while random masking remains competitive. Learning rates from 1.0 × 10−4 to 2.0 × 10−4 perform similarly, with 1.5 × 10−4 best overall in the reported setup.

  • 4.5 Masking strategy: The masking ablation compares random masking with block masking for SiamJEPA.For block masking, a block mask is centered at a randomly selected location, while masks for M1 and M2 are sampled disjointly from remaining visible regions.
  • 4.5 Masking strategy: The experiments use M1 and M2 masking ratios of 0.7, 0.75, and 0.8.Because the loss is symmetric, the effective masking ratios correspond to regions that do not contribute gradients for a given prediction direction.
  • 4.5 Masking strategy: Block masking consistently outperforms random masking in SiamJEPA, particularly at smaller training epochs.This agrees broadly with I-JEPA [Assran et al., 2023], where block masking and contextual information support strong performance.
  • 4.5 Masking strategy: Random masking remains competitive with block masking despite block masking’s stronger performance, especially early in training.
  • 4.6 Effect of learning rate: Learning rates between 1.0 × 10−4 and 2.0 × 10−4 produce comparable SiamJEPA performance.The learning-rate evaluation uses ImageNet linear probing with block masking at a 0.75 mask ratio and weight decay of 0.1.
  • 4.6 Effect of learning rate: A learning rate of 1.5 × 10−4 achieves the best results in the reported setup.The accompanying table notes that smaller learning rates tend to produce better results.

4.7 Effect of CLS token and Mean pooling

Mean pooling yields more effective linear-probing representations than the CLS token, with 10th-layer features becoming separable earlier while final-layer features eventually catch up. This indicates that semantic information remains distributed across informative patch representations.

  • 4.7 Effect of CLS token and Mean pooling: The ablation compares CLS-token representations with mean pooling from ViT-Base’s 10th and 12th layers under fixed decoder, masking, loss, and regularization settings.It uses decoder depth 1, free-bit 0.1, symmetric loss, block masking, and λKL weight decay 0.05 because weight decay 0.1 caused model collapse.
  • 4.7 Effect of CLS token and Mean pooling: During early training, mean pooling from ViT-Base’s 10th layer slightly outperforms final-layer pooling, but 12th-layer representations eventually improve to comparable generalization.The 10th layer acquires linearly separable semantic representations earlier, whereas the final layer initially becomes more specialized for the pretraining objective.
  • 4.7 Effect of CLS token and Mean pooling: Mean pooling outperforms the CLS token for linear probing, indicating that useful semantic information is distributed across patch representations.The result suggests learned patch-level features remain informative for downstream classification rather than being concentrated solely in the CLS token.

4.8 EMA scheduling · 4.9 Predictor depth

The EMA schedule and predictor depth materially affect SiamJEPA’s training behavior and linear probing performance. Smaller EMA momentum avoids mode collapse, while shallow predictors perform best, though both findings remain setting-dependent.

  • 4.8 EMA scheduling: Smaller EMA momentum (0.999) continues improving performance, whereas larger momentum (0.9999) leads to mode collapse.This comparison uses SiamJEPA with λKL = 0.01 and weight decay 0.1.
  • 4.8 EMA scheduling: The EMA schedule was empirically determined, leaving substantial room for improvement through more effective scheduling designs.The reported momentum comparison does not establish that the chosen schedule is optimal.
  • 4.8 EMA scheduling: A larger weight decay provides stronger regularization and appears more suitable for longer training, while smaller weight decay causes earlier performance plateaus.The evaluation uses ImageNet linear probing with block masking at mask ratio 0.75 and λKL = 0.01.
  • 4.9 Predictor depth: The effect of predictor depth remains unresolved because computational constraints and the large hyperparameter space prevent determining whether the observations generalize beyond the experimental setting.The authors explicitly leave this question open.
  • 4.9 Predictor depth: SiamJEPA achieves better linear probing performance with a shallow one- or two-layer Transformer predictor, so the paper uses a single-layer predictor.This predictor is substantially shallower than those commonly used in JEPA-based methods and could reduce parameter requirements if the observation generalizes.
  • 4.9 Predictor depth: Final linear probing performance depends on the evaluated encoder layer, suggesting that predictor depth may change which layer contains the most transferable representation.This behavior may differ for larger models or more challenging datasets, but investigating it requires substantially more computation.

5 Conclusion

The paper investigates Siamese student encoders within JEPA and proposes SiamJEPA, which combines masked Siamese student encoders with an EMA teacher to predict latent representations.

  • SiamJEPA incorporates Siamese student encoders and an exponential moving average (EMA) teacher network within the JEPA framework.
  • The framework processes two independently masked views with Siamese student encoders and predicts latent representations of unmasked target regions.
  • The study investigates the role of Siamese student encoders in JEPA-based representation learning.

6 Future work

Future work should test whether Siamese student encoder benefits generalize to larger models and video pretraining, while addressing sensitivity to implementation and hyperparameter choices. Neuroscience-inspired extensions beyond PhiNets may further improve representations and clarify links between biological and artificial learning.

  • Future directions: Siamese student encoder benefits should be evaluated in larger-scale architectures, including ViT-Large and ViT-Huge, beyond the relatively small models studied here.The current study used smaller architectures to isolate the role of Siamese student encoders, leaving scaling as an open question.
  • Future directions: Final performance is highly sensitive to implementation details and hyperparameters, leaving room for architectural refinements and more systematic optimization despite consistently outperforming masked autoencoder baselines.
  • Future directions: The study’s image-only pretraining scope motivates extending Siamese student encoders to video, where prior work reports effectiveness [Gupta et al., 2023, Jang et al., 2024, Yamada et al., 2025] and recent JEPA frameworks incorporate them [Daithankar et al., 2026, Rao et al., 2026].
  • Future directions: Building on SiamJEPA’s inspiration from PhiNets [Ishikawa et al., 2025, Yamada et al., 2025], additional neuroscience-inspired mechanisms may improve representation quality and illuminate biological–artificial learning relationships.
Loading 2607.04044v2…