PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teachers

Xianhang Li, Chen Huang, Chun-Liang Li, Eran Malach, Josh Susskind, Vimal Thilak, Etai Littwin

arXiv:2509.24317v1cs.LGcs.CV

TL;DR

JEPA methods rely on EMA-updated teachers to prevent collapse, but this complicates model selection and couples teacher and student architectures. SALT instead trains a pixel-reconstruction teacher and freezes it for masked-latent student training; it outperforms V-JEPA 2 under frozen evaluation and offers a better compute-accuracy trade-off. The authors leave the determinants of teacher quality and the mechanism behind gains from an additional student stage unresolved.

  • Problem

    In JEPA, co-evolving student and teacher representations permit collapsed near-zero-loss solutions, prompting EMA-based collapse prevention and complicating model selection.

    Source: §1 · p. 2

  • Method

    SALT trains a target encoder with pixel reconstruction, freezes it, and trains a student to predict its masked-region latents.

    Source: §1 · p. 2

  • Results

    SALT consistently outperforms strong baselines, including V-JEPA 2, under frozen evaluation, and achieves a better accuracy–FLOPs trade-off when teacher-training cost is counted.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    High-quality students can emerge from small, sub-optimal teachers, suggesting compute should be allocated predominantly to the student.

    Source: §1 · p. 2, §7 · p. 10

    What makes a “good” teacher remains unclear, and performance plateaus as model size grows in this setting, likely reflecting data limits.

    Source: §7 · p. 10

Abstract

from arXiv · show

Video Joint Embedding Predictive Architectures (V-JEPA) learn generalizable off-the-shelf video representation by predicting masked regions in latent space with an exponential moving average (EMA)-updated teacher. While EMA prevents representation collapse, it complicates scalable model selection and couples teacher and student architectures. We revisit masked-latent prediction and show that a frozen teacher suffices. Concretely, we (i) train a target encoder with a simple pixel-reconstruction objective under V-JEPA masking, then (ii) freeze it and train a student to predict the teacher's latents on masked regions. This leads to a two-stage, unregularized scheme that we refer to as SALT (Static-teacher Asymmetric Latent Training). SALT decouples optimization into pixel reconstruction (teacher) and masked latent prediction (student), increasing transparency, efficiency, and scalability while preserving the ability of representation to generalize under frozen evaluation. Empirically, our student models outperform recently proposed V-JEPA 2 encoders under frozen backbone evaluation across diverse benchmarks. They are also more compute-optimal: at matched pretraining FLOPs, our method achieves higher probing accuracy, and its scaling curves dominate V-JEPA's accuracy-FLOPs Pareto frontier. Finally, we find that student quality is remarkably robust to teacher quality: high-performing students emerge even with small, sub-optimal teachers. This points to a compute budget allocation that should overwhelmingly favor the student. These results position SALT as a simple, scalable, and compute-efficient alternative to EMA-based self-distillation for video representation learning.

1 Introduction

JEPA methods learn semantic representations by predicting masked regions in embedding space, but their co-evolving student and teacher require collapse-prevention machinery. SALT replaces the EMA teacher with a frozen target encoder and reports stronger compute-accuracy trade-offs, competitive students from small teachers, and a more informative training-time metric.

1 Introduction

  • SALT trains a target encoder with pixel reconstruction, freezes it, then trains a student to predict its masked-region latents, removing EMA and stop-gradient.The method is named Static-teacher Asymmetric Latent Training (SALT).Source: §1 · p. 2
  • High-quality frozen-evaluation features can be learned from much smaller, cheaper teachers, while stronger pretrained encoders yield at most marginal student gains.The authors characterize these smaller teachers as “sub-optimal.”Source: §1 · p. 2
  • SALT achieves a better accuracy–FLOPs trade-off than EMA-based self-distillation when accounting for teacher-training cost.The introduction identifies this comparison as a compute-efficiency result.Source: §1 · p. 2
  • SALT’s student loss correlates strongly with downstream accuracy under frozen-backbone evaluation, unlike EMA methods that require proxy heuristics for model selection.The loss is presented as an informative training-time metric.Source: §1 · p. 2
  • The authors conclude that elaborate online teacher-student dynamics, including EMA-based collapse prevention, may be unnecessary for high-quality representations.The claim is framed as a conclusion suggested by their results.Source: §1 · p. 2

2 Method Overview

The overview contrasts V-JEPA’s EMA-based masked-latent prediction with SALT’s two-stage frozen-teacher alternative. SALT separates teacher pixel reconstruction from student latent prediction, making the objectives easier to interpret and the teacher and student architectures independent.

2.1 V-JEPA

  • V-JEPA predicts masked regions in learned representation space using a context encoder and predictor supervised by an EMA-updated target encoder.The overview notes that V-JEPA 2 uses the same pretraining method with updated hyperparameters.Source: §2.1 · p. 2, §2 · p. 2

2.2 Static-teacher Asymmetric Latent Training (SALT) - A Simplified Video Representation Learning Method

  • SALT first trains a target encoder with pixel reconstruction under V-JEPA-style masking, then freezes it while optimizing a student and predictor with the JEPA objective.Stage 1 uses the VideoMAE pixel-reconstruction objective with a more efficient masking scheme; the Stage 2 teacher is frozen.Source: §2.2 · p. 3
  • SALT’s two objectives are easier to interpret and immune to representation collapse by design, while decoupled teacher and student architectures enable small teachers to train larger students.The overview reports a significant improvement over V-JEPA 2 on SSv2 and a noticeable improvement on Kinetics-400.Source: §2.2 · p. 3

2.2.1 SALT Design Principles

  • SALT’s V-Pixel teacher recipe uses pixel reconstruction with multi-block masking, and student quality remains robust to suboptimal data mixture, teacher size, and compute budget.The authors report that high downstream teacher performance is not necessary for a high-quality student.Source: §2.2.1 · p. 4

3 Experimental Setup

Experiments use a 3.6-million-video training mixture and ViT backbones, with optimization steps matched across SALT and baselines. Evaluation covers frozen-backbone video and image classification as well as intuitive-physics benchmarks.

3 Experimental Setup

  • Training uses approximately 3.6 million videos from Kinetics-710, Something-Something V2, and a 2.8-million subset of Panda70M.The resulting training set is called V-3.6M and differs from the V-JEPA and V-JEPA 2 training datasets.Source: §3 · p. 4
  • Experiments use ViT-L, ViT-H, ViT-giant, and ViT-G models, with optimization steps matched between SALT and baseline methods.The total Stage 1 and Stage 2 steps together equal the number of baseline training steps.Source: §3 · p. 4
  • Evaluation freezes the backbone for video and image classification and also tests intuitive-physics understanding on IntPhys, GRASP, and InfLevel.Video classification includes Kinetics-400, Something-Something-v2, COIN, Jester, and Diving-48; image classification uses ImageNet-1K.Source: §3 · p. 4

4 Experimental Results

Under frozen-backbone evaluation, SALT improves on V-JEPA 2 across scales, uses fewer pretraining FLOPs, and shows a predictive relationship between student loss and downstream accuracy.

Systematic comparison of SALT with existing baselines

  • Across scales, SALT encoders exceed V-JEPA 2 and remain competitive with Perception Encoder; the largest SALT models outperform all baselines on SSv2.Source: §4 · pp. 4–5, Table 1 · p. 5

Static teacher improves representation quality

  • 2.3% higher average accuracy than V-JEPA 2 across six benchmarks; SALT also improves as the student scales.Source: §4 · p. 5, Figure 2 · p. 5

Small teachers unlock compute efficiency

  • A fixed ViT-L teacher trains same-size and larger students, while SALT uses lower total pretraining FLOPs than the EMA-based baseline, even counting teacher pretraining.The savings grow with model size and spatial resolution.Source: §4 · pp. 5–6

SALT enables interpretable model selection

  • Student training loss predicts downstream accuracy, making it a useful signal for selecting student checkpoints.Source: Figure 3 · p. 6

5 Teacher Design Choice Ablation

This section examines teacher-data and masking choices, teacher size, and teacher–student compute allocation to identify design choices for SALT’s static teacher and student training.

5.1 Training Dataset

  • Students trained from each teacher-data choice improved over their corresponding teachers and exceeded V-JEPA 2 except when the teacher used ImageNet-1K.The video-data teachers had comparable performance except for the Something-Something-V2 teacher.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 Teacher Masking Strategy

  • Multi-block masking produced the strongest student, while the V-Pixel teacher achieved 72.5% accuracy with that masking strategy.Students trained with other masking-strategy teachers also improved substantially over their corresponding teachers.Source: Figure 5 · p. 7, §5.2 · p. 8

5.3 Teacher model size

  • Students trained from ViT-L teachers reached 77.4% average accuracy for ViT-L and 78% for ViT-G, including stronger results than larger-teacher alternatives.All student models improved over teachers of the same or smaller size.Source: §5.3 · p. 8

5.4 Impact of Teacher-Student Compute Allocation

  • At 240k total steps, the best student used a teacher trained for only 40k steps, supporting greater compute allocation to the student.SALT outperformed V-JEPA 2 across the tested optimization-step range and at approximately matched FLOPs.Source: §5.4 · p. 8, Figure 7 · p. 9

6 Related Work

Related work spans masking-based pixel reconstruction and latent prediction, including video methods with momentum teachers and methods distilling from frozen pretrained encoders. SALT is distinguished by its teacher choices and frozen-backbone evaluation.

Video foundation models

  • V-JEPA and V-JEPA 2 use online or momentum encoders, while SALT uses a frozen teacher for video representation learning.Other related approaches fix encoders for future-state prediction, whereas SALT targets representation learning.Source: §6 · pp. 9–10

Distillation from frozen teacher encoder

  • Prior frozen-teacher methods often rely on powerful pretrained encoders, whereas SALT reports that weaker frozen teachers can yield students that outperform EMA-based teachers.The cited methods include VideoMAE-based and vision-language-model teachers.Source: §6 · p. 9

Masked video distillation

  • VideoMAE-style masked video distillation (MVD) uses separate image and video encoders, while SALT focuses on video data and improves its teacher recipe.SALT uses teachers no larger than students and evaluates features with frozen backbones, unlike MVD’s fine-tuning evaluation.Source: §6 · p. 10

7 Limitations and Conclusion

The paper concludes that SALT outperforms strong baselines, including V-JEPA-2, in frozen evaluation and that smaller, sub-optimal teachers can produce strong students. It leaves teacher quality and scaling behavior incompletely characterized.

7 Limitations and Conclusion

  • SALT consistently outperforms strong baselines, including V-JEPA-2, across standard benchmarks in frozen-evaluation protocols.The authors describe the framework as simple, compute-efficient, and scalable for video representation learning.Source: §7 · p. 10
  • The ablations do not fully explain what makes a good teacher, and the mechanism behind modest gains from an additional student-training stage remains unclear.The authors focused their experiments on simple scalar diagnostics of teacher quality.Source: §7 · p. 10
  • Performance plateaus as model size grows in the studied setting, likely reflecting data limits; larger pretraining sets may extend the scaling trend.A fuller study of SALT’s scaling behavior is left for future work.Source: §7 · p. 10

A Training Dataset

The training dataset V-3.6M combines Kinetics, Something-Something-v2, and a subset of Panda70M.

A Training Dataset

  • V-3.6M combines Kinetics-710, Something-Something-v2, and approximately 2.8 million Panda70M clips.Kinetics-710 pools Kinetics-400/600/700 training samples after removing duplicates and samples in those datasets’ validation sets.Source: A Training Dataset · p. 15
  • The Panda70M subset was stratified to include clips lasting 4 to 50 seconds, with no other filtering or curation applied.Table 2 lists sample counts for V-3.6M.Source: A Training Dataset · p. 15, Table 2 · p. 15

B Architecture Details

SALT uses Vision Transformers for its video encoders and predictors, with a shared spatial and temporal patching scheme. The predictor output is dimensioned to match its teacher encoder.

B Architecture Details

  • All video encoders and predictors use ViTs with 16 × 16 spatial patches and temporal patches of size 2.The encoder and predictor architectures are detailed in Tables 3 and 4.Source: B Architecture Details · p. 15, Table 3 · p. 15, Table 4 · p. 15
  • RoPE encodes position, and the predictor’s final layer projects to the teacher encoder’s embedding dimension.ViT-L teachers are used for all encoders except the ViT-B model, which uses a ViT-B teacher.Source: B Architecture Details · p. 15

C Training Details

SALT first trains a teacher with V-Pixel, then trains a student using the frozen teacher. The reported results use 240,000 total steps, with additional step-budget ablations.

C Training Details

  • SALT trains the teacher first with V-Pixel, then trains the student against a frozen or static teacher.V-Pixel uses multi-block masking with hyperparameters copied from V-JEPA; Table 5 lists SALT training hyperparameters.Source: C Training Details · p. 16, Table 5 · p. 16
  • The reported results use 240,000 total steps; ablations use 120,000, 160,000, or 240,000 steps.Training uses cosine weight decay and AdamW with β2 = 0.95, and omits V-JEPA’s virtual early stopping; the authors report stable training with a frozen teacher.Source: C Training Details · p. 16

D Evaluation Details

The evaluation section describes attentive probing and benchmark-specific settings for comparing SALT with V-JEPA 2 and other baselines. It also outlines faster evaluation protocols and an intuitive-physics surprise measure.

D Evaluation Details

  • Kinetics-400 and SSv2 comparisons follow V-JEPA 2’s evaluation setting, using 16 frames, 8 clips, and 3 spatial views per input.The attentive probe uses four Transformer blocks followed by a linear classifier; its hyperparameters are reported in Table 6.Source: D Evaluation Details · pp. 16–17, Table 6 · p. 17
  • SALT probing uses 224 × 224 crops, versus 256 × 256 for V-JEPA 2, reducing compute during probing.The authors characterize SALT results as more remarkable given this lower probing compute.Source: D Evaluation Details · p. 17
  • For many ablations and main-paper results, the faster protocol uses one clip and one view while keeping 16 frames per input.Its evaluation hyperparameters follow V-JEPA; results are reported in Figures 1–9, with Table 7 listing the settings.Source: D Evaluation Details · p. 17, Table 7 · p. 17
  • COIN, Diving-48, Jester, and ImageNet-1K are evaluated as frozen-backbone benchmarks, with settings listed in Table 8.The results are reported in Figures 1–9.Source: D Evaluation Details · pp. 17–18, Table 8 · p. 18
  • The intuitive-physics protocol uses surprise scores from a context encoder, predictor or decoder, and target encoder to assess video pairs.The text defines C as the context-frame count and M as the future-frame count; it describes a global score over windows, with stride 2 and relative accuracy derived from video-pair labels.Source: D Evaluation Details · p. 18

E Additional Results

  • The additional-results section provides tables and results supporting figures and tables in the main paper.No specific result is stated in this passage.Source: E Additional Results · p. 18

E.1 Intuitive Physics Benchmarks

The additional analyses report SALT’s intuitive-physics comparisons, compute-allocation and teacher-checkpoint findings, compute estimates, and supplementary table and figure scopes.

E.1 Intuitive Physics Benchmarks

  • SALT’s intuitive-physics accuracy scales with model size and compares favorably with published baselines, while matched-data, matched-budget V-JEPA 2 compares favorably with SALT.The benchmark tests zero-shot physical attributes using surprise scores; Table 9 covers IntPhys, GRASP, and InfLevel.Source: §E.1 · pp. 18–19, Table 9 · p. 19

E.2 Impact of Teacher-Student Compute Allocation

  • The best allocation trains the teacher for 40,000 steps and uses the remaining budget for the student, which improves over its teacher.Figure 8’s x-axis is teacher steps, with the remainder assigned to student optimization.Source: §E.2 · p. 19, Figure 8 · p. 19

E.3 How to choose a Teacher checkpoint?

  • Neither teacher embedding rank nor teacher reconstruction loss predicts the student’s downstream performance in the reported analysis.Figure 9 compares these teacher measures with student accuracy under equal 160k-step stage budgets.Source: §E.3 · pp. 19–20, Figure 9 · p. 20

F Floating Point Operations (FLOPs) Estimation

  • SALT’s same-sized-or-smaller teachers and 224 × 224 inputs lower estimated training compute versus V-JEPA 2; teacher compute is approximated as 2ND.The estimate uses the 6ND Transformer formula and includes a GPU-hour comparison whose model ordering correlates with the FLOPs estimate.Source: F Floating Point Operations (FLOPs)… · pp. 20–21, Table 11 · p. 21

G Compute Budget specified via FLOPs and optimization steps

  • SALT specifies compute by optimization steps, and a trained teacher can be reused to train multiple students.The authors describe step counts as natural for the sequential teacher-then-student schedule.Source: G Compute Budget specified via FLOPs… · p. 21

H Additional Tables

  • Tables 12–16 cover same-pretraining-set comparisons, teacher-data, teacher-size and masking ablations, and matched-step compute–accuracy tradeoffs.The captions identify evaluation datasets and metrics, but do not provide outcome values.Source: Table 12 · p. 22, Table 13 · p. 22, Table 14 · p. 23

I Additional Figures

  • Figure 10 relates RankME to downstream accuracy using the same SALT Stage-1-80k teacher checkpoint.Source: Figure 10 · p. 24
Loading 2509.24317v1…