Source-linked AI summary
LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics
Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner
TL;DR
Video pretraining offers temporal information but remains computationally expensive because existing methods rely on architectural machinery or pixel reconstruction. LeVJEPA uses a single encoder with invariance training and SIGReg, achieving competitive video representations at much lower compute and supporting causal temporal modeling.
Problem
Video provides temporal structure unavailable in static images, but learning video representations remains expensive because clips contain many tokens and prevailing methods add architectural machinery.
Method
LeVJEPA trains a single encoder with an invariance loss between global and local clip views, regularizes embeddings with SIGReg, and controls compute through token dropping.
Results
LeVJEPA matches or exceeds V-JEPA 2 across ViT-S/B/L with 5.6 to 20.8× less pretraining compute, leads the strongest video baseline by 7.6 points on ImageNet-1K, and nearly doubles DINOv2's motion-centric accuracy.
Takeaways & Limitations
Video becomes a viable substrate for general-purpose visual pretraining, while block-causal attention makes temporal ordering a property of the encoder itself without measurable downstream accuracy cost.
Takeaways & Limitations
The objective's behavior at the model and data scales of recent video foundation models remains to be characterized, and dense prediction sufficiency has not been evaluated.
Abstract
from arXiv · showhide
Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing self-supervised methods either prevent representation collapse through architectural asymmetries, coupling an exponential-moving-average target encoder, a stop-gradient, and a capacity-limited predictor, or circumvent it by reconstructing masked content in pixel space. We introduce LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective, which dispenses with both. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg, which excludes collapse with a provable guarantee. The architecture reduces to an encoder and a projector, and the objective to a single hyperparameter. This formulation admits two properties. First, the cost of pretraining is governed by the number of tokens the encoder observes; uniform random token dropping renders this number small while simultaneously improving downstream accuracy. At matched epochs on identical data, LeVJEPA matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6 to 20.8x less pretraining compute, and at matched total FLOPs it exceeds the strongest video baseline by 7.6 points on ImageNet-1K while remaining competitive on motion-centric benchmarks. Second, since no asymmetry between branches is required, the encoder can be trained with block-causal attention at no measurable accuracy cost: temporal ordering becomes a property of the encoder itself. Against a compute-matched DINOv2 trained on frames of the same videos, LeVJEPA approaches the image-pretrained encoder on appearance-centric evaluation while nearly doubling its motion-centric accuracy. These results indicate that, once its computational overhead is removed, video becomes a viable and in several respects preferable substrate for general-purpose visual pretraining.
1 Introduction
LeVJEPA simplifies video pretraining to a collapse-free single-encoder objective and reduces compute by making encoder token count the primary cost. It matches or exceeds established video baselines while supporting causal representations and strong appearance and motion performance.
- Motivation: Video supplies motion, causality, and object permanence unavailable from static images, but its greater token count has made representation learning expensive.Leading video methods further add target encoders, predictors, and masking machinery to prevent collapse or structure learning.
- Approach: LeVJEPA trains one encoder with an invariance loss and SIGReg, eliminating the predictor, target encoder, stop-gradient, and associated stabilization machinery.Its trainable architecture contains only an encoder and projector, while the objective has one fixed hyperparameter.
- Results: 5.6 to 20.8× less total pretraining compute lets LeVJEPA attain accuracy comparable to or exceeding V-JEPA 2 across ViT-S/B/L at matched epochs and identical data.At equal FLOPs, it leads the strongest video baseline by 7.6 points on ImageNet-1K, reaches the highest Kinetics-400 accuracy, and remains competitive on Something-Something-v2.
- Image versus video pretraining: Against compute-matched DINOv2 trained on frames from the same videos, LeVJEPA approaches image-pretrained appearance performance while nearly doubling motion-centric accuracy.A ViT-Tiny trained for 12 hours on one consumer GPU also attains non-trivial ImageNet accuracy.
- Causal representations: Block-causal attention matches fully bidirectional attention under frozen probing, making temporal ordering intrinsic to the encoder without a separate temporal model.Frame representations use current and past frames and can extend to incoming frames without re-encoding.
- Efficiency: Uniform random token dropping reduces per-step computation while improving ImageNet accuracy from 33.9% with all tokens to 47.6% when 95% are discarded.The method makes observed token count a free design parameter rather than part of a prediction task.
2 Related Work
Prior visual pretraining methods avoid collapse through architectural asymmetry, embedding-distribution constraints, or pixel reconstruction. Video methods additionally use temporal masking and separate predictors, whereas LeVJEPA places temporal structure directly in the encoder.
- Image self-supervision: BYOL and DINO use online and EMA target branches with stop-gradient and predictors, while DINO adds multi-crop matching of local and global views.DINOv2 combines this framework with masked-image modeling and scales it on curated data.
- Embedding-distribution constraints: VICReg and Barlow Twins replace architectural asymmetry with invariance plus variance and covariance constraints, but restrict their distributional control to second-order statistics.LeJEPA instead characterizes the isotropic Gaussian as uniquely minimizing worst-case downstream probing risk and uses SIGReg to enforce it.
- Pixel reconstruction from video: VideoMAE uses tube masking because temporal redundancy makes naive masking easy to solve by copying content from adjacent frames.VideoMAEv2 adds a decoder-side mask while retaining pixel reconstruction as the learning task.
- Feature prediction and world models: V-JEPA predicts representations with an EMA target encoder and narrow predictor, while V-JEPA 2 scales the approach across data, model size, schedule, and resolution.V-JEPA 2-AC subsequently fits a separate block-causal predictor over frozen encoder outputs for planning and control.
- LeVJEPA distinction: LeVJEPA differs by placing the temporal constraint in the encoder during pretraining rather than fitting a separate temporal model afterward.This connects its collapse-free simplification with causal frame representations directly produced by the pretrained encoder.
3 Methodology: LeVJEPA
LeVJEPA transfers LeJEPA’s collapse-free objective to video using one shared encoder, global/local-view invariance, and SIGReg regularization. Its video transformer uses sparse token observations, block-causal attention, and frozen-probe evaluation across appearance- and motion-centric benchmarks.
- Training objective: LeVJEPA trains one video encoder with invariance between global and local clip views, while SIGReg explicitly excludes collapsed embeddings.The objective replaces architectural collapse-prevention asymmetries with a distributional constraint.
- Architecture: During pretraining, uniform random dropping discards 95% of patch tokens, leaving a sparse clip observation while determining computational cost.The learnable [cls] token is retained as the clip-level readout and is never dropped.
- Architecture: The trainable architecture comprises a video Vision Transformer and shared projector, with no predictor network or maintained target encoder.A Polyak average is retained only as an evaluation checkpoint and does not participate in the objective.
- Architecture: Block-causal attention lets patch tokens attend within their frame and causally across preceding frames, while the [cls] token attends to all tokens.This makes each frame representation depend only on current and past frames.
- Evaluation: Evaluation freezes pretrained encoders and probes Kinetics-400, Something-Something-v2, and ImageNet-1K for motion and object recognition.Kinetics-400 uses mean-pooled tokens with a linear classifier, whereas the other evaluations use attentive probing.
4 What Matters for Efficient Video Pretraining?
The design-space experiments show that efficiency choices can improve or preserve appearance representations, while their effects differ for motion understanding. Uniform random token dropping is especially effective, local views offer a modest accuracy–cost tradeoff, and temporal aggregation is unnecessary while causal attention preserves accuracy.
- 4.1 Token Dropping Improves Representations: 47.6% ImageNet accuracy at ρ = 0.95 exceeds 33.9% with all tokens processed, while token dropping reduces per-forward-pass cost.The cost reduction reaches a factor of (1 −ρ)^−1 in feed-forward layers.
- 4.1 Token Dropping Improves Representations: 47.4% versus 47.6% ImageNet accuracy at ρ = 0.9 and ρ = 0.95 shows that halving processed tokens leaves accuracy unchanged within observed variability.The most aggressive tested dropping ratio is therefore the most computationally economical configuration in this comparison.
- 4.1 Token Dropping Improves Representations: Motion accuracy declines beyond ρ = 0.3 on Something-Something-v2, but longer training recovers lower-dropping accuracy while retaining lower per-iteration cost.The authors leave motion-preserving dropping schemes at high sparsity for future work.
- 4.1 Token Dropping Improves Representations: 50.7% versus 39.6% ImageNet accuracy favors uniform random dropping over structured tube dropping, with the same ordering on Something-Something-v2.The comparison uses identical spatial retention patterns across frames for tube dropping.
- 4.2 Local View Budget: 47.6% at V = 4 rises to 50.2% at V = 10 and saturates at 49.8% at V = 12, so extra local views provide limited additional accuracy.Each added local view costs approximately 29 processed tokens, less than one fifth of the global-view cost.
- 4.3 Temporal Patch Aggregation Is Not Required: With matched retained-token budgets, the τ = 1 and τ = 2 configurations have indistinguishable dropping-ratio effects, while per-frame patching achieves higher accuracy on both benchmarks.The evaluation sequences are matched to eight temporal slots.
- 4.4 Causal Attention Incurs No Accuracy Penalty: Block-causal attention matches fully bidirectional attention under frozen probing, so temporal causality is obtained without measurable downstream accuracy loss.The causal topology restricts each frame to current and preceding frames.
5 Comparison with Prior Work
Under controlled data, schedule, and compute comparisons, LeVJEPA delivers strong efficiency against video baselines and narrows the appearance-transfer gap with image pretraining while retaining motion advantages.
- Comparison to Video Models: LeVJEPA matches or exceeds V-JEPA 2 across ViT-S/B/L using 5.6 to 20.8× less pretraining compute at matched epochs.At ViT-B, the methods are separated by less than one accuracy point while LeVJEPA uses 4.8 rather than 36.4 ExaFLOPs; at ViT-L, it surpasses V-JEPA 2 by 1.9 points.
- Comparison to Video Models: VideoMAEv2 ranks below LeVJEPA and V-JEPA 2 in accuracy while using an intermediate amount of compute.The comparison is reported under the epoch-matched ImageNet protocol.
- Comparison to Video Models: 7.6 points is LeVJEPA’s ImageNet-1K lead over the strongest video baseline under equal total pretraining FLOPs.It also attains the highest Kinetics-400 accuracy and remains competitive on Something-Something-v2.
- Comparison to Image Models: At matched compute, DINOv2 retains a 3.1-point ImageNet advantage, whereas LeVJEPA reaches 30.4% versus 16.9% on Something-Something-v2.DINOv2 was trained on individual frames from the identical video data at the same total pretraining FLOPs.
- Efficiency and Accessibility: A ViT-Tiny trained for 12 hours on one consumer GPU improved frozen-encoder ImageNet top-1 accuracy from 8.9% to 25.2%.The experiment used unlabeled, uncurated Walking Tours footage totaling approximately 620k frames.
6 Discussion
LeVJEPA simplifies video pretraining while reducing compute, supports causal representations, and broadens video’s role beyond motion-specific applications. Remaining questions concern motion performance under aggressive sparsity, scale, and dense prediction.
- A single encoder, invariance loss, and fixed regularizer achieve competitive video pretraining at substantially lower compute.The objective removes the target encoder, predictor, and associated schedules used for stability.
- At equal compute, video pretraining approaches DINOv2 on appearance-centric evaluation while nearly doubling its motion-centric accuracy.The comparison uses identical source data and total compute.
- Block-causal attention matches bidirectional attention without measurable downstream accuracy loss and provides temporally ordered representations without re-encoding.This supports streaming perception and autoregressive world modeling with a single encoder.
- Aggressive token dropping reduces Something-Something-v2 accuracy at short schedules, although longer training largely recovers the difference within the same compute budget.Preserving temporal correspondences at high sparsity is identified as a natural refinement.
- The evaluation is limited to a restricted corpus and models up to ViT-L, while large-scale behavior and dense prediction sufficiency remain unevaluated.The training signal is applied to a single clip-level token.
- LeVJEPA is presented as efficient enough for commodity hardware, causal by construction, and responsive to growing data.Together, these properties support video as a practical foundation for general-purpose representation learning.
A SIGReg Implementation Details
SIGReg is implemented with numerical quadrature and distributed aggregation over the global batch. Its communication cost is independent of batch size and embedding dimension.
- SIGReg approximates its integral by trapezoidal quadrature over 17 knots in t ∈[0, 3].The statistic uses M = 1,024 random directions per step and is computed separately for each view’s embeddings.
- The empirical characteristic function is averaged across workers with one all-reduce before computing SIGReg on the full global batch.The communicated object is a 2 × M × 17 tensor.
- Communication overhead is independent of both batch size and embedding dimension.
B Architectural Details
The default architecture uses a ViT-B/16 with rotary coordinates for time and space and a compact two-layer projector.
- Unless stated otherwise, experiments use a ViT-B/16 encoder without absolute positional embeddings.
- Rotary embeddings divide each attention head into temporal, vertical, and horizontal coordinate groups.
- The projector is a two-layer MLP mapping d to 2,048, then through batch normalization and GELU to K = 256.It is discarded after pretraining.
C Evaluation Details
Evaluation uses frozen-encoder probes, with attentive probing for token outputs and linear probing on pooled Kinetics-400 representations.
- Attentive probing uses one learnable query with cross-attention over all frozen encoder output tokens.The attended query is processed by a residual connection, MLP, normalization, and linear classifier.
- All encoder parameters remain frozen while the probe and classifier are trained on downstream data.
- For Kinetics-400, output tokens are averaged into one vector before training a linear classifier.