Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning

1. Introduction

• Masked and joint-embedding methods recover missing content from context, while autoregressive methods predict forward token by token but have not matched joint-embedding representation quality.Source: §1 · p. 1
• For still images, local-to-global crop sequences let VANE predict newly revealed regions, enabling joint image–video training without separate stages or modality-specific objectives.Source: §1 · p. 2
• VANE achieves state-of-the-art performance among unified encoders on most evaluated anticipation, global recognition, and dense perception benchmarks.On EPIC-KITCHENS-100, it achieves 45.5 action mean-class recall@5, 7.1 points above V-JEPA 2.1.Source: §1 · pp. 2–3
• VANE predicts the semantic embedding of an entire next frame from past frames, making temporal evolution the primary learning signal.Its causal encoder and state predictor use a frozen semantic tokenizer for targets, while an auxiliary generative decoder encourages fine-grained spatial detail.Source: §1 · pp. 2–3

2. Related Work

• Masked visual methods reconstruct pixels or predict masked representations, but remain centered on content within the observed clip rather than its temporal continuation.VANE contrasts this with predicting future-frame embeddings from past observations while preserving spatial detail through auxiliary reconstruction.Source: §2 · p. 3
• Autoregressive and predictive approaches include next-token prediction, reordered or progressively finer tokens, future-feature regression or contrastive learning, and pretrained-encoder world models.VANE trains the causal encoder itself with next-frame embedding prediction and extends it to image–video pretraining.Source: §2 · p. 3
• VANE predicts the next-frame embedding from observed frames through a causal encoder and state predictor, with image crops and video sequences sharing the objective.Source: Figure 2 · p. 4

3.1. Next-Frame Prediction in Embedding Space

• Each video transition trains the model to predict the next frame in a frozen tokenizer’s semantic embedding space rather than predict pixels.The tokenizer represents content and structure while abstracting over appearance variation.Source: §3.1 · p. 4

3.2. Causal Encoder

• Frame-causal attention lets each context state use only current and preceding frames, while patches within a frame attend bidirectionally.Tube masking supplies only visible context patches, leaving prediction targets unmasked.Source: §3.2 · p. 5

3.3. State Predictor and Semantic Objective

• The state predictor jointly infers each next-frame state from its own history, and projected predictions are trained against frozen tokenizer targets with patch-level L1 loss.Context states provide keys and values without being updated by the predictor.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4. Auxiliary Generative Reconstruction

• A lightweight decoder reconstructs future patches conditioned on predicted states; because futures are not fully determined, it models patch distributions with conditional flow matching.The reconstruction loss back-propagates through the predictor into the encoder.Source: §3.4 · pp. 5–6

3.5. Unified Image–Video Pretraining

• For images, local-to-global crops act as frames, with loss only on newly revealed regions; image and video batches receive equal weight.The progressively expanding context transfers better than fixed-order image cells and masked-image completion.Source: §3.5 · p. 6

4. Experiments

• The evaluation uses frozen encoder features with lightweight task probes, primarily comparing V-JEPA 2 and V-JEPA 2.1 at the same 1B scale.The experiments address anticipation, global classification, and dense perception before ablating design choices.Source: §4 · p. 6

4.1. Anticipation

• VANE reaches 45.5 action mean-class recall@5 on EK100, beating 1B V-JEPA 2.1 by 7.1 points, and reaches 7.81 mAP All on Ego4D STA v2 versus 6.75 for V-JEPA 2.1.The EK100 attentive probe uses frozen encoder features only; Ego4D evaluates localization and prediction of the next-active object and its interaction.Source: Table 1 · p. 7, Table 2 · p. 7, §4.1 · p. 7

4.2. Global Classification

• VANE beats V-JEPA 2.1 on all four global-classification benchmarks, raising average accuracy from 83.6 to 85.4, while V-JEPA 2 remains slightly better on Diving-48.VANE also scores 75.9 on SSv2 versus 63.1 for DINOv3 ViT-H+ and 50.7 for DINOv2 ViT-g.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3. Dense Perception

• VANE improves ADE20K and Cityscapes by 3.2 and 3.6 mIoU over V-JEPA 2.1, and lowers depth RMSE on NYUv2 and KITTI.It remains comparable on VOC (84.5 vs. 84.7); Figure 4 reports cleaner, more coherent ADE20K segments than V-JEPA 2.1.Source: §4.3 · p. 8, Figure 4 · p. 8, Table 4 · p. 9

4.4. Ablation Studies

• With DINOv2 targets fixed, next-frame prediction beats next-token prediction by 3.7 points on IN1K, 19.1 on SSv2, and 22.7 on ADE20K.Replacing pixel targets with frozen DINOv2 embeddings improves SSv2 by 16.2 points and ADE20K by 21.0 mIoU; DINOv2 is the default tokenizer.Source: §4.4 · p. 9, Table 5 · p. 9
• Local-to-global next-crop prediction performs best across the three image benchmarks, improving ADE20K by 2.0 and 6.8 mIoU over next-cell prediction and masked completion.Both ordered objectives outperform masked completion; next-crop supervision covers only regions newly revealed in each crop.Source: Table 5 · p. 9, §4.4 · p. 10
• Adding the generative decoder raises ADE20K from 42.8 to 45.1 mIoU, with smaller gains of 0.2 points on IN1K and 0.3 on SSv2.The authors suggest pixel-space reconstruction complements semantic prediction by preserving fine-grained spatial structure.Source: Table 5 · p. 9, §4.4 · p. 10
• The complete training recipe improves SSv2 and ADE20K by 3.6 and 5.9 points, combining model and data scaling, longer temporal context, and higher resolution.Long-video post-training raises SSv2 from 73.8 to 75.6.Source: §4.4 · p. 10

5. Conclusion

• VANE predicts the next frame in embedding space and extends the same objective to images through local-to-global next-crop prediction.The conclusion frames temporal evolution as the primary learning signal and describes the approach as a scalable principle for visual world modeling.Source: §5 · p. 10

AI use statement

• Generative AI tools assisted only with information search, language editing, and manuscript polishing; authors reviewed and revised all AI-assisted content.The authors state that they take full responsibility for the final manuscript.Source: AI use statement · p. 11

Reproducibility statement

• The paper and appendix provide method, training, dataset, architecture, sampling, and evaluation details for reproducing the experiments.Standard downstream benchmarks follow the protocols cited in the experimental sections.Source: Reproducibility statement · p. 11

Causal encoder

• The causal encoder processes observations in one pass, allowing tokens to attend within their observation and to earlier observations, but never to future ones.Video inputs use temporal-spatial tubelets spanning two sampled frames; each image crop is treated as one observation.Source: A. Model Architecture · p. 15

Context-isolated state predictor

• The state predictor reads historical encoder states as fixed keys and values, while next-observation prediction tokens attend to that context and one another.Predicted states are projected back to encoder width for semantic and generative objectives; different future observations remain isolated.Source: A. Model Architecture · p. 15, Figure 6 · p. 15

Semantic tokenizer

• A frozen DINOv2 ViT-L supplies normalized final patch-token targets, which are aligned across image crops and video frames for a shared semantic projection head.Video features at the same spatial position in a two-frame tubelet are concatenated; image features are duplicated to match the target dimension.Source: A. Model Architecture · pp. 15, 17

Generative decoder

• A patch-isolated six-layer DiffMLP decoder uses predicted states to decode target patches independently, without access to neighboring target patches.The main ViT-g model concatenates each predicted state with a noisy pixel token before projecting into decoder space.Source: A. Model Architecture · pp. 15, 17, Table 6 · p. 16

B. Multi-Stage Training Strategy

• The four-stage recipe begins with learning-rate warm-up, continues with base pretraining, and then increases temporal and spatial context through long-video post-training and high-resolution cooldown.The first two stages use 16-frame clips at 256 × 256 resolution.Source: B. Multi-Stage Training Strategy · p. 17

B.1. Input Sampling and Augmentation

• Videos are sampled at 4 FPS; base pretraining uses 16-frame clips, while long-video post-training expands the observation window to 64 frames.A temporal tubelet spans two consecutive sampled frames, and longer videos are randomly windowed during training.Source: §B.1 · p. 17

B.2. Local-to-Global Image Sampling

• Image training orders three crops from local to global and supervises only patch positions newly revealed by each subsequent crop.Two local crops precede a broader global crop; their original image coordinates are retained for training.Source: §B.2 · pp. 17, 20, Figure 7 · p. 18

B.3. Video Context Masking

• Random multi-block tube masks are applied to historical video context while future targets remain fully supervised.Two mask configurations use different spatial block scales, and their overlapping random placements produce a variable masked-token fraction.Source: §B.3 · p. 20

B.4. Warm-Up

• Warm-up linearly raises the learning rate from 1 × 10^-4 to 6 × 10^-4 over 12K updates before large-scale joint image–video pretraining.Training starts from randomly initialized VANE parameters while the semantic tokenizer remains frozen.Source: §B.4 · p. 20

B.5. Base Pretraining

• Base pretraining runs for 180K updates at a constant 6 × 10^-4 learning rate, processing image and video batches in separate passes before accumulating gradients for one optimizer step.The stage retains 16-frame clips and 256 × 256 inputs.Source: §B.5 · p. 20

B.6. Long-Video Post-Training

• Long-video post-training extends the temporal context from 16 to 64 frames at 256 × 256 for 60K additional updates.The stage uses a global video batch size of 768.Source: §B.6 · p. 20

B.7. High-Resolution Cooldown

• The final stage raises spatial resolution from 256 × 256 to 384 × 384 for 12K updates while retaining a 64-frame observation window.It starts from the long-video checkpoint and linearly decays the learning rate.Source: §B.7 · p. 20
• The two post-training stages require approximately 30.7K GPU-hours in total, compared with 26.6K GPU-hours for warm-up and base pretraining.Source: §B.7 · p. 20
• The multi-stage strategy proceeds from 16-frame clips at 256 × 256 to 64-frame clips, then a high-resolution stage.Source: Table 7 · p. 21

C.1. Qualitative Short-Term Anticipation

• In every Ego4D STA v2 example, a top-scoring prediction matches the ground-truth verb–noun pair and localizes the correct object with comparable time to contact.Other candidates are typically plausible alternatives in the same scene.Source: §C.1 · p. 21, Figure 9 · p. 22

C.2. Qualitative Dense Perception

• ADE20K and Cityscapes examples compare input images, ground truth, and semantic-segmentation predictions from identical dense readouts on frozen features.These qualitative examples complement aggregate metrics by illustrating preservation of object boundaries and fine-grained scene structure.Source: §C.2 · p. 21
• NYUv2 and KITTI examples show monocular-depth predictions, spanning indoor geometric structure and long-range outdoor scenes with different depth statistics.Predictions use the same frozen-feature evaluation protocol as the quantitative experiments.Source: §C.2 · p. 21

• The analyses cover prediction targets, image-prediction strategies, generative supervision, scaling, and the training schedule.Source: D. Additional Ablation Studies · p. 22
• The appendix first compares VANE with V-JEPA 2.1 under matched pretraining, then examines image crops, state-predictor attention, and prediction granularity.Source: D. Additional Ablation Studies · p. 22
• Unless otherwise specified, ablations use a ViT-L encoder trained for 50K iterations.Source: D. Additional Ablation Studies · p. 22

D.1. Masked versus Next-Frame Prediction under Matched Pretraining

• The controlled comparison matches VISTA pretraining data, ViT-L encoder scale, and 50K iterations while contrasting bidirectional masked feature prediction with next-frame prediction.Frozen encoder features are evaluated using the same downstream protocols as in the main paper.Source: §D.1 · p. 22
• VANE improves IN1K accuracy by 2.7 points and ADE20K by 6.7 mIoU, with SSv2 accuracy of 72.3% versus 72.0% for V-JEPA 2.1.The comparison matches pretraining corpus, encoder scale, and iteration budget; Table 8 reports top-1 accuracy on IN1K and SSv2 and mIoU on ADE20K.Source: §D.1 · p. 22, Table 8 · p. 26
• The matched-target comparison replaces V-JEPA 2.1’s EMA target encoder with the same frozen DINOv2 ViT-L tokenizer used by VANE.This variant retains V-JEPA 2.1’s bidirectional masked feature-prediction objective.Source: §D.1 · p. 22
• With both pretraining corpus and semantic target matched, VANE retains gains of 1.8 points on IN1K, 1.7 points on SSv2, and 3.6 mIoU on ADE20K.The frozen DINOv2 target improves the masked baseline on IN1K and ADE20K but decreases SSv2 accuracy.Source: §D.1 · p. 26

D.2. Number of Image Crops

• Across two, three, and four local-to-global crops, performance is relatively stable, with three crops yielding the best overall results.Two views provide less intermediate context, while four views bring no further improvement; three crops are the default.Source: §D.2 · p. 26, Table 9 · p. 26

D.3. State-Predictor Attention

• Context-isolated attention keeps historical encoder states read-only, while prediction tokens attend to them to construct the future visual state.The control instead uses causal self-attention to jointly update history and prediction tokens under the same causal boundary and otherwise unchanged setup.Source: §D.3 · pp. 26–27
• 3.3 points on IN1K accuracy, 5.5 points on SSv2 accuracy, and 9.6 mIoU on ADE20K favor context-isolated prediction over causal self-attention.Table 10 reports the ablation; the largest gain is on ADE20K.Source: §D.3 · p. 26, Table 10 · p. 27
• The results favor separating history encoding from future-state prediction, with encoder outputs carrying historical information and the predictor reading those states.The reported frozen-feature transfer improvement is particularly notable for dense perception.Source: §D.3 · p. 27

D.4. Prediction Granularity

• Next-token prediction orders patches temporally then spatially, predicting each token from preceding frames and earlier tokens in its current frame.The comparison holds the semantic target, encoder scale, training schedule, batch size, resolution, and clip length fixed.Source: §D.4 · p. 27
• Next-frame prediction jointly predicts an entire target frame from preceding observations, with all patches in observed frames available but none from the target frame.The two objectives use the same semantic and generative objectives but factorize the sequence differently.Source: §D.4 · p. 27