PaperlayerView exact revision on arXiv

Source-linked AI summary

VL-JEPA: Joint Embedding Predictive Architecture for Vision-language

Delong Chen, Mustafa Shukor, Theo Moutakanni, Willy Chung, Jade Yu, Tejaswi Kasarla, Yejin Bang, Allen Bolourchi, Yann LeCun, Pascale Fung

arXiv:2512.10942v2cs.CV

TL;DR

VL-JEPA targets the training cost and inference latency of token-generative vision-language models, especially for real-time and multi-task applications. It predicts continuous target-text embeddings with a JEPA-based architecture and decodes them only when needed. The model improves matched-training performance with fewer trainable parameters, reduces decoding operations by 2.85×, and supports classification, retrieval, and VQA in one architecture.

  • Problem

    Token-generative VLMs spend computation modeling surface linguistic variation and incur latency from autoregressive decoding, limiting efficient real-time vision-language processing.

  • Method

    VL-JEPA uses visual and target-text encoders plus a predictor trained to map visual embeddings and textual queries to continuous target embeddings, with optional text decoding at inference.

  • Results

    VL-JEPA outperforms matched token-predictive VLMs while using roughly half the trainable parameters, reduces decoding operations by 2.85×, and performs strongly across classification, retrieval, and VQA benchmarks.

  • Takeaways & Limitations

    Embedding prediction provides a unified route to generation, selective decoding, open-vocabulary classification, text-to-video retrieval, and discriminative VQA.

Abstract

from arXiv · show

We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA). Instead of autoregressively generating tokens as in classical VLMs, VL-JEPA predicts continuous embeddings of the target texts. By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability. In a strictly controlled comparison against standard token-space VLM training with the same vision encoder and training data, VL-JEPA achieves stronger performance while having 50% fewer trainable parameters. At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text. We show that VL-JEPA natively supports selective decoding that reduces the number of decoding operations by 2.85x while maintaining similar performance compared to non-adaptive uniform decoding. Beyond generation, the VL-JEPA's embedding space naturally supports open-vocabulary classification, text-to-video retrieval, and discriminative VQA without any architecture modification. On eight video classification and eight video retrieval datasets, the average performance VL-JEPA surpasses that of CLIP, SigLIP2, and Perception Encoder. At the same time, the model achieves comparable performance as classical VLMs (InstructBLIP, QwenVL) on four VQA datasets: GQA, TallyQA, POPE and POPEv2, despite only having 1.6B parameters.

1 Introduction

VL-JEPA addresses the cost and latency of token-generative VLMs by predicting semantic target embeddings instead of reconstructing token sequences. Its unified architecture supports efficient real-time inference and multiple vision-language tasks, with stronger matched-training performance and fewer trainable parameters.

  • Motivation: Real-world vision-language systems require low-latency, low-cost responses for tasks including captioning, retrieval, VQA, tracking, reasoning, and planning.These requirements are especially important for wearable devices, robots, and live video applications.
  • Limitations of Existing VLMs: Token-generative VLMs model both task-relevant semantics and task-irrelevant linguistic variation, increasing training effort and producing unnecessary token sequences.Autoregressive decoding also delays semantic updates for streaming video because token-by-token generation must finish before the response is revealed.
  • Approach: VL-JEPA predicts target embeddings from visual embeddings and textual queries, optimizing embedding-space distance rather than token-space reconstruction.A lightweight decoder converts predicted embeddings to text only when needed.
  • Controlled Evaluation: Under matched training conditions, VL-JEPA achieves consistently higher zero-shot captioning and classification performance with roughly half the trainable parameters.The controlled comparison uses the same vision encoder, resolution, frame rate, training data, batch size, and iteration count.
  • Efficiency: 2.85× fewer decoding operations are achieved through selective decoding while preserving output quality measured by average CIDEr scores.Decoding occurs when the predicted embedding stream changes significantly.
  • Multi-task Capability: VL-JEPASFT handles classification, retrieval, and VQA within one architecture, matching established VLM families on four VQA datasets while VL-JEPABASE surpasses CLIP, SigLIP2, and Perception Encoder across eight classification and eight retrieval datasets.The SFT stage improves classification through exposure to in-domain data and equips the model with VQA capabilities.

2 Methodology

VL-JEPA combines visual and textual encoders with a predictor that learns target-text embeddings, then decodes them only when human-readable output is required. The same embedding space supports generation, discriminative VQA, open-vocabulary classification, retrieval, and selective streaming updates.

  • Architecture: VL-JEPA trains on visual input, a textual query, and a textual target represented as triplets ⟨X_V, X_Q, Y⟩.The architecture contains an X-Encoder, Predictor, Y-Encoder, and Y-Decoder.
  • Architecture: The X-Encoder maps visual inputs to compact continuous embeddings, while the Predictor maps visual embeddings and a textual query to a predicted target embedding.The predictor is the core component and uses the query as conditioning.
  • Target Representation: The Y-Encoder maps textual targets into a continuous latent space intended to abstract away task-irrelevant information, and the Y-Decoder translates predictions into text only at inference when necessary.The decoder is not involved during the main VL-JEPA training phase.
  • Training Objective: VL-JEPA replaces token-space loss with embedding-space prediction and uses regularization to avoid representation collapse.The implementation adopts InfoNCE, whose terms include representation alignment between normalized prediction and target embeddings.
  • Training Objective: Embedding-space learning groups semantically similar alternative answers more closely than sparse one-hot token representations, simplifying ill-posed prediction targets.This can reduce the need to fit multiple disjoint high-density regions in token space.
  • Multi-tasking: A single unified architecture supports text generation, discriminative VQA, open-vocabulary classification, and text-to-video retrieval by comparing predicted and candidate-text embeddings.For retrieval, candidate videos are ranked by similarity to the encoded textual query.
  • Selective Decoding: Non-autoregressive prediction produces continuous semantic streams that can be monitored and selectively decoded for low-latency online video applications.The design targets streaming settings such as action tracking, scene recognition, planning, navigation, and robotics.

3 Implementation of VL-JEPA

VL-JEPA is implemented as a two-stage vision-language system with separate encoders and a predictor operating in a shared embedding space. Large-scale pretraining establishes alignment, followed by supervised finetuning for VQA while retaining classification and retrieval capabilities.

  • Architecture: The X-Encoder uses a frozen V-JEPA 2 ViT-L with 304M parameters and uniformly sampled 256² video frames.
  • Architecture: The predictor initializes eight Transformer layers from Llama-3.2-1B, yielding 490M trainable parameters, while jointly attending to vision and query embeddings.Average pooling over non-[PAD] query tokens produces the predicted target embedding.
  • Architecture: The Y-Encoder initializes from EmbeddingGemma-300M, and projection heads create a shared 1,536-dimensional embedding space.
  • Two-stage training: Pretraining combines image-text data from Datacomp and YFCC-100M with video-text data from Action100M, progressing from one-frame images to eight- and 32-frame videos.The resulting model is called VL-JEPABASE and is evaluated for zero-shot classification and retrieval.
  • Two-stage training: Supervised finetuning uses a mixture containing 25M VQA, 2.8M captioning, and 1.8M classification samples to add VQA capabilities while preserving pretrained alignment.The model is trained for 83k steps with batch size 3,072.

4 Experiments

The experiments evaluate VL-JEPA across classification, retrieval, VQA, and world-modeling benchmarks. Results show strong zero-shot performance against generalist baselines and resource-efficient multitask capability after finetuning.

  • Evaluation setup: Evaluation covers eight classification datasets, eight retrieval datasets, and discriminative VQA benchmarks emphasizing visual perception.Zero-shot comparisons use CLIP, SigLIP2, and Perception Encoder, while VQA evaluation includes GQA, TallyQA, POPE, and POPEv2.
  • Classification and retrieval: 52.5 versus 44.7 average accuracy across 8 classification datasets and 63.7 versus 58.1 average recall@1 across 8 retrieval datasets favor VL-JEPABASE over PE-Core-G.
  • Classification and retrieval: VL-JEPABASE is stronger on motion-centric benchmarks but relatively weaker on appearance-centric benchmarks, while VL-JEPASFT approaches individually optimized specialist models.The reported difference is discussed alongside substantially fewer vision-language pairs than PE-Core-G.
  • VQA and multitasking: VL-JEPASFT outperforms many established VLM baselines while using significantly fewer computational resources through a unified architecture and single embedding space.The same embedding space supports VQA, classification, and retrieval.

4.3 WorldPrediction-WM

WorldPrediction-WM tests whether VL-JEPA can identify actions explaining transitions between initial and final world states. The model selects among candidate video clips by comparing state and action embeddings.

  • Evaluation setup: The benchmark provides initial and final images and asks the model to identify, among four candidate clips, the action explaining the transition.
  • Evaluation setup: VL-JEPA compares a state embedding with encoded action-candidate embeddings and selects the closest candidate.
  • Results: 65.7% top-1 accuracy makes VL-JEPASFT a new state of the art on WorldPrediction-WM.VL-JEPABASE attains 63.9% top-1 accuracy.
  • Results: VL-JEPA surpasses existing VLMs of comparable or larger scale and exceeds the listed frontier LLMs on this benchmark.

4.4 Action Anticipation

Finetuned VL-JEPA is evaluated for future-action prediction on EPIC-KITCHENS-100 and COIN. It performs strongly across anticipation settings and establishes a reported best result on COIN next-step forecasting.

  • EPIC-KITCHENS-100: VL-JEPA outperforms V-JEPA2 with a ViT-L-256px vision encoder across all EPIC-KITCHENS-100 anticipation times.
  • Evaluation setup: Action anticipation forecasts the next action after a specified temporal gap from a context-video window on EPIC-KITCHENS-100 and COIN.
  • EPIC-KITCHENS-100: 34.18 Recall@5 at the standard 1-second anticipation interval surpasses V-JEPA 2 with the same ViT-L-256px encoder by 1.48 points.
  • COIN: 56.2% on COIN next-step forecasting outperforms the listed VideoLLM-online, VideoLLM-MoD, and ProVideLLM baselines.The results support finetuning VL-JEPA for semantic uncertain prediction tasks.

4.5 Embedding Prediction vs. Token Prediction: A Controlled Comparison

Under matched training conditions, VL-JEPA predicts embeddings instead of tokens and achieves stronger captioning and classification performance with fewer trainable parameters. Its embedding stream also supports adaptive decoding that reduces decoding operations while preserving output quality.

  • Controlled training setup: VL-JEPA and the VLM baseline share the same vision encoder, resolution, frame rate, training data, batch size, and iterations; only the prediction objective differs.VL-JEPA uses a 0.5B predictor for target embeddings, whereas the baseline uses a 1B LLM for next-token prediction.
  • Performance: 14.8 CIDEr and 41.0% top-5 accuracy for VL-JEPA versus 7.1 CIDEr and 27.2% for the VLM at 15M samples.The comparison reports video captioning CIDEr and video classification top-5 accuracy, respectively.
  • Performance: At 5M samples, VL-JEPA reaches 14.7 CIDEr and 35.3% top-5 accuracy, showing a sharper performance increase than the VLM.The baseline reaches 7.1 CIDEr and 27.2% top-5 accuracy at 15M samples.
  • Performance: Both models have comparable performance after 500K samples, with 1.23 versus 1.35 CIDEr and 14.9% versus 14.0% top-5 accuracy for VL-JEPA and VLM.These values are reported for video captioning and video classification, respectively.
  • Selective decoding: Adaptive selection Pareto-dominates uniform sampling across decoding frequencies, with selection guided by predicted embeddings rather than fixed intervals.Average pooling provides consistent gains for both strategies by denoising and stabilizing embeddings before decoding.

4.7 Evaluation of Y-Encoder

The Y-Encoder evaluation tests sensitivity to semantic and lexical changes using hard-negative text benchmarks. VL-JEPA_BASE achieves higher triplet-based accuracy than the strongest listed alternative on both datasets.

  • Results: 63.9% on SugarCrepe++ and 42.9% on VISLA are achieved by VL-JEPA_BASE in micro-average accuracy.The benchmarks use triplets containing two semantically similar descriptions and one altered negative description.
  • Results: 58.6% on SugarCrepe++ and 40.4% on VISLA are achieved by the strongest other models, PE-Core and SigLIP2 respectively.VL-JEPA_BASE exceeds PE-Core on SugarCrepe++ and SigLIP2 on VISLA.
  • Results: 58.4% on SugarCrepe++ and 39.5% on VISLA are achieved by the finetuned VL-JEPA_SFT model.The table reports triplet-based accuracy percentages for both datasets.

4.8 Ablation Study

The ablation study examines pretraining, learning-rate scaling, loss choice, predictor design, and encoder selection. Results identify several settings associated with stronger classification, retrieval, and VQA performance.

  • Pretraining: Dropping query-free pretraining hurts classification by 21.7 and retrieval by 17.3.The ablation evaluates average classification top-1 accuracy and text-to-video retrieval recall@1 across eight datasets each.
  • Learning rate: A learning-rate multiplier of around 0.05 to 0.10 for the Y-Encoder is the reported sweet spot; faster or slower learning degrades performance.This result is part of the SFT-stage ablation study.
  • Loss function: InfoNCE generally outperforms cosine, L1, and L2 losses, while cosine performs better than InfoNCE on VQA.Only InfoNCE includes anti-collapse regularization and supports an unfrozen Y-Encoder.
  • Predictor: Using causal rather than bidirectional attention reduces VQA performance by 1.9.The reported explanation is that visual tokens cannot attend to query tokens when query tokens are appended after visual tokens.
  • Encoder and predictor: Larger encoders generally perform better, while visually aligned PE text encoders have a significant advantage in classification and retrieval.The ablations also report that more predictor layers especially improve VQA performance and that Llama-3 initialization benefits VQA.

5 Related Works

Related work spans JEPA representation learning, CLIP-style joint embeddings, generative VLMs, efficiency methods, and latent-space language modeling. VL-JEPA is positioned as a general-purpose vision-language JEPA that combines task coverage with latent-space prediction and selective readout.

  • JEPA models: VL-JEPA extends JEPA from unimodal representation learning and narrow world-modeling domains to general-purpose vision-language tasks.It performs conditional latent prediction over vision and text.
  • Vision-language models: CLIP-style models independently encode images and texts into a shared latent space, whereas generative VLMs connect vision encoders to language models trained in text space.The related-work comparison distinguishes non-predictive joint embeddings from autoregressive generation.
  • Positioning: VL-JEPA combines CLIP-like open-domain features with conditional generation through a readout text decoder.Its embedding-space training can leverage web-scale noisy image-text pairs.
  • Positioning: VL-JEPA is described as more efficient to learn than generative VLMs because it optimizes in latent rather than data space.The paper frames this as integrating architectural advantages and task coverage from CLIPs and VLMs.
  • Efficient VLMs: Efficiency research addresses training through parameter updates and inference through pruning parameters or visual tokens.These approaches form the surrounding context for VL-JEPA's efficiency claims.
  • Latent-space modeling: Latent-space language-modeling work explores unimodal latent reasoning, while VL-JEPA targets a shared multimodal latent space for vision and text.The paper presents this direction as a foundation for future multimodal latent-space reasoning.

6 Conclusion

VL-JEPA shifts supervision from discrete token generation to continuous semantic embeddings, simplifying learning and supporting non-autoregressive prediction. The model outperforms matched generative VLMs while improving efficiency, enabling unified classification, retrieval, and real-time video applications.

  • VL-JEPA shifts supervision from discrete token space to continuous semantic embedding space, avoiding redundant modeling of surface linguistic variability.
  • VL-JEPA outperforms generative VLMs trained with cross-entropy loss under matched training data budgets while achieving superior training efficiency and lower inference latency.
  • The embedding-based design supports open-vocabulary classification and cross-modal retrieval within a single unified architecture.
  • Continuous semantic embeddings make VL-JEPA suited to real-time video applications, where selective decoding can improve responsiveness and efficiency.
Loading 2512.10942v2…