PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model

Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan, Mingfei Chen, Jianglin Lu, Ang Li, Yun Fu

arXiv:2603.22281v2cs.CVcs.AIcs.CLcs.LGcs.RO

TL;DR

Existing latent world models have limited temporal context and semantic grounding, while VLMs are poorly suited to dense fine-grained prediction. ThinkJEPA combines dense JEPA dynamics with long-horizon VLM guidance through dual-temporal sampling and hierarchical multi-layer representation extraction, outperforming reported baselines across four benchmarks with more robust long-horizon behavior.

  • Problem

    JEPA-style latent world models often use short dense observation windows that limit temporal context and weaken long-horizon semantic grounding.

  • Method

    ThinkJEPA uses dense JEPA dynamics and a uniformly sampled VLM thinker, injecting hierarchical multi-depth VLM representations into the latent predictor.

  • Results

    ThinkJEPA consistently outperforms VLM-only, JEPA-only, and task-specific trajectory baselines across EgoDex, EgoExo4D, BAIR Robo, and Physion.

  • Takeaways & Limitations

    The framework preserves JEPA-style latent forecasting while enriching future prediction with long-horizon semantic and knowledge-aware guidance.

  • Takeaways & Limitations

    Rollout remains susceptible to error accumulation and is limited by the local temporal context within each prediction window.

Abstract

from arXiv · show

Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility. Vision--language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets. We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance via a dual-temporal pathway: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM \emph{thinker} branch with a larger temporal stride for knowledge-rich guidance. To transfer the VLM's progressive reasoning signals effectively, we introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction. Experiments on hand-manipulation trajectory prediction show that our method outperforms both a strong VLM-only baseline and a JEPA-predictor baseline, and yields more robust long-horizon rollout behavior.

1 Introduction

ThinkJEPA addresses limited temporal context and weak semantic grounding in JEPA-style latent world models by combining dense dynamics prediction with VLM-based long-horizon guidance. Its hierarchical representation extraction and dual-temporal design support stronger trajectory prediction across multiple benchmarks.

  • JEPA-style latent world models can capture fine-grained motion but may miss long-horizon semantics because prediction commonly uses short, densely sampled observation windows.
  • VLMs provide long-range context, entity understanding, and general world knowledge, but sparse compute-constrained sampling makes them unsuitable as standalone dense predictors.
  • A hierarchical pyramid module aggregates multi-depth VLM representations to preserve visual reasoning signals and produce guidance features compatible with the JEPA predictor.
  • ThinkJEPA combines a dense-frame JEPA pathway for motion and interaction cues with a uniformly sampled VLM pathway for long-horizon semantic guidance.
  • ThinkJEPA consistently improves over diverse latent world-model and trajectory-prediction baselines across EgoDex, EgoExo4D, BAIR Robot Pushing, and Physion.

2 Related Works

The related work positions latent world models, vision-language models, and multimodal fusion as neighboring foundations for combining efficient predictive abstractions with semantic guidance.

  • Latent World Models: Latent world models learn predictive environmental abstractions in representation space for forecasting, planning, and control.
  • Vision-Language Models: Vision-language models align visual and textual modalities through large-scale data to support multimodal representation learning and understanding.
  • Multimodal Fusion: Multimodal fusion uses language as a high-level signal for visual generation and embodied decision-making systems.

3 Methodology

ThinkJEPA combines dense JEPA forecasting with a uniformly sampled VLM thinker to reconcile fine-grained dynamics with long-horizon semantic context. Its multi-depth guidance is injected into the JEPA predictor, while recursive rollout extends forecasts beyond one observation window but remains vulnerable to error accumulation.

  • Dual-Temporal Perception Field Sampling Architecture: ThinkJEPA constructs synchronized dense and uniformly sampled clips, assigning fine-grained latent forecasting to JEPA and long-range semantic reasoning to the VLM thinker.The VLM branch uses a larger temporal perception field, while the JEPA branch retains high-frequency motion and interaction cues.
  • Dual-Temporal Perception Field Sampling Architecture: Uniform VLM sampling provides long-horizon context under compute constraints, but discards dense motion details and is therefore used for semantic guidance rather than dense dynamics prediction.The implementation uses Qwen3-VL (Thinking) and caches intermediate representations for conditioning.
  • JEPA-style Latent Tokenization and Forecasting: The dense JEPA branch encodes per-frame spatial tokens and forecasts future latent tokens from past tokens for downstream trajectory regression.Predicted latents serve as target representations for downstream heads, while VLM guidance supplies complementary semantic context.
  • JEPA-style Latent Tokenization and Forecasting: Recursive rollout repeatedly feeds predicted future latents into subsequent prediction steps, enabling arbitrarily long-horizon latent forecasts.This strategy extends beyond the JEPA observation window but is limited by local temporal context and susceptible to error accumulation.
  • VLM Thinker and Guidance Injection: VLM guidance is injected into the JEPA predictor layer-wise, conditioning future latent prediction on dense past tokens, VLM features, and an optional text prompt.The prompt is generated from a general summarization request to focus the thinker on relevant entities and events.
  • VLM Thinker and Guidance Injection: A hierarchical pyramid module combines visual tokens and intermediate language-model states, pooling and projecting multi-depth signals into predictor-compatible guidance features.The design preserves visual reasoning cues across VLM depth rather than relying only on final-layer representations.

4 Experiments

ThinkJEPA is evaluated across trajectory forecasting, robotic latent rollout, physical-scene forecasting, ablations, and recursive long-horizon prediction. Across these settings, it consistently improves over strong latent-world-model, VLM-only, and task-specific baselines while combining dense dynamics with long-horizon guidance.

  • Experimental setup: ThinkJEPA is evaluated on egocentric trajectory forecasting, robotic latent rollout, and action-free physical-scene forecasting.The benchmarks are EgoDex, EgoExo4D, BAIR Robot Pushing, and Physion.
  • Main comparisons: ThinkJEPA achieves lower ADE/FDE and higher Acc than single-branch baselines on EgoDex and EgoExo4D.It also improves latent forecasting metrics FD, SL1, and CD, surpassing both V-JEPA Predictor and Qwen3-VL Thinking.
  • Task-specific trajectory comparison: ThinkJEPA achieves the best ADE/FDE among the compared EgoDex trajectory prediction methods.The comparison includes decoder-only and encoder-decoder predictors trained with Behavior Cloning, DDPM, and Flow Matching objectives.
  • BAIR Robot Pushing: ThinkJEPA reduces BAIR latent rollout FD from 68.711 to 67.049, SL1 from 1.117 to 1.086, and CD from 0.301 to 0.285 versus V-JEPA Predictor.The evaluation uses action-conditioned three-step rollout in frozen V-JEPA latent space.
  • Physion: ThinkJEPA improves Physion object-contact prediction over V-JEPA Predictor, raising Acc from 0.741 to 0.754 and AUC from 0.821 to 0.828.All non-oracle methods use only the observation prefix.
  • Ablations: The all-layer pyramid variant performs best overall, while combined encoding and autoregressive VLM tokens outperform either source alone.Layer selection shows a trade-off: last-layer guidance favors trajectory metrics, whereas mid-layer guidance favors latent forecasting quality.
  • Long-horizon recursive rollout: ThinkJEPA achieves the best trajectory performance across all evaluated rollout horizons, whereas Qwen3-VL Thinking degrades sharply and V-JEPA Predictor accumulates error.Recursive rollout is evaluated at horizons H ∈{4, 8, 16, 32}.
  • Qualitative results: Qualitative predictions from ThinkJEPA are smoother, more diverse, and better aligned with the reference hand than V-JEPA baseline trajectories.The baseline often produces temporally collapsed predictions with overlapping points across timesteps and joints.

5 Conclusion

ThinkJEPA combines long-horizon VLM semantic guidance with dense JEPA latent dynamics through dual-temporal perception and multi-depth feature injection. Across four benchmarks, it consistently outperforms VLM-only, JEPA-only, and task-specific trajectory baselines while preserving the JEPA forecasting interface.

  • 5 Conclusion: ThinkJEPA combines uniformly sampled VLM observations with dense JEPA observations to provide semantic guidance alongside fine-grained latent dynamics.The VLM branch supplies long-horizon guidance, while the JEPA branch preserves dense latent forecasting.
  • 5 Conclusion: Pyramid-extracted, multi-depth VLM representations are injected into the JEPA predictor through layer-wise modulation.This enriches future latent prediction with knowledge-aware guidance without replacing the latent forecasting interface.
  • 5 Conclusion: ThinkJEPA consistently outperforms VLM-only, JEPA-only, and task-specific trajectory baselines across EgoDex, EgoExo4D, BAIR Robot Pushing, and Physion.The benchmarks cover egocentric trajectory prediction, recursive rollout, action-conditioned robotic prediction, and action-free physical-scene forecasting.
  • 5 Conclusion: EgoDex evaluates egocentric dexterous manipulation using videos paired with 3D hand and finger pose annotations.The same cached train/test split is used for VLM-only, V-JEPA-only, ThinkJEPA, and ablation methods.
  • 5 Conclusion: EgoExo4D tests transfer to broader human-motion forecasting using synchronized egocentric and exocentric videos with 3D pose and gaze annotations.The dataset contains annotations for 3D body pose, 3D hand pose, and gaze.
  • 5 Conclusion: BAIR and Physion evaluations use controlled training, validation, and test splits to assess robotic rollout and action-free physical-scene forecasting.BAIR uses 2,000 training and 500 validation videos with the official 256-video test split; Physion uses 2,000 training, 400 validation, and 1,200 test videos.

A.2.1 Trajectory metrics.

The evaluation combines trajectory, latent-forecasting, and recursive-rollout metrics with multiple baselines and ablations. It measures both downstream 3D prediction quality and representation-level forecasting across human manipulation, robotic interaction, and physical-scene settings.

  • A.2.1 Trajectory metrics.: ADE, FDE, and threshold Accuracy measure mean trajectory error, final-frame error, and the fraction of joint predictions within 0.05 m.These metrics evaluate predicted versus ground-truth 3D trajectories across future frames and joints.
  • A.2.1 Trajectory metrics.: FD, SL1, and CD measure latent forecasting quality using feature ℓ2, SmoothL1, and cosine distances independently of the downstream trajectory head.The metrics compare predicted and target V-JEPA latents.
  • A.2.1 Trajectory metrics.: Recursive rollout reports ADE@H, FDE@H, and latent-block distances at horizons H ∈ {4, 8, 16, 32}.BAIR rollout additionally reports FD/L2, SL1, and CD over autoregressively predicted latent blocks.
  • A.2.1 Trajectory metrics.: Physion uses object-contact Accuracy and ROC-AUC, which are distinct from the 0.05 m trajectory Accuracy used for EgoDex and EgoExo4D.This separates physical-scene contact classification from joint-position forecasting.
  • A.2.1 Trajectory metrics.: ThinkJEPA injects encoding-token and autoregressive-token VLM guidance into the V-JEPA predictor through layer-wise modulation.The guidance is derived from multi-depth VLM representations rather than replacing dense V-JEPA tokens.
  • A.2.1 Trajectory metrics.: The baselines include VLM-only, V-JEPA-only, persistence, DINO-WM, and task-specific Behavior Cloning, DDPM, and Flow Matching predictors.These comparisons test semantic-only, dense-latent, no-dynamics, alternative-representation, and task-specific forecasting approaches.
  • A.2.1 Trajectory metrics.: Additional ablations vary VLM token sources, layer selections, conditioning mechanisms, and prompt-only inference.Encoder denotes encoding tokens, while AR denotes autoregressive tokens.
  • A.2.1 Trajectory metrics.: BAIR uses four 5-frame latent blocks, observes the first block, and performs three-step autoregressive rollout in frozen V-JEPA latent space.Action-conditioned variants also receive robot actions, whereas visual-only variants use observed latent blocks alone.

B.7.1 Evaluation.

The evaluation tests object-contact prediction, VLM token sources, layer selection, recursive rollout, and prompt-conditioned features under matched experimental settings. It also specifies how VLM conditioning is supplied without replacing the visual backbone.

  • B.7.1 Evaluation.: Physion object-contact prediction reports Accuracy, Balanced Accuracy, and ROC-AUC from observed and predicted future latent tokens.These metrics differ from the 0.05 m trajectory Accuracy used for EgoDex and EgoExo4D.
  • B.7.1 Evaluation.: Token-source ablations compare encoding tokens, autoregressive tokens, their combinations with the V-JEPA predictor, and variants without the dual-temporal pathway or thinker module.Encoder tokens come from input-side representations, while AR tokens come from generation-side hidden states.
  • B.7.1 Evaluation.: Layer-selection ablations compare last-layer, mid-layer, and all-layer pyramid guidance under the same training and evaluation protocol.The comparison isolates the role of hierarchical pyramid extraction.
  • B.7.1 Evaluation.: Recursive rollout evaluates horizons H ∈ {4, 8, 16, 32} using ADE@H, FDE@H, Accuracy@H, and latent-distance diagnostics.Each rollout step uses a short-window predictor with Tp=4 and Tf=4.
  • B.7.1 Evaluation.: Prompt-conditioned VLM features use cached visual features as the primary trajectory input and language-modulated VLM features as external conditioning.The setup changes only the conditioning path relative to the main backbone and downstream head.
  • B.7.1 Evaluation.: The VLM branch provides encoder-side and autoregressive generation-side features that are injected into, rather than replacing, the visual predictor.This preserves the visual backbone as the main predictive substrate.

C.3 Analysis.

Prompt-conditioned VLM features trade slightly weaker trajectory metrics for slightly better latent forecasting than full ThinkJEPA. The temporal-stride study directly compares sampling granularity while holding the predictor and conditioning setup fixed.

  • C.3 Analysis.: 0.069/0.062/0.495 versus 0.061/0.056/0.596 on ADE/FDE/Acc shows prompt-conditioned features trail full ThinkJEPA on trajectory prediction.The prompt-conditioned variant is compared with the full ThinkJEPA model using the same metric order.
  • C.3 Analysis.: 74.007/1.248/0.315 versus 74.747/1.263/0.324 on FD/SL1/CD shows prompt-conditioned features slightly improve latent forecasting.The result supports prompt-conditioned VLM features as an effective representation-guidance choice.
  • C.3 Analysis.: The temporal-stride ablation compares two strides while keeping predictor architecture, training budget, and conditioning mechanism fixed.This isolates the effect of temporal sampling granularity.
  • C.3 Analysis.: Stride 1 retains all 64 uniformly sampled EgoDex points, whereas stride 2 subsamples the sequence before the past/future split.The setup contrasts denser temporal input with a coarser representation.

D.3 Analysis.

The analyses compare temporal sampling and VLM-conditioning choices under controlled settings. Dense sampling and the full ThinkJEPA design provide the strongest reported downstream trajectory performance.

  • Stride 1 outperforms stride 2 on all reported metrics for trajectory prediction and latent forecasting.
  • The full ThinkJEPA model achieves the best ADE/FDE/Acc overall on the evaluated split.
  • FiLM, cross-attention, and AdaLN are competitive conditioning operators under the same backbone, data split, and training budget.FiLM provides the strongest latent forecasting quality among the three variants.
  • Compared with the controlled conditioning variants, full ThinkJEPA achieves substantially better trajectory prediction on ADE/FDE/Acc.
  • The ablation compares removing the VLM branch with retaining it while removing deepstack/thinking-token contributions.

F.3 Analysis.

The analyses show that the complete VLM guidance pathway is most effective for downstream trajectory prediction, while FiLM is selected to improve latent feature prediction. The fair VLM-only comparison uses matched supervision rather than zero-shot prompting.

  • Both direct-visual and deepstack-token-removal ablations are weaker than full ThinkJEPA in downstream trajectory performance.ThinkJEPA achieves the best ADE/FDE/Acc overall.
  • FiLM is the default because it directly modulates predictor latent features while preserving the JEPA-style latent forecasting interface.
  • The pure VLM baseline directly prompts Qwen3-VL (Thinking) to output future hand trajectories without a trained trajectory head.
  • The main VLM-only comparison uses a trained downstream head and matched supervision, unlike the pure prompt-only baseline.

G.3 Analysis.

Direct prompting of a general-purpose VLM is inadequate for fine-grained metric-space trajectory forecasting. ThinkJEPA substantially outperforms the prompt-only baseline, which also has poor structured-output parsing success.

  • ADE/FDE of 10.855/10.927 for prompt-only Qwen3-VL versus 0.061/0.056 for ThinkJEPA shows a large performance gap.
  • Poor parsing success indicates that structured trajectory generation is unstable under pure prompting.
  • The supplementary configuration uses cached Qwen3-VL (Thinking) features in a VLM-injected V-JEPA predictor with latent dimension Dp = 384.

I Limitations and Future Directions

The paper identifies scope boundaries around planning, generalization, scaling, and deployment safety. Future work targets closed-loop agentic use, broader scenarios, longer videos, and more efficient temporal guidance.

  • Current experiments focus on predictive latent modeling rather than closed-loop planning or policy deployment.
  • Generalization across diverse scenarios remains challenging for current latent world models, motivating stronger VLM thinkers with broader world knowledge.
  • Scaling to longer videos and more diverse embodied environments may require more efficient VLM feature caching and adaptive temporal guidance.
  • Deployment in robotics or downstream decision-making requires validation under diverse real-world conditions because incorrect forecasts, bias, or distribution shift may create unsafe or unfair decisions.
  • Egocentric and embodied video data may raise privacy concerns without appropriate consent and safeguards.
Loading 2603.22281v2…