PaperlayerView exact revision on arXiv

Source-linked AI summary

What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?

Basile Terver, Tsung-Yen Yang, Jean Ponce, Adrien Bardes, Yann LeCun

arXiv:2512.24497v4cs.AIcs.LGcs.ROstat.ML

TL;DR

The paper asks how to efficiently learn dynamics in pretrained visual-encoder embedding spaces for physical-task planning. It studies JEPA-WM training and planning choices, then combines its findings into a model reported to outperform DINO-WM and V-JEPA-2-AC.

  • Problem

    The paper identifies limited systematic study of how to efficiently learn embedding-space dynamics with pretrained visual encoders for manipulation and navigation planning.

    Source: §1 · p. 2

  • Method

    The paper systematically studies JEPA-WM training and planning design choices, then combines the findings into optimized models for its robotic environments.

    Source: §1 · p. 2

  • Results

    The proposed JEPA-WM outperforms DINO-WM and V-JEPA-2-AC, and the study finds that encoder type, proprioception, and rollout training matter for planning performance.

    Source: §1 · p. 2, §6 · p. 12

  • Takeaways & Limitations

    For the studied tasks, fine object segmentation and enough context to infer velocity are beneficial, while excessively long context can harm performance.

    Source: §5.2 · p. 10, §6 · p. 12

Abstract

from arXiv · show

A long-standing challenge in AI is to develop agents capable of solving a wide range of physical tasks and generalizing to new, unseen tasks and environments. A popular recent approach involves training a world model from state-action trajectories and subsequently use it with a planning algorithm to solve new tasks. Planning is commonly performed in the input space, but a recent family of methods has introduced planning algorithms that optimize in the learned representation space of the world model, with the promise that abstracting irrelevant details yields more efficient planning. In this work, we characterize models from this family as JEPA-WMs and investigate the technical choices that make algorithms from this class work. We propose a comprehensive study of several key components with the objective of finding the optimal approach within the family. We conducted experiments using both simulated environments and real-world robotic data, and studied how the model architecture, the training objective, and the planning algorithm affect planning success. We combine our findings to propose a model that outperforms two established baselines, DINO-WM and V-JEPA-2-AC, in both navigation and manipulation tasks. Code, data and checkpoints are available at https://github.com/facebookresearch/jepa-wms.

1 Introduction

The paper studies how to efficiently learn dynamics in pretrained visual-encoder embedding spaces for manipulation and navigation planning. It formalizes JEPA-WMs, surveys prior results, and evaluates design choices to develop a model that outperforms established baselines.

1 Introduction

  • JEPA-WMs are action-conditioned video models trained with embedding-space prediction alone, without reconstruction, reward prediction, or value/policy heads.The paper presents them as a unified implementation recipe, not a novel algorithm.Source: §1 · p. 2
  • Prior work reports JEPA-WM results in navigation, reward-free goal-conditioned planning, dense prediction, and object manipulation.The cited examples include PLDM, DINO-WM, DINO-World, and V-JEPA-2-AC.Source: §1 · p. 2
  • The paper identifies a gap in thorough evidence on efficiently learning embedding-space dynamics for manipulation and navigation planning with pretrained visual encoders.Its focus is dynamics learning rather than representation learning.Source: §1 · p. 2
  • The study varies key training and planning components, then uses the findings to propose a JEPA-WM that outperforms DINO-WM and V-JEPA-2-AC.Components include rollout loss, predictor architecture, context length, proprioception, encoder type, model size, data augmentation, and planning optimizer.Source: §1 · p. 2

2 Related work

Related work places JEPA-WMs among broader world-model approaches to learning and planning. It also distinguishes explicit latent dynamics from alternative paradigms and identifies complementary work on representations, goal-conditioned learning, and robotics.

World modeling and planning

  • World models support exploration, sample efficiency, transfer, subgoal discovery, and planning, using either generative transitions or deterministic latent dynamics.Latent deterministic models trade stochastic expressiveness for computational efficiency and latent abstraction (Terver et al., 2026; Zhou et al., 2024a; Bar et al., 2025).Source: §2 · pp. 2–3

Explicit and implicit world models

  • The study focuses on explicit autoregressive latent dynamics, while implicit TD-JEPA folds temporal abstraction into successor-feature representations.The paper compares these paradigms in Section B (Bagatella et al., 2026).Source: §2 · p. 3

Goal-conditioned RL

  • Goal-conditioned RL uses unlabeled data and has been combined with planning, hierarchical subgoals, offline RL, and geometrically grounded distances.OGBench provides a systematic evaluation of offline GCRL across locomotion and manipulation (Park et al., 2025).Source: §2 · p. 3

Robotics

  • The robotics context includes classical MPC, visual servoing, and manipulation policies such as VLA models and flow-matching action generation.The cited examples include RT-X, RT-1, RT-2, and the π series (Vuong et al., 2023; et al., 2023; Zitkovich et al., 2023; Black et al., 2024; Intelligence et al., 2025b;a).Source: §2 · pp. 3–4

3 Background

The background formalizes a common JEPA-WM training and planning setup rather than introducing a new method. Encoders embed observations and actions, a predictor learns future embeddings, and planning scores predicted trajectories against a goal.

3 Background

  • A JEPA-WM uses a frozen visual encoder, optional proprioceptive encoder, action encoder, and predictor to learn state transitions in embedding space.The study uses ViT architectures and MSE between predicted and target embeddings; its predictor uses frame-causal attention.Source: §3 · p. 4
  • Training predicts future embeddings from a past observation-and-action window, with causal attention training across context lengths and multiple actions.The maximum training context is W = 3; the loss is computed pairwise for visual and proprioceptive predictions and targets.Source: §3 · p. 4
  • Planning unrolls the predictor over candidate action sequences and scores the resulting trajectory against the goal embedding using visual and, optionally, proprioceptive distance.The objective can weight proprioceptive distance by α, and the predictor unrolling function supplies the trajectory representation.Source: §3 · pp. 4–5

4 Studied design choices

The study independently varies planner, training, architecture, and scaling choices from a DINO-WM-based configuration. Its experiments compare alternatives intended to identify suitable JEPA-WM designs across simulated and robotic tasks.

Planner

  • The planner comparison evaluates CEM, Nevergrad, Adam, and gradient descent with L1 or L2 embedding distance; CEM with L2 performs best overall.The paper also fixes common planning hyperparameters after exploring their effect on success.Source: §4 · pp. 5–6, Figure 3 · p. 7

Multistep rollout training

  • The study tests multistep rollout losses alongside teacher forcing, including losses through six prediction steps.Models trained beyond one-step loss use W = 7 context; truncated backpropagation discards accumulated gradients and backpropagates only the final prediction error.Source: §4 · p. 6

Proprioception and training context size

  • Proprioceptive input is compared with visual-only training, while context lengths from W = 1 to W = 14 test whether longer training context supports longer rollouts.Source: §4 · p. 6

Encoder type

  • Encoder experiments compare local DINOv2 and DINOv3 features with V-JEPA and V-JEPA-2 video encoders, while matching visual-token counts across encoders.The selected frame strategy duplicates adjacent frames and encodes each pair independently as a two-frame video.Source: §4 · p. 6

Predictor architecture

  • Predictor experiments compare feature and sequence conditioning, mixed conditioning, AdaLN, and AdaLN-zero, with different positional embeddings.AdaLN allows action information to influence all predictor layers rather than only the input.Source: §4 · pp. 6–7

Model and data scaling

  • Scaling experiments vary encoder size, predictor depth, and training-data fraction, comparing the proposed model with DINO-WM and V-JEPA-2-AC.The base configuration is DINO-WM without proprioception, using a ViT-S encoder and depth-6 predictor; data fractions are 2%, 10%, 50%, and 100%.Source: §4 · pp. 5, 7

5 Experiments

The experiments evaluate JEPA-WMs across simulated and real-robotic settings, studying how design choices affect planning. The resulting recipe outperforms DINO-WM and V-JEPA-2-AC in most environments.

5.1 Evaluation Setup.

  • Experiments cover Metaworld, Push-T, Wall, PointMaze, DROID, and zero-shot Robocasa; DROID models are also evaluated on 16 real Franka-arm videos.DROID-trained models are not fine-tuned on Robocasa trajectories; DROID evaluation scores planned actions against dataset actions.Source: §5.1 · p. 7

5.2 Results

  • CEM with L2 cost is best or competitive across environments, while optimizer performance depends on task structure.Adam L2 is best on Metaworld; gradient-based methods perform poorly on non-greedy navigation and DROID, while NG matches CEM on DROID and Robocasa without hyperparameter tuning.Source: §5.2 · pp. 8–9
  • Two-step rollout training improves planning over pure teacher forcing, but further rollout training reduces performance in simulated environments.The best rollout horizon is K=6 on DROID; training and planning context must satisfy W_p ≤ W, and two context frames help infer velocity.Source: §5.2 · pp. 9–10
  • Proprioception consistently improves performance, while DINO encoders outperform V-JEPA encoders in the tested visual-encoder comparisons.Proprioception supplies precise metric information near goals; it is omitted for Robocasa transfer because the proprioceptive spaces are not aligned.Source: §5.2 · pp. 9–10
  • AdaLN with RoPE has the best average predictor-architecture performance, but its advantage is slight and task-dependent.Increasing model size or predictor depth does not improve simulated-task performance, whereas both correlate positively with DROID planning performance.Source: §5.2 · pp. 10–11
  • Planning performance increases with more training data across the considered datasets, but scaling is less clear on Robocasa when training on DROID.The authors attribute the Robocasa pattern to the domain gap between DROID and Robocasa.Source: §5.2 · p. 11

5.3 Our proposed optimum in the class of JEPA-WMs

  • The proposed JEPA-WM outperforms DINO-WM and V-JEPA-2-AC in most environments, with a qualitative comparison showing better prediction of action effects.The recipe uses smaller models with proprioception for simulated environments and larger models without proprioception for DROID and Robocasa.Source: §5.3 · p. 12, Table 2 · p. 12
  • The proposed recipe uses CEM L2 throughout, DINOv2 for simulated environments, and DINOv3 for photorealistic DROID and Robocasa.Simulated tasks use ViT-S models with depth 6, AdaLN, RoPE, proprioception, two-step rollout loss, and W=3; DROID and Robocasa use ViT-L models with depth 12 and no proprioception.Source: §5.3 · p. 12

6 Conclusion

The conclusion summarizes design-choice findings, reports stronger JEPA-WM performance than concurrent baselines, and notes a limitation of deterministic prediction. Extended related work contrasts explicit and implicit models and situates JEPA-WMs among alternative planning approaches.

6 Conclusion

  • The proposed models outperform DINO-WM and V-JEPA-2-AC, while the study finds that scaling benefits depend on task complexity and planner performance varies by task.Larger models improve consistently on DROID and Robocasa but not simulated environments; CEM L2 performs best overall, while gradient planners work on smooth landscapes and fail on navigation or contact-rich tasks.Source: §6 · p. 12
  • Deterministic MSE prediction learns conditional means of potentially multimodal futures; aleatoric settings may require latent-variable or latent-diffusion extensions.The benchmarks mitigate this limitation through deterministic dynamics, latent abstraction of irrelevant variability, and closed-loop MPC.Source: §6 · p. 12

Broader Impact Statement

  • The broader-impact statement anticipates no particular risk, while acknowledging possible downstream misuse and the environmental cost of large-model training.It advocates efficient training and sharing pretrained models to reduce redundant computation.Source: Broader Impact Statement · p. 13

A Notation

  • Table 3 provides a summary of notation used throughout the paper.Source: Table 3 · p. 23

B Extended Related Work

  • Alternative latent planners differ in dynamics, optimization, and training assumptions; JEPA-WMs offer reward-free visual training and flexible test-time goals as a complementary setting.Gradient-based optimization can be effective on smooth cost landscapes but prone to local minima in navigation; direct diffusion-planner comparison remains challenging across differing settings.Source: B Extended Related Work · p. 24
  • Explicit models defer computation to test-time planning, whereas implicit TD-JEPA front-loads training and can evaluate new rewards in one forward pass.Explicit models also generate counterfactual embedding trajectories and permit cost functions independent of the learned dynamics; a direct empirical comparison is left for future work.Source: B Extended Related Work · pp. 24–25
  • Reward- and reconstruction-based MBRL methods train or plan using predicted rewards, while JEPA-WMs avoid reward engineering but lack straightforward cumulative-reward optimization without a separate reward model.The discussion covers PlaNet, Dreamer, MuZero, and TD-MPC2 as examples of reward-prediction, reconstruction, or reward-maximization approaches.Source: B Extended Related Work · p. 25

C Training details

The training details compare rollout, action-conditioning, augmentation, and encoder choices. They favor planning-aligned mixed-context training, report pitfalls in video encoding and cropping, and document a corrected V-JEPA-2-AC reproduction.

Action conditioning of the predictor

  • AdaLN injects actions through per-layer scale and shift modulation; its consistent conditioning and compute efficiency may help explain its superior empirical performance.Unlike feature or sequence conditioning, AdaLN avoids increasing feature or sequence dimensions; the expected conditioning advantage is an interpretation rather than a stated causal result.Source: C Training details · p. 26

V-JEPA-2-AC reproduction

  • The V-JEPA-2-AC reproduction uncovered a miscomputed 2-step rollout loss; after fixing it, the public checkpoint scores much lower on the DROID protocol than retrained models.Despite worse planning performance, the public predictor produces comparable image decodings after unrolling and appears to pass the counterfactual test.Source: C Training details · pp. 27–28

Data augmentation ablations

  • Random-resize-crop augmentation hurts performance when the crop sometimes leaves the agent partly out of view.The tested crop uses aspect ratios from 0.75 to 1.33 and scales from 0.3 to 1.0; horizontal flipping is omitted to preserve action-state correspondence.Source: C Training details · p. 28

Ablations on models trained with video encoders

  • For video encoders, MSE beats L1 and smooth L1, while duplicating frames and encoding each pair independently performs best among the tested sequence-encoding approaches.Treating sampled frames as consecutive or duplicating them within a video encoder performs worse than frame-wise pair encoding.Source: C Training details · pp. 28–29

Encoder comparison details

  • DINOv2/v3 and V-JEPA/2 use frozen encoders with different pretraining, preprocessing, and patch sizes; DINOv2/v3's stronger performance may reflect finer object segmentation.Video encoders require duplicating each frame; the authors use raw patch tokens without aggregation or projections.Source: C Training details · p. 29, Table 7 · p. 29

Multistep rollout predictor training

  • Equal-order rollout performs worse because its inputs are less aligned with planning-time contexts that mix ground-truth embeddings and predictions.Although it computes more parallel loss gradients than Last-gradient-only, it does not train on the same mixed context used during planning.Source: C Training details · p. 30
  • The 2-step Last-gradient-only strategy with random initial context yields the best success rate, and TBPTT benefits success rate and training time across strategies.The predictor is better aligned with planning when trained on a mix of encoder outputs and its own predictions; the All-gradients variant does not outperform Last-gradient-only.Source: C Training details · p. 30

D Error propagation in autoregressive latent prediction

The appendix bounds autoregressive prediction error and examines how rollout horizon, context length, and training rollout length affect its growth. It also describes when Lipschitz regularization can reduce that growth.

D Error propagation in autoregressive latent prediction

  • For a one-step context, errors grow exponentially when Λ > 1, linearly when Λ = 1, and saturate at δ/(1 − Λ) when Λ < 1.The bounds assume the initial embedding is ground truth and a per-step prediction error bounded by δ.Source: D Error propagation in autoregressive… · p. 31
  • Under σmin(JPθ) ≥ ℓ > 1 and an aligned initial error above δ/(ℓ − 1), the lower bound is Ω(ℓ^H), while the upper bound is O(Λ^H).The rates coincide when the Jacobian is conformal; otherwise ℓ and Λ may differ.Source: D Error propagation in autoregressive… · pp. 32–33
  • For sliding-window predictors, errors are O(ρ^H) when the companion matrix has spectral radius ρ > 1, as is typical without explicit contractivity regularization (Miyato et al., 2018; Pascanu et al., 2013).The asymptotic rate is governed by the spectral radius of the companion matrix.Source: D Error propagation in autoregressive… · p. 33
  • Larger training rollout length K trades higher ground-truth one-step error δK against a potentially lower effective Lipschitz constant ΛK.The analysis suggests small K for clean simulated dynamics but larger K for noisy real-world data, consistent with the reported experimental shift.Source: D Error propagation in autoregressive… · p. 34
  • Spectral normalization can constrain Λ to approximately 1, changing the simplified predictor’s error growth from exponential to linear.For sliding-window predictors, polynomial growth is also possible when ρ = 1 and the companion matrix has a Jordan block larger than one; the text describes this case as degenerate.Source: D Error propagation in autoregressive… · p. 34

E Planning environments and datasets

The appendix describes the datasets and evaluation setups used across simulated and robotic environments. It also notes how context length, task selection, and transfer conditions constrain those evaluations.

E Planning environments and datasets

  • Training normalizes actions and optional proprioception per dataset, but actions are not normalized for DROID; the study describes its data as scarce except for Push-T.For DROID-to-Robocasa transfer, actions are repeated five times and rescaled to the simulator’s action ranges.Source: E Planning environments and datasets · pp. 35–36
  • Increasing context length can discard short DROID videos, while performance stays essentially constant beyond W = 3 on Push-T and Maze.At W = 14, only 86% of DROID data is retained; Wall’s 50-frame trajectories limit its context length to W = 9.Source: E Planning environments and datasets · p. 35
  • DROID evaluation uses 16 videos focused on object interaction and arm navigation, with Action Score reported as a rescaling of negative Action Error.For training, the batch sampler can draw from either the left or right camera, which gave slightly lower validation loss than using only one camera.Source: E Planning environments and datasets · pp. 35–36
  • Robocasa evaluation uses custom, shorter-horizon pick-and-place tasks and 16 teleoperated trajectories, with camera viewpoint and gripper matched to DROID.The evaluation defines six subtasks from reach, pick, and place segments; models have low success on Pick because small end-effector position errors matter.Source: E Planning environments and datasets · p. 36
  • Metaworld training uses 12,600 episodes across 42 tasks, and evaluation plans for 100 steps toward a goal generated by an expert episode under a randomized seed.The dataset retains the first 100 episodes for each of 126 task-and-seed buffers to limit expert-trajectory proportion and promote diversity.Source: E Planning environments and datasets · pp. 36–37
  • The study evaluates Push-T, PointMaze, and Wall using environment-specific initial and goal states; Push-T evaluation selects states whose goal can be reached within the planning horizon.The datasets comprise noisy expert replays for Push-T, 2,000 random trajectories for PointMaze, and 1,920 random 50-step trajectories for Wall.Source: E Planning environments and datasets · p. 38

F Planning Optimization

The section defines the planning objective and compares optimization procedures for choosing action sequences. It describes sampling-based and gradient-based planners, as well as replanning in Metaworld.

F Planning Optimization

  • CEM fits a time-dependent diagonal-covariance Gaussian to sampled action trajectories, evaluates their predicted costs, and updates the distribution using top trajectories.The planner’s initial mean is zero and its covariance is the identity.Source: F Planning Optimization · p. 39
  • In Metaworld, MPC replans after executing a plan; a reported 100-step episode shows NG stuck against a wall and CEM imprecise near the goal.The displayed figure compares average objective and standard deviation for top imagined trajectories with the best trajectory’s planning loss.Source: Figure 9 · p. 39, F Planning Optimization · p. 39
  • NGOpt selects a particular diagonal CMA-ES variant in these setups; other diagonal-CMA parameterizations can reduce success by 20% on Wall, Maze, and Push-T.The elitist variant is one reported example of a parameterization associated with the drop.Source: F Planning Optimization · p. 40
  • Gradient Descent optimizes actions through the differentiable world model, while Adam replaces vanilla gradient descent with moment-based updates.Both planners add Gaussian noise after gradient steps; their listed default iteration count is 500.Source: F Planning Optimization · p. 40

G.1 Additional results

The additional results compare JEPA-WM design choices and planners across navigation and manipulation tasks. They report task-dependent conditioning and optimizer preferences, as well as distinct planning failures and manipulation errors.

G.1 Additional results

  • Sequence conditioning outperforms feature conditioning on DROID, Robocasa, and Metaworld, while feature conditioning wins on Wall and results are comparable on Maze and Push-T.The Wall trend replicates across different resolutions and model sizes; the comparison uses matched action ratios.Source: §G.1 · p. 41
  • On Wall, rollout losses match between conditioning methods, but sequence-conditioned planning sometimes selects very high-magnitude actions while feature-conditioned planning selects moderate actions.Sequence conditioning exposes the action token to self-attention from the first block; feature conditioning concatenates action embeddings to visual tokens.Source: §G.1 · p. 41
  • The optimal JEPA-WM consistently outperforms DINO-WM and V-JEPA-2-AC on most tasks and planner configurations, with final-checkpoint variance results also supporting reliable outperformance.Table 11 compares models across planner configurations; Table 12 reports final-checkpoint variance across three seeds.Source: §G.1 · pp. 41–42, Table 11 · p. 42, Table 12 · p. 43
  • On Wall, GD achieves zero performance through either entering the wall without passing the door or reaching a local cost minimum at image borders.Figure 10 illustrates both failure cases.Source: §G.1 · p. 42, Figure 10 · p. 44
  • On Robocasa, the model succeeds at Place but has lower success at Pick, misestimates arm position, and predicts a left-shifted state on Reach.The leftward shift is less clear on DROID.Source: §G.1 · p. 42
  • On Metaworld, pose-control tasks are solved and long-term bin-picking unrolling is captured, but imagined grasps can diverge from simulator outcomes.The reported gap motivates separate optimization for the gripper action dimension.Source: §G.1 · pp. 42–43
  • On Push-T, CEM performs better than NG; because the task requires very precise actions, it is essential to step actions only after the planner converges.The passage also describes NG as more explorative and says the larger model converges faster and brings a higher maximal success rate on Push-T.Source: §G.1 · p. 43
  • The ViT-L model’s relative embedding-space distance to the goal is approximately ten times smaller than the ViT-S model’s on Metaworld reach-wall.The authors relate this difference to the larger model’s higher capacity to embed information.Source: §G.1 · p. 43

G.2 Evaluation metrics

The evaluation tracks success alongside several validation metrics because success can be noisy and sparse. The additional metrics measure embedding prediction, decoded state, visual reconstruction, and planned-action error.

G.2 Evaluation metrics

  • At each epoch, evaluations use 96 episodes, except DROID with 64 and Robocasa with 32; reported success is averaged over episodes and across recent epochs.The recent-epoch average uses 10 epochs for most datasets and 100 for DROID.Source: §G.2 · pp. 44–45
  • Because planning success can be noisy, seed-dependent, and sparse early in training, the study tracks other metrics and their correlation with success.The reported success rates average across three seeds and evaluation episodes.Source: §G.2 · p. 45
  • Embedding error measures L1 and L2 distances between predicted and target embeddings during validation rollouts, with step-one L2 equal to the teacher-forcing loss.The predictor is unrolled for multiple steps on validation batches every 300 training iterations.Source: §G.2 · p. 45
  • State-decoding error compares decoded predictions from unrolled features with future ground-truth states, while visual decoding assesses reconstructed frames qualitatively and with LPIPS.The state decoder is trained on the frozen encoder before predictor training; visual frames are decoded independently as a probing tool.Source: §G.2 · p. 46
  • Action error compares planner outputs with dataset ground-truth actions; on DROID, the study tracks the first three position-control dimensions because gripper and orientation errors dominate.Unclipped planner actions can increase total error because actions are not normalized.Source: §G.2 · p. 46

G.3 Is there a proxy for success rate?

The study examines whether validation metrics can serve as proxies for planning success. It finds the planning objective most correlated with success overall, while noting that prediction metrics and the heuristic planning evaluation are not fully aligned.

G.3 Is there a proxy for success rate?

  • The planning objective, visual embedding loss, is the metric most correlated with success in both environments; only in Metaworld do unroll metrics beyond step one correlate better than step-one metrics.The study averages Spearman correlations across models to avoid Simpson’s paradox.Source: §G.3 · p. 46
  • Validation prediction losses are imperfect proxies because prediction validation is not fully aligned with goal-conditioned planning, which minimizes final imagined-state embedding distance to the goal.The planning evaluation is described as a heuristic.Source: §G.3 · p. 47
  • On DROID, out-of-distribution planner actions can severely harm plan accuracy and obscure predictor improvement; better world models can also leave gradient planners stuck in local cost minima.The latter issue is reported for gradient-based planners.Source: §G.3 · p. 47

G.4 Success over epochs

The supplementary figures document planning and rollout examples alongside success-rate trajectories across evaluation setups. They include visualized discrepancies between imagined plans and simulator or dataset outcomes.

G.4 Success over epochs

  • On collected Franka arm trajectories, the model sometimes fails to capture object interaction, its most frequent failure case.Figure 14 pairs decoded unrollings with the dataset ground-truth trajectories.Source: Figure 14 · p. 50
  • DINO-WM open-loop rollout examples show 18 model actions corresponding to 90 elementary simulator actions.The figure places ground-truth simulator stepping below predictor-rollout decoding.Source: Figure 15 · p. 51
  • In a Metaworld bin-picking example, the planned prediction indicates a picked object, but stepping half the plan in the simulator leaves the object unpicked.The arm reaches the target location without the object.Source: Figure 16 · p. 51
  • Planning-cost convergence examples compare CEM and NG across 2D environments and DROID, while Metaworld reach-wall figures compare planned trajectories from ViT-S and ViT-L models.The figures show the best trajectory’s planning cost over optimization steps and embedding-space distances to the goal, respectively.Source: Figure 17 · p. 52, Figure 18 · p. 53
  • The end-of-training model plans to approach the gripper from the box, yet its total action error exceeds the first-epoch model’s because gripper and orientation dimensions contribute most error.The comparison covers seven action dimensions.Source: Figure 19 · p. 53
  • Success-rate evolution is plotted across epochs for multiple evaluation setups and tasks, including comparisons of model size, proprioception, and multistep rollout.Figure 21’s rows correspond to Metaworld reach, Metaworld reach-wall, Wall, Push-T, and Point Maze; each epoch averages 96 episodes.Source: §G.4 · p. 49, Figure 21 · p. 55

Related papers

Loading 2512.24497v4…