PaperlayerView exact revision on arXiv

Source-linked AI summary

The Planning Limits of Latent World Models

Ali Alrasheed, Basim Azam, Naveed Akhtar

arXiv:2609.39235v1cs.ROcs.CV

TL;DR

Existing evaluations show what world models can do, but leave unclear when their predictions remain useful for planning. This paper tests action-conditioned predictors with frozen visual encoders and finds that reliable planning depends on target distance relative to imagined rollout length.

  • Problem

    Prior evaluations provide limited evidence about when latent world-model predictions reliably guide planning and where they fail.

    Source: §1 · p. 1

  • Method

    The paper evaluates action-conditioned predictors built on five frozen visual encoders across Meta-World manipulation tasks and BridgeData V2 robot data, varying planning distance and model factors.

    Source: §1 · pp. 1–2

  • Results

    With five-step rollouts, reliable action ranking extends only five to ten steps ahead, while tasks take 16 to 53 steps; an 81-fold larger predictor and longer-rollout training do not extend the range.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Planning with latent world models benefits from keeping targets within imagined trajectories or using closer subgoals, and the model can improve a VLA policy by selecting among its proposed actions.

    Source: §8 · p. 9

    Distance-based scoring assumes good actions move closer to the target at every step, an assumption that can fail during temporary detours.

    Source: §5.5 · p. 6

Abstract

from arXiv · show

World models offer a promising way to help robots understand how the physical world evolves and plan complex behaviours through imagination. Yet existing studies mainly demonstrate what these models can accomplish, leaving unclear when their predictions remain useful for planning and where they fail. We study this question using action-conditioned predictors built on five frozen self-supervised visual backbones: V-JEPA 2, V-JEPA 2.1, VideoMAEv2, VideoPrism, and DINOv2. We use frozen backbones to test representations intended to transfer across environments. We evaluate these models on diverse Meta-World manipulation tasks and real-robot interactions from BridgeData V2. We find that a world model guides action selection reliably only when the goal lies within, or slightly beyond, the trajectory it imagines during planning. With five-step rollouts, the length the predictor was trained on, the world model ranks actions reliably only for targets five to ten control steps ahead, whereas task goals lie 16 to 53 steps away. Neither an 81-fold larger predictor nor longer-rollout training extends this range; the encoder affects both range and closed-loop success, with V-JEPA 2.1 performing most consistently. More fundamentally, the limit persists under perfect prediction: using the real simulator, success falls from 92% to 41% as the target moves from five to twenty steps ahead of a five-step rollout. Planning therefore requires either longer imagined trajectories or closer subgoals. For distant goals, pure imagination succeeds in 23% of episodes, planning with feedback (MPC) raises success to 30%, imagining as far as the goal to 47%, and nearby expert subgoals to 76%. Used within its plannable range, a world model can also improve a vision-language-action (VLA) policy: choosing among eight actions the VLA proposes raises its success from 65% to 77% across 16 different tasks.

1 INTRODUCTION

Latent world models predict action-conditioned futures in learned representations, but prior evaluations leave unclear when those predictions reliably guide planning. This paper tests that limit across frozen visual encoders, simulated manipulation tasks, and real-robot data.

1 INTRODUCTION

  • Existing evaluations show what world models can accomplish but say less about when and why they fail.Prediction accuracy alone does not always correspond to control performance.Source: §1 · p. 1
  • The paper asks when latent world models provide reliable planning signals and what factors limit or extend their useful distance.It evaluates action-conditioned predictors built on five frozen visual encoders across Meta-World and BridgeData V2.Source: §1 · pp. 1–2
  • Even a perfect predictor fails when the target lies beyond its imagined rollout.The paper uses the real simulator to distinguish prediction errors from the effect of target distance.Source: Figure 1 · p. 2, §1 · p. 2
  • With five-step rollouts, reliable action ranking extends only five to ten steps ahead, while tasks require 16 to 53 steps.The range does not extend with a larger predictor or longer-rollout training; the encoder affects it, with V-JEPA 2.1 most consistent.Source: §1 · p. 2

2 RELATED WORK

Prior work develops latent world models and evaluates prediction and control, while this paper focuses on how far imagined outcomes remain useful for goal-directed action selection. It studies frozen self-supervised encoders and connects that question to intermediate goals and action selection.

2 RELATED WORK

  • Latent world models predict in learned feature spaces rather than pixels, and this paper trains task-specific action-conditioned predictors on frozen self-supervised encoders.Many other world models learn their representation from the target environment alone.Source: §2 · p. 2
  • Prediction metrics can disagree with control performance, motivating closed-loop evaluation of world models.Related work also addresses multi-step prediction, short rollouts, and what representations encode about manipulation and physics.Source: §2 · pp. 2–3
  • The paper asks how far imagined outcomes from latent world models built on frozen encoders remain useful for goal-directed action selection, and what factors set that distance.Prior work has examined predictive models for evaluating VLA or generative action proposals and hierarchical planning with intermediate targets.Source: §2 · p. 3

3 EVALUATION AND SETUP

The evaluation compares action-conditioned predictors using five frozen visual encoders on Meta-World manipulation tasks and offline BridgeData V2 episodes. It measures how far ahead imagined trajectories reliably rank expert actions above random alternatives.

3 EVALUATION AND SETUP

  • The study evaluates five frozen encoders on sixteen Meta-World tasks and analyzes 2,958 WidowX episodes from BridgeData V2.Meta-World includes thirteen training tasks, three held-out tasks, four diagnostic articulated-object tasks, and additional variants and new-object or motion tasks.Source: §3 · p. 3
  • At sampled points in successful demonstrations, the model scores the next expert sequence and 100 random alternatives by comparing imagined trajectories with a future expert observation.Unless stated otherwise, the rollout horizon is K = 5 and target lookaheads are L ∈{1, 2, 3, 5, 10, 20}; lower cosine-distance scores indicate closer approaches.Source: §3 · p. 3
  • The plannable range is the largest tested target lookahead at which the task-averaged expert percentile is at most 10.That threshold means the expert sequence outranks at least 90% of random alternatives; lower expert percentiles are better.Source: §3 · p. 3
  • Figure 2 contrasts action identification and expert ranking as target distance changes, then compares each task’s plannable range with the expert’s completion length.Panel (a) uses left and right axes for identification and expert percentile; panel (b) marks plannable range in red and task duration in black.Source: Figure 2 · p. 4

4 HOW FAR AHEAD CAN WORLD MODELS GUIDE ACTION SELECTION?

The paper measures how far imagined outcomes can guide action selection, finding that a five-step rollout has a short useful range relative to task duration. The same rollout dependence appears in offline robot data and in comparisons of prediction with planning.

4 HOW FAR AHEAD CAN WORLD MODELS GUIDE ACTION SELECTION?

  • The main comparisons use K = 5, the predictor’s training rollout length, and test expert actions against 100 random alternatives.K sets how far ahead each candidate is imagined; longer rollouts increase computation and give prediction errors more steps to build up.Source: §4 · p. 4

4.1 THE PLANNABLE RANGE IS SHORT RELATIVE TO TASK COMPLETION

  • A five-step rollout has a measured range of ten control steps for V-JEPA 2.1 and five to ten across the five encoders, versus task durations of 16–53 steps.The strongest measured range covers about 30% of a typical 33-step demonstration; none of the encoders meets the ranking threshold for final task observations.Source: §4.1 · p. 4

4.2 A SIMILAR PATTERN APPEARS IN OFFLINE REAL-ROBOT DATA

  • Across 2,958 BridgeData V2 WidowX episodes, recorded-action ranking weakens with target distance, with learned-predictor ranges of zero to five dataset steps.The offline results are not directly comparable to simulation because the robot, action space, and control rate differ, and they do not represent real-robot control.Source: §4.2 · p. 4

4.3 PREDICTION AND PLANNING SHARE THE SAME ROLLOUT DEPENDENCE

  • When action identification and ranking use the same rollout length, both are most reliable when the target is at the rollout’s end and weaken when it is nearer or farther.Identification improves with target distance when the model imagines all the way to the target.Source: §4.3 · p. 5

5 WHAT DETERMINES THE PLANNABLE RANGE?

Among the tested factors, only longer imagined rollouts extend the plannable range beyond ten steps; predictor capacity and recursive training do not. The encoder also changes the range, while the short-rollout limit persists even with perfect prediction.

5.1 INCREASING PREDICTOR CAPACITY DOES NOT EXTEND THE PLANNABLE RANGE

  • An 81-fold increase in predictor capacity improves validation loss and latent prediction but leaves the five-step planning range unchanged.Source: §5.1 · p. 5

5.2 RECURSIVE TRAINING HELPS, BUT ITS BENEFIT SATURATES QUICKLY

  • For V-JEPA 2.1, one-to-five-step recursive training extends the range from five to ten steps, but training up to twenty steps adds no further range.VideoMAEv2 does not improve at any training horizon; once the rollout reaches the target, five-step-trained V-JEPA 2.1 ranks expert actions reliably thirty steps ahead.Source: §5.2 · p. 5

5.3 LONGER IMAGINED ROLLOUTS EXTEND THE RANGE AND IMPROVE CONTROL

  • Only longer rollouts extend the plannable range beyond ten steps, and longer rollouts also improve closed-loop success more than feedback at the same computation budget.Feedback helps most once the rollout reaches the goal.Source: §5.3 · p. 5, Figure 3 · p. 6

5.4 THE ENCODER CHANGES THE RANGE, AND V-JEPA 2.1 IS THE MOST CONSISTENT

  • Changing the frozen encoder shifts the plannable range between five and ten steps, with V-JEPA 2.1 among the two best in all three comparisons and the most consistent overall.The authors attribute the differences to the resulting representation as a whole, not one pre-training choice.Source: §5.4 · p. 6

5.5 DISTANCE-BASED SCORES ASSUME THAT EVERY STEP MOVES CLOSER TO THE TARGET

  • With a five-step rollout, perfect simulator prediction still sees success fall from 92% to 41% as the target moves from five to twenty steps ahead.At twenty steps, the learned model reaches 40%, close to the simulator; Euclidean distance also does not extend the range.Source: §5.5 · p. 6

6 IS PURE IMAGINATION ENOUGH TO COMPLETE TASKS?

The paper tests closed-loop planning on tasks lasting 16 to 53 steps, using goal images or nearby expert subgoals. Pure imagination succeeds in few episodes, feedback helps modestly with short rollouts, and five-step subgoals help most.

6.1 PURE IMAGINATION FAILS IN MOST EPISODES

  • Pure imagination succeeds in 23% of episodes with a five-step rollout; a thirty-step rollout raises success to 36% but uses six times more computation.Without further real observations, each new plan starts from a predicted state, allowing errors to build over episodes of up to 100 control steps.Source: §6.1 · p. 7

6.2 FEEDBACK AND REPLANNING HELP ONLY A LITTLE WITH A SHORT ROLLOUT

  • With MPC, feedback raises five-step-rollout success from 23% to 30%; with a thirty-step rollout, it raises success from 36% to 52%.The planner executes one control step, observes the real outcome, and replans; feedback helps more when the rollout reaches the goal.Source: §6.2 · p. 7

6.3 NEARBY SUBGOALS HELP MOST

  • Replacing the final goal with expert subgoals five steps ahead raises MPC success from 30% to 76%, above the 47% reached by a thirty-step rollout toward the goal at the same budget.With a randomly initialized predictor and the same subgoals, success is 0.8%.Source: §6.3 · p. 8
  • Across all three tested encoders, success rises as targets move closer, but the best encoder depends on target distance; obtaining the expert-episode subgoals without an expert rollout remains open.Source: §6.3 · p. 8

7 CAN WORLD MODELS IMPROVE VLA? TOWARDS COMBINED CONTROL

A VLA narrows candidate search to task-conditioned actions, and a world model ranks their imagined outcomes; this combination improves success when the proposals include useful actions.

7.1 A WORLD MODEL CAN IMPROVE VLA ACTION SELECTION

  • With eight proposals, the world model selects among actions the VLA proposes; it improves the policy only when those proposals include useful actions.The experiments use a fine-tuned π0 policy and V-JEPA 2.1; they do not establish a controller that works without demonstrations.Source: §7 · p. 8, §7.1 · p. 8
  • Success rises from 65% to 77% across sixteen main tasks, with no task worse; held-out variants rise from 25% to 49%.On seven tasks with room for improvement, the learned world model achieves 82% of the gain from simulator-based scoring, while an untrained predictor performs below the VLA alone.Source: §7.1 · p. 8, Figure 6 · p. 9

7.2 VLA PROPOSALS REDUCE THE CANDIDATE SEARCH

  • Success on the seven-task subset peaks at sixteen candidates and stops improving beyond that, while candidate count does not measure computation time.The VLA narrows the search to task-conditioned candidates, and the world model chooses which to execute.Source: §7.2 · p. 9

8 CONCLUSION

The conclusion locates the planning boundary in target distance relative to imagined rollout length; the appendices connect that analysis to other model families and document the evaluation setup.

8 CONCLUSION

  • With five-step rollouts, reliable ranking spans 5–10 control steps while tasks require 16–53; 81-fold predictor scaling and longer-rollout training do not extend the range.The encoder changes the range between five and ten steps, with V-JEPA 2.1 performing most consistently.Source: §8 · p. 9
  • A rollout long enough to reach the goal raised success from 30% to 47% at the same computational cost; feedback alone raised success from 23% to 30% with five-step rollouts and from 36% to 52% with thirty-step rollouts.Current latent world models complete long tasks well only when targets lie within their imagined trajectories; distant goals may require generated intermediate targets or reliable distant-goal planning (Appendix P).Source: §8 · p. 9

A EXTENDED RELATED WORK

  • The analysis separates prediction quality from goal-directed action ranking and finds both depend on rollout; extending rollouts to the target improves ranking and control, though less than exact dynamics.It relates this distinction to objective mismatch in model-based control.Source: A EXTENDED RELATED WORK · p. 13
  • Experiments use expert-demonstration intermediate targets and a fine-tuned π0 to generate task-conditioned proposals, comparing learned, untrained, and simulator-based evaluation.The paper frames its intermediate-target experiments as complementary to methods that generate such targets.Source: A EXTENDED RELATED WORK · p. 13

A.1 RELATION TO OTHER WORLD-MODEL FAMILIES

  • The evaluated predictors use frozen pretrained encoders, unlike families that learn representations with planning objectives; testing rollout-distance relations in those families remains future work.The authors suggest varying rollout before attributing a short planning range to the representation.Source: §A.1 · p. 13

B.1 WORLD-MODEL ARCHITECTURE AND TRAINING

  • Five frozen ViT-L encoders share the baseline predictor configuration, trained autoregressively over five steps; episode-level validation is distinct from task-held-out evaluation.Encoder weights are not adapted on Meta-World, and the reported checkpoints use training seed zero, with additional seeds for selected range checks.Source: §B.1 · pp. 13–14

B.2 TASK SETS AND OBSERVATION ACCESS

The evaluation spans Meta-World tasks, BridgeData V2 recordings, and several planning protocols. It defines plannable range through expert-action ranking against random alternatives, with measurement boundaries that affect interpretation.

B.2 TASK SETS AND OBSERVATION ACCESS

  • The main Meta-World evaluation covers sixteen tasks, with thirteen training tasks and three held-out tasks; four articulated-object tasks form the diagnostic set.The additional eight tasks comprise four near variants and four new tasks excluded from predictor and VLA fine-tuning.Source: §B.2 · p. 14

B.3 PLANNER SETTINGS

  • Closed-loop CEM planning uses candidate sequences of K control-step blocks, three iterations, and a warm start; MPC adds each new observation before replanning.A subgoal lies L control steps ahead on an expert rollout; the final frame can come from the same episode or another reset.Source: §B.3 · pp. 14–15, Table 1 · p. 15

B.4 BRIDGEDATA EVALUATION

  • BridgeData V2 contains 2,958 usable WidowX episodes but lacks success labels, so its offline scores are not directly comparable with success-gated simulator scores.Bridge steps are recorded frames at 5 Hz, unlike Meta-World control steps, and physical time is not equated across datasets.Source: §B.4 · p. 15

C.2 WINDOW POPULATIONS AND THRESHOLD SENSITIVITY

  • Across reporting thresholds, five-step ranges remain at or below twenty control steps and shorter than typical task lengths, but the scalar can be right-censored or non-monotone.Goals beyond episode ends are omitted, so valid-window populations can change with lookahead.Source: §C.2 · p. 16, §C.1 · p. 16, Table 2 · p. 17

C.3 PREDICTION ERROR AND RECORDED-ACTION IDENTIFICATION

The section evaluates decoded object-position prediction error and recorded-action identification. These probes use validation data and distinct measurement procedures.

C.3 PREDICTION ERROR AND RECORDED-ACTION IDENTIFICATION

  • Prediction error is measured on validation transitions with more than 5 mm of object displacement, using a ridge decoder fitted only on training episodes.The primary error is mean Euclidean object-position error in millimetres; curves are reported by lookahead rather than compressed to a scalar horizon.Source: §C.3 · p. 17
  • Recorded-action identification tests whether the recorded sequence ranks first against fifteen sampled alternatives, with chance at 1/16.It uses 720 validation windows from four articulated tasks and compares predicted and observed future endpoints by latent mean-squared distance.Source: §C.3 · p. 17
  • Decoded object-position error is plotted against prediction lookahead for V-JEPA 2.1 and constant-velocity and predict-the-mean baselines.Lower error is better.Source: Figure 8 · p. 17

D THE ROLLOUT HORIZON

The rollout horizon shapes both offline ranking and closed-loop performance: measurements are strongest when the goal lies near the imagined trajectory’s endpoint. Longer rollouts improve distant-goal results, while candidate difficulty is needed to distinguish encoders when random alternatives saturate.

D THE ROLLOUT HORIZON

  • Candidate-score spread peaks when goal lookahead L equals the five-step rollout K, and falls on either side, including on fixed windows.On the fixed-window condition, spread declines from 0.403 at L = 5 to 0.034 at L = 20; expert-to-random contrast falls from 0.954 to 0.070.Source: D THE ROLLOUT HORIZON · p. 18, Table 3 · p. 18

D.1 SWEEPING THE ROLLOUT HORIZON

  • At K = 20, both predictors rank the expert ahead of all 100 random candidates at L = 20, moving P ∗ from 10 to 20, a lower bound on range.At K = 10, the corresponding percentiles remain above threshold at 15.71 and 15.86; at K = 5 they are 32.90 and 37.07.Source: §D.1 · p. 18, Table 4 · p. 19

E MATCHING THE ROLLOUT IN CLOSED-LOOP CONTROL

F HARDER ALTERNATIVES

  • Against random alternatives at a matched rollout, expert percentiles can saturate at zero, preventing the measurement from distinguishing encoder quality.Harder perturbed alternatives restore separation, so encoder comparisons depend on the candidate distribution.Source: F HARDER ALTERNATIVES · p. 21
  • Against perturbed-expert alternatives, V-JEPA 2.1 has mean percentile 6.54 at σ = 0.25 and a thirty-step rollout, but 21.64 at σ = 0.1.At a matched twenty-step rollout with σ = 0.25, encoder percentiles span 13.02 for V-JEPA 2.1 to 35.43 for VideoPrism; random alternatives yield only a 0.34-point spread.Source: F HARDER ALTERNATIVES · p. 21, Table 7 · p. 22, Figure 10 · p. 22

G COMPARING PREDICTION AND RANKING UNDER ONE PROTOCOL

H PREDICTOR CAPACITY, TRAINING HORIZON AND SEARCH BUDGET

Across the capacity, target-distance, and control comparisons, reliable planning depends on the relation between imagined rollout length and target distance, not simply on predictor scale or training horizon. Nearby subgoals and world-model selection among VLA-proposed actions improve control in the evaluated settings.

H PREDICTOR CAPACITY, TRAINING HORIZON AND SEARCH BUDGET

  • The five-step rollout’s plannable range remains ten as predictor capacity grows, while V-JEPA 2.1 reaches range ten with training horizons of five, ten, or twenty steps.The baseline has 3.75 M parameters, and the largest predictor has 305.5 M; training beyond five recursive steps improves expert-action percentile by at most 3.2 points.Source: H PREDICTOR CAPACITY, TRAINING HORIZON… · pp. 23–24, Figure 12 · p. 24
  • Both baseline and tenfold-larger CEM searches achieve 0/60 successes on three failure tasks, so the larger search does not recover these failures.The comparison keeps rollout horizon and objective fixed.Source: H PREDICTOR CAPACITY, TRAINING HORIZON… · p. 24

I LATENT DISTANCE AND EXPERT-ACTION RANKING

  • Observed distance to a goal can decrease along demonstrations even when learned rollouts fail to rank the expert action, and four of five encoders have unchanged range under squared-Euclidean scoring.The fifth encoder’s range shortens with squared-Euclidean scoring; both tested scores compare a short trajectory with a target state.Source: I LATENT DISTANCE AND EXPERT-ACTION… · p. 24, Figure 13 · p. 25

J PREDICTING WHERE PLANNING WORKS

  • Seven of eight additional task-success predictions are correct using offline range: positive range predicts at least 95% success, while zero range predicts at most 30%.Positive-range encoder–task pairs also have higher closed-loop success across thirteen training tasks and three encoders.Source: J PREDICTING WHERE PLANNING WORKS · p. 25, Figure 14 · p. 25

L THE EXACT-DYNAMICS REFERENCE

  • With simulator dynamics and a fixed five-step rollout, shorter target lookahead yields 165 paired gains and no losses across sixteen tasks.The simulator removes learned prediction error, but distant-goal failure remains when rollout stays short.Source: L THE EXACT-DYNAMICS REFERENCE · p. 26, Table 10 · p. 26

M ACTING WITHOUT OBSERVATIONS

  • On near variants, the world model maintains success above 90% for up to ten control steps without new observations, near the offline plannable range; the VLA remains below 30%.Source: M ACTING WITHOUT OBSERVATIONS · p. 27

N.1 STANDALONE VLA AND WORLD-MODEL PERFORMANCE

  • On four near variants, standalone world-model success is 98.8% versus 25.0% for the VLA, but both remain near floor on four tasks with new objects or motion patterns.The comparison uses different candidate-generation procedures and the world-model planner receives demonstration subgoals.Source: §N.1 · p. 28, Figure 17 · p. 28, Table 11 · p. 28

N.4 THE BASELINE EXECUTES THE SAME NUMBER OF ACTIONS

  • World-model selection improves success by 13.4 percentage points over a matched VLA chunk baseline, with 45 paired gains and 2 losses; the learned critic recovers 82.2% of simulator-scoring improvement on seven tasks.The matched baseline and selection execute two actions per control interval; the critic comparison is not significant (p = 0.134).Source: §N.4 · p. 29, Figure 18 · p. 30, Table 12 · p. 30

P SCOPE OF THE RESULTS

The results span simulated manipulation and offline real-robot interactions, but the tested rollout length, task trajectories, and subgoal source constrain what the results establish.

P SCOPE OF THE RESULTS

  • Offline action-ranking distance drops on 2,958 WidowX episodes as target distance grows, alongside Meta-World closed-loop evaluations on sixteen main and eight additional tasks.The range is measured on four articulated-object tasks; most comparisons use one training seed.Source: P SCOPE OF THE RESULTS · p. 32
  • Rollouts are tested only up to thirty control steps, and few tasks provide enough expert steps to test targets beyond thirty.Ranges reaching the largest tested lookahead are lower bounds; longer rollouts also require more computation and fewer candidate evaluations at fixed budget.Source: P SCOPE OF THE RESULTS · p. 32
  • Control gains using nearby subgoals rely on expert rollouts from the same episode; generating such subgoals without expert rollouts remains future work.The plannable range is a goal-ranking diagnostic, not an optimiser-independent planning bound.Source: P SCOPE OF THE RESULTS · p. 32

Related papers

Loading 2609.39235v1…