DeepJEPA: Scaling World Models from Within

1 Introduction

• World-model planners usually scale outward through more candidates, longer horizons, or more search, while assigning fixed predictor depth to every transition.That uniformity mismatches smooth free-space motion with contact and multi-object coupling, where action preferences can change abruptly.Source: §1 · p. 2
• Transition-level deliberation is recurrent computation inside one imagined step before the temporal rollout continues.An update is treated as an internal action whose value depends on whether it improves the external action selected after planning.Source: §1 · p. 2
• A fixed-depth sweep finds that K = 4 never improves the aggregate mean over K = 1 across five settings, and intermediate depths are non-monotonic.The passage concludes that computation allocation, rather than more inner compute by itself, is the scaling problem.Source: §1 · p. 2
• DeepJEPA makes each latent transition a budgeted computation process, using a weight-tied recurrent cell and a learned continue head for each candidate–time pair.At inference, the decision uses the current recurrent state and does not alter candidate budget, rollout horizon, or goal cost.Source: §1 · p. 2
• Across five visual-control settings, adaptive depth improves the best fixed-depth mean on four settings and matches it on Cube Single, averaging K = 1.00–1.26 updates.The comparison spans Reacher, Cube Double, Cube Triple, and PushT H = 10 for improvements, with a match on Cube Single.Source: §1 · pp. 2–3
• Useful computation concentrates around physical interaction and changes CEM rankings and elite membership rather than uniformly improving object-state decodability.Small latent corrections can change the selected action near a stochastic search boundary.Source: §1 · pp. 2–3

World models and latent planning

• Earlier world-model and latent-planning methods scale rollouts, optimization, or search while usually fixing computation inside each transition.JEPA methods predict features rather than reconstruct pixels, and V-JEPA 2-AC and LeWM use action-conditioned latent prediction for planning (Assran et al., 2023; Bardes et al., 2024; Assran et al., 2025; Maes et al., 2026).Source: §2 · p. 3

Robot policies and world action models

• Robot-policy work links visual and language representations to action, while newer methods couple action generation to predictive video or explicit world-model structure.DeepJEPA instead asks how much internal computation each imagined transition receives once a world model is used inside a planner.Source: §2 · p. 3

Adaptive test-time computation

• Adaptive test-time methods vary sampling, search, verification, or network depth, while DeepJEPA allocates depth to one transition of one candidate trajectory.Unlike test-time training or adaptive planning methods described here, DeepJEPA neither updates parameters nor changes the outer planning budget; its value is determined by resulting rank.Source: §2 · p. 3

3.1 Budgeted recurrent transitions and learned halting

• A shared recurrent cell generates successive latent predictions, while a continue head selects depth separately for each candidate and rollout step.The predictor is trained through Kmax = 4; its continue head is trained with binary cross entropy, and inference halts at the first depth whose predicted continuation probability is at most η, or at Kmax.Source: §3.1 · p. 4

3.2 Planner-relative computation and CEM

• DeepJEPA casts each additional update as computation valued through the fixed outer planner, while its learned stopping rule uses a reward-free proxy.Candidate distributions, elite count, CEM iterations, and rollout length remain fixed, isolating internal transition depth from outward search budget.Source: §3.2 · pp. 4–5

3.3 Theoretical support: when refinement can change CEM

• Adaptive depth improves the fixed-depth frontier on four of five settings, as summarized by Table 1.The caption reports mean success rates over three training/evaluation seed pairs and identifies the last row as recurrent updates, not percent.Source: Table 1 · p. 6
• If the largest absolute candidate-cost correction is less than half the elite margin, one more update preserves the elite set and standard unweighted CEM proposal statistics.When the margin contracts, a candidate-specific correction may cross the elite boundary; the guarantee is sufficient, not a claim that deeper prediction always improves control.Source: §3.3 · p. 6

• Fixed-depth and learned-depth evaluations share roots and CEM seeds within recurrent checkpoints, using three training/evaluation seed pairs and 50 episodes per pair.The main comparison uses the highest mean-success point from the threshold sweep.Source: §4 · p. 7
• Uniform K = 4 never exceeds K = 1 across the five settings, and intermediate fixed depths are non-monotonic.These comparisons hold the recurrent checkpoint, episode roots, candidate budget, and CEM seeds fixed.Source: §4 · p. 7
• DeepJEPA beats the best fixed recurrent mean on Reacher (85.3%), Cube Double (74.0%), Cube Triple (77.3%), and PushT (11.6%), and matches Cube Single (79.3%).Mean depths are 1.03, 1.26, 1.22, 1.02, and 1.00, respectively.Source: §4 · pp. 7–8
• Refinement rises from 1.24% immediately before PushT contact to 9.02% at onset, while sustained Cube Triple contact raises refinement probability by 12.5 points.PushT refinement remains at 8.03% one step after onset; Cube Triple contact release reduces refinement by 9.9 points.Source: §4 · p. 8, Figure 4 · p. 9
• 93–99% of roots retain the same outcome; depth changes decisions near CEM elite boundaries, where candidate-specific cost corrections can reorder elites.Across 450 Cube Triple root–search-seed pairs, the net effect is +1.1 points with a 95% interval crossing zero; exact-batch replay reproduces 297 of 300 outcome labels.Source: §4 · p. 9, Figure 5 · p. 10
• PushT probe errors shift by only 0.0105 and 0.0036 pixels, while Cube Triple object-position error worsens by 1.7 µm, providing no uniform decodability gain.The result is consistent with sparse candidate-specific updates mattering through relative cost ordering rather than globally better state features.Source: §4 · pp. 9–10, Figure 6 · p. 11

• Search chooses which futures to consider, while the transition model chooses which changes deserve deeper processing; contact onset and sustained interaction emerge as high-compute regimes without contact labels.Source: §5 · p. 11
• Refinement cannot change the CEM update when its largest cost correction is below half the elite margin, but smaller corrections can matter as that margin contracts.Source: §5 · p. 11
• Future continue targets could estimate rank margin, elite-set stability, or action sensitivity to allocate computation at the prediction-to-decision interface.Source: §5 · p. 11

• Across five visual-control settings, DeepJEPA improves or matches the fixed-depth frontier while remaining near K = 1 on average.Source: §6 · p. 11
• Deeper updates concentrate around physical interaction and can reshape CEM rankings near decision boundaries without uniformly improving state decodability.Source: §6 · p. 11
• When candidate cost corrections cannot cross the elite margin, another update leaves the CEM proposal unchanged; when the margin contracts, a small correction can redirect search.The conclusion suggests rank stability or action sensitivity as more direct future training signals.Source: §6 · pp. 11–12

• Reacher and Cube use 300 CEM candidates, 30 iterations, 30 elites, and 100-step episodes; Cube goals are sampled 25 dataset steps after the initial observation.Source: A Experimental Protocol · p. 16, Table 3 · p. 16
• Each setting uses three training/evaluation seed pairs and 50 episodes per pair; fixed K = 1, 2, 3, 4 and learned halting share checkpoints, roots, and CEM seeds.The reported main comparison selects the best mean-success threshold from the full sweep.Source: A Experimental Protocol · p. 16, Table 3 · p. 16
• One-step probes use 10,000 evaluation transitions, while contact analyses use 100 roots and one checkpoint per task; Table 4 reports success-rate standard deviations across the three seed pairs.Source: A Experimental Protocol · p. 16, Table 4 · p. 16

B Decision-Theoretic Details

• The allocation objective trades downstream action utility against the price of recurrent updates beyond K = 1.It assigns depth Ki,h to candidate i at imagined step h and prices each extra update with λcmp.Source: B Decision-Theoretic Details · p. 17
• Holding the candidate distribution, rollout horizon, elite count, and CEM iterations fixed isolates internal transition depth from outward search budget.Source: B Decision-Theoretic Details · p. 17
• With nonincreasing expected marginal gains, the one-step policy continues only while an update’s expected utility gain exceeds its price.This threshold argument does not establish global optimality for a finite stochastic CEM run.Source: B Decision-Theoretic Details · p. 17
• Because counterfactual task utility is unavailable for every imagined update, the continue head uses relative latent-error reduction as a proxy rather than true value of computation.The head amortizes whether that proxy exceeds τrel.Source: B Decision-Theoretic Details · p. 17

C Proof of Elite-Set Stability

• The proof compares costs for a fixed batch of candidate action sequences, with an elite count M smaller than the batch size N.The margin is the separation at the elite boundary, and one update changes costs without changing the action sequences.Source: C Proof of Elite-Set Stability · p. 17
• If the elite-boundary margin exceeds twice the largest cost correction, no elite and non-elite candidate can exchange order after one update.The condition also assumes a positive boundary margin, ruling out a tie.Source: C Proof of Elite-Set Stability · pp. 17–18
• Under that condition, the M candidates that were cheapest remain precisely the M smallest-cost candidates after refinement.Source: C Proof of Elite-Set Stability · p. 18
• With the elite set unchanged and action sequences fixed, the standard unweighted CEM update produces identical proposal mean and covariance.Source: C Proof of Elite-Set Stability · p. 18

D Scope and Limitations

• The main evaluation uses three training and evaluation seed pairs and 150 episodes per setting, with thresholds selected from the measured sweep rather than held-out validation.Its values characterize an attainable success-compute frontier, not a deployment-time threshold-selection rule.Source: D Scope and Limitations · p. 18
• Contact and probe studies use one checkpoint per task, so their uncertainty reflects sampled roots or transitions rather than training variation.Source: D Scope and Limitations · p. 18
• Mean selected depth does not imply proportional wall-clock savings under batched execution, where a few continuing candidates can retain the cost of another synchronized update.PushT success remains low in absolute terms, and helped or harmed episode labels depend on the learned model and sampled CEM population.Source: D Scope and Limitations · p. 18