Learning Intrinsic Dynamics in Latent World Models

• With trajectories held fixed, Concat IDM’s gain over Plain JEPA grows from 15.3 pp at K=1 to 68.1 pp at K=24; at 24 appearances, success rises from 9.1% to 77.2%.Across three independent Push-T training runs, its paired improvement over Plain JEPA is 52.9 percentage points larger than with a single appearance.Source: Figure 1 · p. 1, §1 · p. 2
• Action predictability is reconstructing the executed action from consecutive latent states; the hypothesis is that this encourages distinct latent representations for different action outcomes from the same state.This matters because appearance can vary without changing physical transitions, while jointly trained encoders and predictors can achieve accurate latent prediction without preserving action-response distinctions.Source: §1 · p. 2
• When distinguishable outcomes of different actions from the same state and appearance are merged, squared inverse prediction error has a positive lower bound, although forward prediction can remain exact.Thus, action prediction penalizes lost action–consequence distinctions that forward prediction may overlook.Source: §1 · p. 2
• Supervision interventions link planning gains to encoder updates from correctly paired action labels, while frozen-encoder probes decode actions more accurately from transitions than from either endpoint alone.Together, these findings support action predictability as an inductive bias toward intrinsic dynamics, not a guarantee of complete dynamics recovery.Source: §1 · p. 2
• Concat IDM predicts actions from concatenated consecutive latent states, updates the world model’s shared encoder using offline action labels, and leaves the test-time planner unchanged.It is implemented on the LeWorldModel backbone and compared with Plain JEPA, which uses the same backbone without the inverse-dynamics objective.Source: §1 · p. 2

Learning latent dynamics

• The study asks how visual diversity over fixed physical trajectories affects jointly learned representations and the benefit of inverse supervision, a different axis from prior latent world-model work.Prior work includes latent planning in PlaNet, imagination-based behavior learning in Dreamer, and task-oriented predictive control in TD-MPC (Hafner et al., 2019; 2020; Hansen et al., 2022).Source: §2 · p. 3

Inverse dynamics and action predictability

• Inverse-dynamics supervision supports control-relevant representations, while this analysis isolates how inverse prediction penalizes merging distinguishable action consequences.Unlike concurrent ACID, the inverse loss constrains the shared encoder during training and its head is discarded for planning.Source: §2 · p. 3

External visual factors

• Methods for visual variation include domain randomization, augmentation, bisimulation, and matched-history consistency, whereas this study varies appearances without explicit invariance or matched-view supervision.Examples include CURL, RAD and DrQ (Laskin et al., 2020b; Kostrikov et al., 2020), and ACPC.Source: §2 · p. 3

3.1 INTRINSIC DYNAMICS AND EXTERNAL FACTORS

• Intrinsic dynamics are the action-conditioned physical transitions, whereas external factors change the rendering without changing those transitions.The setup distinguishes the physical state at the executed action from factors that affect only visual observation.Source: §3.1 · p. 3
• The appearance-diverse datasets reuse the same physical trajectories and action sequences while assigning each trajectory one rendering from a finite palette, without matched renderings during training.The study also varies episode-level appearance assignment while keeping the trajectory pool and palette fixed.Source: §3.1 · p. 3

3.2 ACTION PREDICTABILITY AND TRANSITION DISTINCTIONS

• When an encoder merges distinguishable outcomes of different actions, inverse prediction incurs a positive error bound even though a forward predictor can predict their shared latent successor exactly.The characterization extends to arbitrary transition distributions: encoding adds inverse-prediction error when it removes distinctions needed to recover the conditional action mean.Source: §3.2 · p. 4

3.3 CONCAT IDM AND TRAINING OBJECTIVE

• Concat IDM gives the inverse-dynamics decoder both endpoint representations through concatenation, and its inverse loss propagates through both endpoint encodings to the shared encoder without directly updating the forward predictor.In the main experiments, SIGReg weight β=0.09 and IDM weight λ=1.Source: §3.3 · p. 5
• The stop-gradient and shuffled-target interventions test whether gains require correctly paired action labels to update the encoder, while ∆z-IDM tests displacement inputs against concatenated endpoints.The stop-gradient variant trains the inverse head but blocks its gradients from the encoder; shuffled targets break action-transition correspondence.Source: §3.3 · p. 5

3.4 LATENT PLANNING

• At test time, the frozen encoder and predictor roll candidate action sequences forward, and CEM replans in closed loop by minimizing terminal full-latent-space goal cost.The agent executes the first action block of the selected sequence before receiving a new observation and replanning.Source: §3.4 · p. 5

4.2 PERFORMANCE ACROSS CONTROL TASKS

• Concat IDM improves planning across Push-T, TwoRoom, and Cube without changing the planner; high-diversity gains reach 68.1 and 44.1 percentage points on Push-T and TwoRoom.On Cube, Concat IDM gains 10.2 points at K=1 and 14.7 at K=24; Reacher is saturated at 100% for both methods.Source: §4.2 · p. 6
• On Push-T at K=24, NFE falls from 1.353±0.108 to 0.652±0.032, while best progress rises from 0.481±0.046 to 0.674±0.007.TwoRoom shows the same direction: NFE falls from 0.798±0.015 to 0.457±0.019 and best progress rises from 0.387±0.008 to 0.568±0.010.Source: §4.2 · p. 6

4.3 ROLE OF APPEARANCE DIVERSITY

• The high-minus-low visual-diversity difference in Concat IDM’s planning gain on Push-T is +52.9±7.1 percentage points, positive in all three matched training blocks.The comparison renders the same physical trajectory pool under K=1 and K=24 and matches training blocks across conditions.Source: §4.3 · p. 7
• Across nine changed training appearance assignments on Push-T, Concat IDM achieves 74.2–78.4% success versus Plain JEPA’s 5.6–61.2%, improving success and NFE in every matched pair.Evaluation renderings remain fixed, so the comparison tests sensitivity to how trajectories were grouped by appearance during training.Source: §4.3 · p. 7

4.4 ACTION INFORMATION AND PLANNING COSTS

• Frozen-encoder probes decode actions from concatenated endpoints at R2 0.799±0.001 with Concat IDM versus 0.251±0.209 with Plain JEPA.Concat IDM’s displacement score is 0.846±0.002 versus 0.254±0.003 from its better individual endpoint; Δz-IDM reaches 0.850±0.002 despite lower mean Push-T planning success.Source: §4.4 · p. 8
• At H=5, Concat IDM’s predicted-versus-realized latent-cost Spearman correlation is 0.395–0.411, compared with 0.153–0.170 for Plain JEPA.Normalized selection regret is 0.961–1.067 versus 1.664–3.107, and top-10% overlap is 0.263–0.277 versus 0.147–0.170; these within-model rollout diagnostics are distinct from closed-loop physical-task success.Source: §4.4 · p. 8, Figure 5 · p. 9

4.5 SENSITIVITY AND SCOPE

• At λ=1 on Push-T, Concat IDM raises success from 9.1% to 77.2% and lowers NFE from 1.353 to 0.652; at epoch 3, success is 63.1% versus 10.6%.Success is 71.0% at λ=0.1 and 85.8% at λ=10; Plain JEPA peaks at epoch 5 then falls, while Concat IDM continues improving through epoch 10 and has lower NFE at every reported epoch.Source: §4.5 · pp. 8–9, Table 3 · p. 9
• The primary finding concerns seen appearances with matching current and goal renderings; on a disjoint test palette, success is 4.2% for Plain JEPA and 13.4% for Concat IDM.Appearance remains decodable, and directly measuring action-outcome merging in trained models is identified as a further step.Source: §4.5 · p. 9

5 CONCLUSION

• Concat IDM improves planning under controlled appearance variation, with larger gains at high visual diversity and benefits linked to correctly paired action supervision, accessible transition information, and improved latent rollout consistency.The conclusion recommends supervising action–consequence distinctions alongside forward prediction when learning planning representations from visually diverse observations.Source: §5 · p. 9

AI USE STATEMENT

• The authors used generative AI for prose, theoretical analysis, code development, and experiment management, and say they verified scientific claims, protocols, numbers, and analyses.They take responsibility for the final content, including AI-assisted text, claims, and artifacts.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• Code, configuration files, and evaluation scripts are planned for public release; the reported protocols use three independently trained models and 500 evaluation episodes per task, method, and appearance-set size.The additional Push-T assignment study uses three repetitions per method under each of three episode assignments on the same 500 tasks; success includes all episodes, while continuous-error metrics exclude nonfinite errors and initial distances below 10^-6.Source: REPRODUCIBILITY STATEMENT · p. 10