PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualPatchWorld: Code World Models as Latent Structured Representations for Planning

Jiaxin Bai, Jiaxuan Xiong

arXiv:2607.25236v1cs.CLcs.RO

TL;DR

World models must balance scalable learning with explicit, editable dynamics that support planning. VisualPatchWorld addresses this by selecting a qualitative program structure, fitting it over multi-step traces, and using it for model-predictive control. It achieves 69.0% mean planning success, improves the strongest code baseline by 23.5 points, and leaves contact-rich pushing as the main gap to physics-engine performance.

  • Problem

    Existing methods provide either scalable but opaque neural dynamics or executable programs without ensuring the correct qualitative form and planning utility.

  • Method

    VPW uses active probes to select a qualitative dynamical sketch, fits its parameters with multi-step rollout loss, and applies the executable program in model-predictive control.

  • Results

    69.0% mean planning success across four control domains exceeds the strongest code baseline by 23.5 points, with near-engine performance on navigation and grasp-rich control.

  • Takeaways & Limitations

    The results support automatically constructed, inspectable code world models as practical planning models across many control domains.

  • Takeaways & Limitations

    Contact-rich pushing remains harder under induced-only scoring, although selective engine re-scoring closes most of the gap.

Abstract

from arXiv · show

Different research lines use the term world model in different ways, yet they share a common aim: to capture how the world evolves under action in a form that supports perception, simulation, and planning. Two prominent realizations are neural predictors that learn dynamics in continuous vector spaces, and hand-built physics engines that expose explicit state and physical laws. Neural predictors scale from data but leave the form of the dynamics implicit; physics engines are inspectable and editable but difficult to construct at scale. We introduce VisualPatchWorld (VPW), which represents world dynamics as code. VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces by minimizing multi-step prediction error. The resulting programs can be rolled forward like a simulator, inspected in source form, and used inside model-predictive control; image-derived scene graphs can supply the live state at replan time. Across comparisons with prior code-based world models, VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points. The largest gains arise when choosing the correct qualitative dynamics is essential. Under the same planner, the induced models approach ground-truth engine success on navigation and grasp-rich control; a residual gap remains for contact-rich pushing, and checking a shortlist of promising plans in the engine closes most of that gap. These results establish a practical route toward automatically constructed code world models that are useful for planning. Code is available at https://github.com/HKBU-KnowComp/VisualPatchWorld/.

1 Introduction

VisualPatchWorld addresses the gap between scalable but opaque neural dynamics and executable code models that may use the wrong qualitative form. It selects a dynamical sketch, fits it for multi-step prediction, and uses the resulting program for planning, achieving strong gains across control domains.

  • 1 Introduction: World models support rendering, simulation, and planning, but hand-built physics engines are difficult to scale because each environment needs a human-authored dynamics model.Neural alternatives scale from interaction data but leave the dynamics form implicit, while code-based methods expose programs without necessarily recovering the right long-horizon structure.
  • 1 Introduction: Existing approaches either learn scalable but opaque dynamics or produce executable programs without ensuring the correct qualitative form or planning utility.The introduction frames these limitations as questions about automatic simulator recovery, structure selection, action-sequence ranking, and the remaining gap to physics engines.
  • 1 Introduction: VPW first selects a qualitative dynamical sketch with active probes, then fits its free parameters from state–action traces using multi-step rollout loss.The induced executable program can be rolled forward, inspected, and used to score candidate actions in model-predictive control; image-derived scene graphs provide live replanning state.
  • 1 Introduction: 69.0% mean success across four control domains exceeds the strongest code baseline by 23.5 points under a shared planner.The largest improvements occur where qualitative structure matters: arm reaching rises from 8% to 72%, cube manipulation reaches 86%, and pushing reaches 22%.
  • 1 Introduction: VPW approaches ground-truth engine success on navigation and grasp-rich control, while engine re-scoring closes most of the remaining contact-rich pushing gap.This supports a hybrid use in which the induced program performs search and the engine selectively verifies promising plans.

2 Problem Definition

The paper defines simulator-style world-model recovery as learning an executable transition program from observations and actions over structured scene descriptions. Evaluation focuses on multi-step closed-loop planning, while separating perception effects from dynamics quality.

  • 2 Problem Definition: The goal is to recover an executable transition program that humans and algorithms can inspect, edit, and roll forward from interaction trajectories.The long-term visual setting factors dynamics induction over structured traces from image-based state estimation for replanning.
  • 2 Problem Definition: The mining task learns f_θ so that g_t+1 = f_θ(g_t,a_t) over multi-step horizons, with scene descriptions extracted from observations.This separates visual abstraction ϕ from dynamics induction over structured state–action–next-state triples.
  • 2 Problem Definition: An executable world model must accept structured state, update it under actions, roll forward over action sequences, and expose an inspectable transition law.Unlike a neural latent predictor, its failed rollout can be examined and revised directly.
  • 2 Problem Definition: Closed-loop control on held-out start and goal pairs is the downstream evaluation, with receding-horizon model-predictive control serving as an application test rather than the training loss.Starts and goals remain within the same domains unless an explicit out-of-distribution test is stated.
  • 2 Problem Definition: Evaluation separates simulator-style rollout behavior, qualitative-form recovery, and candidate-action scoring relative to a ground-truth physics engine.Controlled splits localize errors across perception source, hypothesis class, rollout fidelity, and planning protocol.

3 Related Work

Related work spans latent neural world models, executable code models, and methods for selecting structured dynamics. VPW combines these lines by using structured visual abstraction, active sketch selection, and contact-aware planning.

  • 3 Related Work: Latent world models learn environment evolution in continuous embeddings and provide scalable planning representations, but failed rollouts do not expose an editable transition law.Object-centric and causal variants add structure inside the latent representation while retaining this opacity.
  • 3 Related Work: Executable code-world-model methods recover programs through code search, interaction learning, code editing, probabilistic programs, or weighted programmatic experts.These approaches move toward inspectable dynamics but differ in how they construct and compose executable models.
  • 3 Related Work: Scientific law-discovery work shows that discriminating probes can identify the correct law class when static regression fails on complex systems.VPW applies a restricted version of this structure-selection idea to robot-control dynamics.

4 Method

VPW converts visual interaction trajectories into structured transition records and executable Python dynamics. Its two-level induction separates qualitative sketch selection from parameter fitting, while the resulting program is evaluated for planner-facing multi-step control.

  • 4 Method: VPW first converts observations into structured scene descriptions, then exports paired before/after descriptions as transition records for program induction.The scene graph preserves object poses and relations such as distance, near-contact flags, and relative direction.
  • 4 Method: Separating sketch selection from parameter fitting prevents low one-step error from masking an incorrect dynamical form needed for control.The same two-level process is applied across domains while only the small environment-specific sketch family changes.
  • 4 Method: Three interchangeable extractors provide oracle, tool-based, and VLM-based scene graphs with different levels and types of perception error.The oracle removes perception error, the tool path is the default engineered RGB interface, and the VLM path provides semantic but unreliable metric coordinates.
  • 4 Method: Each exported transition record contains a pre-action scene description, the action, and a post-action scene description separated by frame stride δ.Action discretization must match δ; misalignment can preserve strong one-step prediction while degrading multi-step rollouts.
  • 4 Method: VPW selects a dynamical sketch through active probing and fits its free parameters under multi-step rollout loss to obtain an executable Python transition program.The program satisfies g_t+1 = f_θ(g_t,a_t) and plugs directly into the planner.

4 CEM-MPC Planning

VPW evaluates whether induced programs support goal-directed control through receding-horizon CEM-MPC, separating live-state observation from imagined-future scoring and optionally using engine verification.

  • Model preparation: Level 1 selects a qualitative dynamical sketch by active probing from a compact domain-specific hypothesis family before parameter fitting.Candidate alternatives include contact versus free motion, linear versus nonlinear transport, and grip-gated versus always-coupled object motion.
  • Model preparation: Level 2 fits the selected sketch’s free parameters by minimizing accumulated multi-step rollout error over the planner horizon.This penalizes models that fit single steps but drift during longer rollouts; the output is an executable transition program for planning.
  • Planning procedure: CEM-MPC samples action sequences, rolls them forward under a scoring model, updates its distribution from the best samples, and executes only the first r actions.The planner then re-observes the state before replanning.
  • Evaluation protocol: Planning conditions independently vary the source of the replan observation and the scoring model used to rank imagined futures.Observation can come from live-image tool paths or oracle simulator state, while scoring can be induced, hybrid, or simulator-based.
  • Evaluation protocol: Hybrid scoring uses the induced program for all N samples, then has the ground-truth engine re-score the top 30% shortlist before selection.Simulator scoring uses the ground-truth engine for every candidate and serves as the physics ceiling.
  • Evaluation protocol: The pipeline keeps planner settings fixed within each environment and separates observation from imagined-future scoring to attribute failures to perception, dynamics, or scoring.All conditions execute actions in the true simulator.

5 Experiments

VPW is evaluated across four LeWM environments using shared frozen planners, common simulator-state traces, and executable code baselines. Experiments show that selecting the correct dynamical structure improves planning, while hybrid physics verification addresses remaining contact-rich gaps.

  • 5.1 Experimental Setting: VPW recovers domain-specific sketches: contact-driven PD control for PushT, linear navigation, grip-gated contact, and joint-space dynamics with forward kinematics.Level 1 selects the sketch family outcome, while Level 2 fits its parameters under multi-step rollout loss.
  • 5.1 Experimental Setting: The evaluation spans navigation, contact manipulation, 3D rearrangement, and continuous control across four LeWM environments.Each evaluation uses 50 starts with goals sampled 25 steps ahead and one frozen planner per environment.
  • 5.3 Comparison against code baselines: 69.0% four-task mean success exceeds POMDP-Coder by 23.5 points under identical starts and planner settings.VPW also exceeds PatchWorld by 26.0 points.
  • 5.3 Comparison against code baselines: Choosing the correct dynamical class drives gains, with Reacher reaching 72%, Cube 86%, and PushT improving from near-zero prior-code success to 22%.The baselines and VPW share training traces, oracle replan state, and a frozen CEM planner, isolating hypothesis-class differences.
  • 5.5 When induced CEM is enough: Induced scoring recovers 0.96 of MuJoCo success on Two-room, 0.91 on Cube, 0.72 on Reacher, and 0.23 on PushT.Contact-rich PushT remains difficult because sparse contact events make small ranking errors consequential.
  • 5.6 Hybrid scoring for contact gaps: Hybrid scoring raises PushT from 22% to 88% with Tool+Hybrid and 96% with Oracle+Hybrid, while verifying only a shortlist of candidates.The method uses 0.3N physics queries per CEM iteration and retains the induced program as the search engine.

6 Discussion

VPW’s executable programs provide a planner-facing interface that supports rollout scoring, replanning, and inspection. The approach approaches engine performance in several domains, while contact-rich ranking remains the main boundary and can benefit from selective engine verification.

  • Planner-facing interface: Executable VPW programs let CEM roll out predictions, score goal distance, and replan with fresh observations.The transition law remains inspectable while the planner optimizes under the recovered scoring landscape.
  • Planner-facing interface: The two-level procedure selects a qualitative dynamical form before fitting its parameters for multi-step prediction.This shared induction process is paired with a planner that searches under the recovered dynamics.
  • What remains versus the engine: Induced scoring recovers most of the MuJoCo ceiling on Two-room and Cube, reaches 72% on Reacher versus a 100% engine ceiling, and leaves a larger PushT gap.The primary executable comparison uses frozen planners whose MuJoCo ground-truth success is at least 90%.
  • What remains versus the engine: Selective hybrid verification using 0.3N_physics queries per CEM iteration closes most of the contact gap while retaining the induced program for search.This makes engine checking an optional refinement when induced ranking is insufficient.
  • Implications: Image-derived scene graphs can provide live state at replanning time without changing the induction pipeline.The discussion identifies visual end-to-end induction, larger sketch libraries, and real-robot deployment as future directions.

7 Conclusion

VisualPatchWorld turns interaction trajectories into executable, inspectable code world models by selecting qualitative dynamics and fitting their parameters for planning. It improves over prior code models, approaches engine performance in several domains, but remains weaker on contact-rich pushing.

  • Conclusion: VPW learns world dynamics as inspectable executable code and uses the induced program for model-predictive control with optional image-based replanning.Its pipeline separates scene-graph state extraction, transition export, two-level program induction, and planning.
  • Conclusion: Correct qualitative structure and multi-step fitting make the induced programs both competitive for planning and editable when rollouts fail.The code representation exposes the transition law for inspection and revision rather than hiding it in a neural embedding.
  • Conclusion: Future work should induce programs from visual scene-graph trajectories, broaden sketch families, and evaluate out-of-distribution and real-robot settings.
  • Conclusion: Tool-based scene graphs recover oracle geometry sufficiently for RGB replanning, while VLM graphs retain uncalibrated metric coordinates.The VLM PushT coordinate error is 236 px, so calibrated tool extraction is used for RGB replanning.

F Reproducibility and Multi-Seed Intervals

The evaluation reports seed-42 point estimates alongside five-seed intervals under frozen planners and shared starts. VPW remains stable overall, while hybrid scoring substantially improves PushT.

  • Multi-seed results: 69.0% is the seed-42 Oracle+Induced suite mean, compared with 67.5±1.5% across five seeds.The five-seed results use seeds 42–46 with 50 starts per seed.
  • Multi-seed results: 95.2±5.0% is PushT’s five-seed hybrid-scoring mean, versus near 20% under induced-only scoring.
Loading 2607.25236v1…