PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context World Modeling for Robotic Control

Siyin Wang, Junhao Shi, Senyu Fei, Zhaoyang Fu, Li Ji, Jingjing Gong, Xipeng Qiu

arXiv:2606.26025v3cs.ROcs.CV

TL;DR

Standard VLA models lack explicit conditioning on system configuration, limiting their generalization to novel setups. ICWM uses self-generated exploratory interactions as context for system identification, and experiments report improvements over standard VLA training on novel camera viewpoints.

  • Problem

    Standard VLA formulations assume the system configuration ψ is fixed and known, while novel configurations are associated with degraded performance.

    Source: §3.1 · p. 3

  • Method

    Before task execution, ICWM prepends visual transitions from short, task-agnostic exploratory movements as context for implicit system identification, without task-specific demonstrations or parameter updates.

    Source: §1 · p. 2

  • Results

    ICWM significantly improves over standard VLA training on novel camera viewpoints in simulation and real-robot experiments, with reported generalization to semantic scene variations and robot morphology changes.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The findings show that self-generated interaction clips can provide test-time sensory-motor context without parameter updates.

    Source: §1 · p. 2, §7 · p. 10

    The standard VLA formulation assumes the system configuration is fixed and known, an assumption identified as limiting generalization to novel configurations.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.

1 Introduction

ICWM reframes VLA generalization as test-time system identification: robots use self-generated exploratory interactions to infer their current system configuration before task execution. The paper reports improved performance on novel camera viewpoints in simulation and on real robots.

1 Introduction

  • Like a human operator calibrating unfamiliar joystick controls, a robot can explore and infer input-output dynamics before switching to goal-directed control.This calibration is self-generated and requires no prior task knowledge, external guidance, or task-specific experience under the new mapping.Source: §1 · p. 1
  • Standard VLA models [2] condition on current observations and language while treating system configuration as fixed, so deployment changes can degrade performance [4].The prevailing remedy is setup-specific fine-tuning, which requires human intervention for each new setup.Source: §1 · pp. 1–2
  • Unlike ICL methods [6] that use context to specify behavior through test-time demonstrations, ICWM uses context to identify how the system operates.This reframes VLA generalization as a test-time system identification problem and identifies absent explicit conditioning on ψ as an underexplored failure mode.Source: §1 · p. 2
  • ICWM’s task-agnostic, self-probed clips enable implicit test-time system identification without parameter updates or task demonstrations, improving novel-viewpoint performance in simulation and real-robot experiments.The robot prepends visual transitions from short random exploratory movements as context before task execution.Source: §1 · p. 2
  • ICWM also generalizes to semantic scene variations and robot morphological changes.The paper reports validation on simulation benchmarks and real-robot experiments.Source: §1 · p. 2

2 Related Work

Prior robotic in-context methods use context to specify behavior and typically depend on demonstrations or task-time signals, whereas ICWM uses task-agnostic self-generated interactions to infer how a system operates. Unlike world models with dedicated parameters and objectives, ICWM performs implicit world modeling through standard sequence modeling without additional parameters.

2.1 In-Context Adaptation for Robotics

  • ICL imitation and cross-embodiment methods use expert trajectories or play videos to specify behavior, and require human-provided demonstrations at test time; ICWM instead needs no demonstrations, reward, or parameter updates.ICWM uses task-agnostic self-generated interactions to understand how the system operates.Source: §2.1 · pp. 2–3
  • Gradient-based meta-learning enables rapid test-time fine-tuning, while recurrent meta-RL infers which task to perform from interaction history but relies on reward signals unavailable before task execution.Source: §2.1 · p. 3

2.2 World Modeling for Robotic Control

  • World models predict future observations in pixel or latent space, infer actions from visual changes, or unify both objectives; these approaches use dedicated parameters and training objectives, unlike ICWM’s implicit, zero-additional-parameter modeling.ICWM uses standard sequence modeling to infer time-invariant causal structure, including control mappings and camera viewpoints, from task-agnostic interaction histories.Source: §2.2 · p. 3

3 Preliminary and Motivation

Standard VLA policies treat system configuration as fixed, which leaves them poorly equipped for novel setups; under partial observability and information-preserving transitions, interaction context carries more information about configuration than a single observation, even when actions are task-agnostic.

3.1 Standard VLA Formulation and Its Limitation

  • Standard VLA policies map current observations and instructions to actions but implicitly assume fixed, known configurations, leaving them without context to interpret novel setups.Actions are often emitted as chunks for temporal smoothness and control stability.Source: §3.1 · p. 3

3.2 Interaction Context Enriches Configuration Information

  • Under partial observability and information-preserving transitions, any interaction context from an action sequence carries strictly more information about ψ than a single observation.The latent state separates time-invariant system configuration ψ from time-varying scene state ξk.Source: §3.2 · pp. 3–4
  • Because the result holds for any action distribution, task-agnostic random movements can enrich information about ψ without carrying task-specific information.The proof is provided in Appendix A.Source: §3.2 · p. 4

4 In-Context World Modeling

ICWM turns the VLA from a static mapping into an adaptive inference mechanism that recovers latent system configuration from task-agnostic interactions. The resulting interaction context represents the system’s dynamics and helps adapt actions to the visual and physical setup at test time.

4 In-Context World Modeling

  • ICWM uses task-agnostic interactions to recover latent system configuration and adapt action selection to the test-time visual and physical setup.The interaction context represents the world dynamics under the current configuration.Source: §4 · p. 4

4.1 In-Context Training

  • Training shares configuration-inference parameters with the VLA and prepends diverse interaction clips, whose configuration-dependent variation teaches the model to extract system dynamics.Clips are sampled from training trajectories and viewpoints, providing an implicit signal to learn the action-to-observation mapping for each setup.Source: §4.1 · pp. 4–5

4.2 Test-Time Active Probing and Inference

  • At deployment, ICWM adapts to novel configurations without task-specific demonstrations, gradient updates, prior calibration, or knowledge of the target environment.Before task execution, the robot performs safe, spatially diverse probes to random target poses, avoiding contact with task-relevant objects and preserving the initial state.Source: §4.2 · p. 5
  • After probing, the policy processes the inferred configuration representation before the task query, then generates actions conditioned on that representation, the current observation, and the language instruction.Because configuration inference shares the VLA backbone, this is implemented as a single forward pass.Source: §4.2 · p. 5

5 Experiments

ICWM improves performance on unseen viewpoints in LIBERO simulation, with particularly large gains on long-horizon tasks. The experiments also evaluate cross-view generalization using simulated and real-robot setups.

5.1 Experimental Setup

  • Evaluation covered LIBERO’s four suites at eight training and six unseen viewpoints, and four UR5e manipulation tasks across six held-out camera views.The simulated protocol comprised 500 × 15 × 4 episodes; real-robot evaluation used 25 trials per task per novel viewpoint, 600 total.Source: §5.1 · pp. 5–6

5.2 Simulation Results

  • ICWM raises OOD success by 13.0% over MV and 9.5% over EXP; on LIBERO-Long, it exceeds MV by 29.9% seen and 26.3% unseen.These are the largest relative margins across LIBERO suites for the long-horizon results.Source: Figure 4 · p. 6, §5.2 · p. 6

5.1 Experimental Setup

  • Controlled baselines isolate interaction context: MV shares ICWM’s architecture and multi-view data but lacks context, while EXP also receives ground-truth camera angles as text.Pretrained contextual references were fine-tuned on a single viewpoint and were not treated as direct benchmarks.Source: §5.1 · p. 6

5.2 Simulation Results

  • ICWM mitigates cascading failures in long-horizon tasks by continuously grounding actions in system dynamics despite viewpoint-induced spatial errors.The paper attributes baseline failures to small spatial errors accumulating over a task.Source: §5.2 · pp. 6–7

6 Analysis

The analysis shows that ICWM depends on interaction context to infer system configuration, and that this capability supports adaptation across probing strategies and out-of-distribution scene and morphology changes. Its learned representations cluster by viewpoint, while in-context adaptation does not emerge from imitation learning alone.

6.1 What Does the Interaction Context Contribute?

  • Removing images caused the largest average collapse (−56.4%), while false context scored 18.9 versus 22.0 with no context, showing that missing or misaligned context harms adaptation.Removing actions degrades performance more moderately; the ablation indicates that visual outcomes and paired action–observation information contribute to system-dynamics inference.Source: §6.1 · p. 7, Table 1 · p. 8
  • A behavior-cloning policy trained without in-context supervision achieves under 1% performance when interaction tokens are prepended, indicating that this capability must be explicitly incentivized during training.The comparison tests whether in-context adaptation emerges from standard sequence modeling under the same interaction context.Source: §6.1 · pp. 7–8
  • Across six OOD viewpoints, the hidden representations show tight within-viewpoint clustering and clear between-viewpoint separation, indicating stability and identifiability.The t-SNE visualization includes 1,024 points per viewpoint.Source: §6.1 · p. 8, Figure 7 · p. 8

6.2 What Probing Strategy is Effective?

  • All four probing strategies outperform the multi-view baseline by 15–27%, although performance varies by viewpoint and no strategy dominates across all viewpoints.The strategies are Random, XY-only, Z-only, and R-only; the results attribute ICWM’s benefit to the interaction format rather than a particular movement pattern.Source: §6.2 · p. 8, Table 2 · p. 8

6.3 Does ICWM Generalize Beyond Camera Shifts?

  • ICWM exceeds MV under distractor objects (35.0 vs. 27.5) and novel table textures (41.2 vs. 37.5), with more moderate gains than under viewpoint shifts.The passage attributes the more moderate gains as likely reflecting scarce diverse scene-configuration data, rather than a fundamental limitation of the mechanism.Source: §6.3 · p. 8, Figure 8 · p. 9
  • With rigid spacers of ΔL ∈{20, 40, 80} mm altering gripper-flange kinematics, ICWM consistently outperforms MV across all offsets with a stable margin.This result supports adaptation by inferring altered kinematics from probing context.Source: §6.3 · pp. 8–9, Figure 8 · p. 9
  • On WindowX, as link-length offset rose from 10% to 20%, MV success fell from 57% to 28%, while ICWM fell from 77% to 62%, widening its margin from 20 to 34 points.The model trained on 100% and 70% link lengths and was evaluated zero-shot at interpolated 90% and 80% configurations; the result is consistent with inference of altered kinematics from probing context.Source: §6.3 · p. 9, Figure 9 · p. 9

7 Conclusion · Appendix

ICWM addresses the lack of explicit system-configuration conditioning in modern VLA models by using self-generated interactions to infer how a system operates. The paper reports reduced spatial ambiguities under novel viewpoints and extension to semantic scene variations and robot morphological changes.

7 Conclusion

  • The work identifies the absence of explicit system-configuration conditioning in modern VLA models as an underexplored limitation to generalization.Source: §7 · p. 9
  • Inspired by human motor adaptation, ICWM transforms the VLA policy from a static mapping into an adaptive inference engine.Source: §7 · p. 9
  • Across simulation benchmarks and real-world platforms, ICWM substantially reduces spatial ambiguities under novel viewpoints.The conclusion also reports extension to semantic scene variations and robot morphological changes.Source: §7 · p. 10
  • ICWM also extends to semantic scene variations and robot morphological changes.These extensions are reported alongside the novel-viewpoint results.Source: §7 · p. 10
  • ICWM conditions the policy on self-generated forward-interaction clips to capture the underlying sensory-motor relationship at test time.Source: §7 · p. 10
  • This test-time inference requires no parameter updates.Source: §7 · p. 10

A Proof of Proposition 1 · B Details of Simulated Experiment Setup · B.1 Experimental Configuration

Proposition 1 establishes that a history of interactions carries more information about system configuration than a single observation. The simulated setup tests viewpoint generalization using held-out camera angles and four LIBERO suites, with training data re-rendered from expert demonstrations.

A Proof of Proposition 1

  • Interaction history carries strictly more information about system configuration ψ than a single observation o0 alone.The proof uses the data processing inequality to connect the stronger information statement to the proposition.Source: A Proof of Proposition 1 · p. 15
  • The proof applies the mutual-information chain rule and d-separation: action nodes are exogenous roots, blocking paths from the initial state to actions.It then argues that conditioning on o0 activates a collider path to later observations, and that the state chain preserves information about s0 under A2.Source: A Proof of Proposition 1 · p. 15

B.1 Experimental Configuration

  • Eight in-domain camera angles are used for training and six withheld out-of-domain angles for evaluation, testing viewpoint generalization across 14 azimuths.The train angles are 30°, 60°, 90°, 120°, 240°, 270°, 300°, and 330°; test angles are 45°, 135°, 225°, 255°, 285°, and 315°.Source: §B.1 · p. 15, Figure 11 · p. 16
  • The LIBERO benchmark comprises Spatial, Goal, Object, and Long suites for spatial reasoning, goal-conditioning, semantic grounding, and extended multi-stage manipulation, respectively.LIBERO-Long includes 10 complex manipulation sequences.Source: §B.1 · pp. 15–16
  • Training data is synthesized by replaying expert benchmark trajectories and re-rendering them from the eight in-domain camera poses.Preprocessing filters unsuccessful episodes and removes redundant frames with near-zero action norms and a static gripper, following OpenVLA [2].Source: §B.1 · p. 16

B.2 Detailed Results · C Details of Real-world Experiments · C.1 Experimental Configuration

ICWM achieves a 25.0% average OOD success rate on LIBERO-Long, compared with 19.8% for the Multi-View baseline. The real-world evaluation tests adaptation to unseen camera views across four manipulation tasks.

B.2 Detailed Results

  • At 135°, occlusion and reduced visible workspace challenge all methods, sometimes taking manipulation targets out of view; the limitation is not specific to ICWM.The paper attributes the difficulty to viewpoint-specific geometric constraints.Source: §B.2 · p. 16
  • In rollout visualizations, ICWM re-aligns its end-effector within the first task steps despite initial spatial offsets, enabling successful grasps that baselines often miss due to depth and coordinate errors.Figure 12 also illustrates precise grasping and multi-stage execution without environment-specific fine-tuning.Source: §B.2 · pp. 16–17, Figure 12 · p. 18
  • 25.0% average OOD success on LIBERO-Long, versus 19.8% for Multi-View (MV); ICWM outperforms multi-view behavior cloning and explicit configuration approaches in most settings.The OOD success-rate breakdown is reported in Table 4.Source: §B.2 · p. 16, Table 4 · p. 19

C.1 Experimental Configuration

  • The real-robot workstation uses a 6-DoF UR5e with a Robotiq 2F-85 gripper and 12 cameras at varied elevations and azimuths, creating perspective-induced spatial distortions.The camera array is positioned to provide comprehensive workspace coverage.Source: §C.1 · p. 19
  • The zero-shot protocol trains on 6 cameras and withholds 6 for testing; each task has approximately 100–150 teleoperated human demonstrations.The split tests previously unseen perspectives, with success intended to depend on functional system identification rather than memorized camera-to-robot geometries.Source: §C.1 · pp. 19–20
  • The four tasks test spatial disambiguation, fine-grained cup stacking, handle-centric basket grasping, and semantic object-instruction alignment in clutter under unfamiliar views.They involve placing a toy into a basket, stacking a yellow cup onto a red cup, lifting a basket, and placing an eggplant onto a red plate.Source: §C.1 · pp. 19–20

C.2 Details of Generalization Experiments · D Details of Self-Exploration Probing

The generalization experiments examine scene-level perturbations and altered robot kinematics, including interpolated link-length configurations. The real-world platform uses a UR5e manipulator and a 12-camera perception array.

C.2 Details of Generalization Experiments

  • The real-world platform uses a 6-DoF UR5e with a parallel gripper and 12 cameras distributed across elevations and azimuthal angles.Source: Figure 13 · p. 20
  • Semantic-perturbation tests cover 4 tasks, 4 in-domain viewpoints, and 10 trials per condition, or 160 trials per bar.Source: §C.2 · p. 20
  • One semantic variation places 10 task-irrelevant distractor objects in the workspace.Source: §C.2 · p. 20
  • A second semantic variation replaces the table surface with 4 unseen textures absent during training.Source: §C.2 · p. 20
  • UR5E morphological tests attach rigid spacers of ΔL ∈{20, 40, 80} mm to the gripper flange to alter forward kinematics at test time.Tests span 4 tasks, 4 in-domain viewpoints, 10 trials, and 3 morphological settings.Source: §C.2 · p. 20
  • On WindowX, link lengths are scaled to 100%, 90%, 80%, and 70%; training uses the 100% and 70% boundary configurations, while evaluation uses the interpolated 90% and 80% configurations without parameter updates.Source: §C.2 · p. 20
  • WindowX evaluation covers 3 tasks × 20 trials × 2 morphological settings, totaling 120 trials.Source: §C.2 · p. 20

D.1 Implementation Details · D.2 Case Visualization

ICWM uses brief, task-agnostic robot exploration to build an in-context prefix that captures the current system’s action–observation mapping before task execution. The implementation constrains real-robot probing to the reachable workspace and reuses its context, while the case visualization shows random interaction clips from real-world and simulated settings.

D.1 Implementation Details

  • Safe, task-agnostic self-exploration synchronizes the agent’s internal coordinate system with the current camera viewpoint before execution.The exploration is designed with real-robot deployment safety in mind.Source: §D.1 · p. 20
  • The robot moves toward randomly sampled targets in fixed steps, without needing to reach them, to collect diverse visual–motor transitions as the in-context prefix.These interaction fragments expose the action–observation mapping under the current configuration.Source: §D.1 · pp. 20–21
  • For real-robot probing, targets are sampled within the reachable workspace, which is derived automatically from forward kinematics and joint limits without manual annotation or scene-specific calibration.Fixed-step motions stay within operable joint limits and span diverse workspace regions.Source: §D.1 · pp. 21–22
  • The base-frame workspace remains invariant to camera-viewpoint shifts and semantic scene changes; zero-shot adaptation requires no task-specific demonstrations or parameter updates, and probing is unnecessary in simulation.The bounding box requires only a one-time specification for a fixed physical workstation setup.Source: §D.1 · p. 22
  • Twenty probing actions take approximately 5–6 seconds once per deployment; each inference step samples N=5 triplets from the recorded transitions, and the pool is reused throughout the task.Reusing the same configuration across tasks further amortizes the one-time overhead.Source: §D.1 · p. 22

D.2 Case Visualization

  • Start and end frames of random-action interaction clips are shown for both real-world and simulation settings.The visualization uses os and oe to denote the start and end frames, respectively.Source: §D.2 · p. 22, Figure 16 · p. 22

Related papers

Loading 2606.26025v3…