PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Context World Modeling for Robotic Control

Siyin Wang, Junhao Shi, Senyu Fei, Zhaoyang Fu, Li Ji, Jingjing Gong, Xipeng Qiu

arXiv:2606.26025v3cs.ROcs.CV

TL;DR

Modern VLA models struggle to generalize across novel system configurations because they lack explicit configuration conditioning. ICWM uses self-generated exploratory interactions for in-context system identification and improves performance on novel camera viewpoints in simulation and real-robot experiments.

  • Problem

    Modern VLA models lack explicit conditioning on system configurations such as camera viewpoints and robot morphology, limiting evidence for generalization across setups.

  • Method

    ICWM prepends self-generated, task-agnostic exploratory transitions to the context, enabling implicit system identification and test-time adaptation without parameter updates.

  • Results

    ICWM substantially improves over standard VLA training on novel camera viewpoints across simulation benchmarks and real-robot experiments.

  • Takeaways & Limitations

    Self-generated exploratory interactions can support adaptation to novel viewpoints, semantic scene variations, and robot morphological changes without task-specific demonstrations.

  • Takeaways & Limitations

    The standard VLA formulation assumes that the system configuration, including camera viewpoints and robot morphology, is fixed and known.

Abstract

from arXiv · show

Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.

1 Introduction

Modern VLA models [2] treat system configuration as fixed, causing failures when deployment conditions differ from training and leaving fine-tuning as a human-dependent remedy. ICWM reframes this as test-time system identification, using self-generated exploratory transitions to infer configuration and adapt without parameter updates or task-specific demonstrations.

  • Problem: VLA models [2] condition on current observations and language while treating system configuration ψ as fixed, so novel configurations cause action-observation mismatches and degraded performance [4].Relevant configurations include camera viewpoints and robot morphology.
  • Problem: System-specific fine-tuning remains the prevailing remedy, but it requires human intervention for every new setup and conflicts with generalist deployment.
  • Motivation: Existing In-Context Learning methods [6] use context to specify what to do through human demonstrations, whereas the paper identifies the missing capability as learning how the system operates.The paper frames this gap as a test-time system identification problem: recovering ψ.
  • Method: ICWM repurposes the context window for system identification by prepending visual transitions from a short sequence of random, task-agnostic exploratory movements.The model implicitly recovers the current system configuration from self-probed clips before task execution and adapts its actions accordingly.
  • Evaluation: ICWM is evaluated in simulation and on real robots, improving over standard VLA training on novel camera viewpoints while extending generalization to semantic scene variations and morphological changes.The adaptation requires neither parameter updates nor task-specific demonstrations.

2 Related Work

Prior robotic in-context methods use context to specify behavior through demonstrations, whereas world-modeling approaches explicitly learn environment dynamics with dedicated models and objectives. Related work also includes meta-learning for rapid adaptation, motivating ICWM’s distinct framing of context.

  • In-Context Learning: In-context learning [10] [11] enables on-the-fly skill acquisition, while robotic variants perform next-token prediction over observation-action sequences from expert trajectories [5, 12, 13].These trajectories may come from teleoperation or retrieved offline buffers.
  • In-Context Learning: Unlike task-agnostic adaptation, existing in-context methods use context to specify behavior and require human-provided demonstrations at test time [5, 6, 12–14].Human play videos have also been used for cross-embodiment transfer [6] [14].
  • World Modeling: World-modeling methods capture environment dynamics through forward models, inverse dynamics models, or unified objectives, all with dedicated parameters and training objectives [19] [23].Forward models predict future observations in pixel or latent space, whereas inverse models infer actions from observed visual changes.

3 Preliminary and Motivation

Standard VLA policies infer actions from the current observation and language while implicitly assuming a fixed system configuration, causing degraded performance when deployment configurations differ. The paper motivates recovering this configuration from interaction history, which contains more information than any single observation under partial observability and information-preserving dynamics.

  • Standard VLA formulation: A standard VLA policy πθ maps the current multimodal observation and language instruction to an action, often represented as an action chunk for smooth and stable control.Its parameters are optimized on data collected under specific system setups.
  • Motivation: Novel system configurations degrade standard VLA performance because observations alone do not specify camera viewpoints, mounting offsets, or robot morphology.Without explicit system configuration ψ, training marginalizes over configurations and deployment observations can have ambiguous observation-action correspondences.
  • Information content: Under partial observability and information-preserving transitions, any interaction context T contains strictly more information about ψ than a single observation.The result applies to any action sequence and follows because ψ is time-invariant while individual images cannot uniquely identify viewpoint or kinematics.
  • Information content: Task-agnostic random movements can enrich information about ψ despite carrying no task-specific information.This follows because the information advantage holds for any action distribution.

4 In-Context World Modeling

ICWM transforms a VLA into an adaptive inference mechanism that recovers latent system configuration from task-agnostic interactions. It enables demonstration-free adaptation to novel configurations without gradient updates, prior calibration, or target-environment knowledge.

  • 4 In-Context World Modeling: ICWM recovers latent configuration ψ from environmental interactions, transforming the VLA from static mapping into adaptive inference.Conditioning on interactions collected under the current configuration represents world dynamics and adapts action selection to the encountered visual and physical setup.
  • 4 In-Context World Modeling: Training prepends N randomly sampled task-agnostic interaction clips from diverse trajectories, viewpoints, and system configurations to each task query.Variation in interaction context with the underlying ψ provides an implicit training signal for configuration inference.
  • 4 In-Context World Modeling: At deployment, ICWM uses a two-phase protocol—active probing followed by in-context execution—to adapt without task-specific demonstrations or parameter updates.The protocol also requires no prior calibration or knowledge of the target environment.
  • Active Probing Phase: During active probing, the robot performs N task-agnostic actions toward random safe-workspace poses, recording spatially diverse transitions without disturbing task-relevant objects.The probing actions sample multiple directions relative to the end-effector to cover local dynamics while preserving the task initial state.
  • In-Context Execution Phase: During execution, Ψ processes the interaction context before the current observation and language instruction, producing actions aligned with the deployed physical setup.Because Ψ shares parameters with the VLA backbone, configuration-aware hidden states are built in a single forward pass.

5 Experiments

Experiments evaluate whether ICWM adapts vision-language-action models to novel system configurations without parameter updates, emphasizing viewpoint generalization, implicit system identification, and broader versatility. On simulated and real-robot benchmarks, ICWM consistently outperforms the evaluated baselines across seen and unseen viewpoints.

  • Simulated Benchmark: The simulated LIBERO benchmark evaluates Spatial, Object, Goal, and Long task suites across eight training viewpoints and six unseen OOD viewpoints, totaling 500 × 15 × 4 episodes.The cross-view protocol tests both in-domain and out-of-distribution viewpoints.
  • Real-Robot Setup: The real-robot evaluation uses a UR5e with 12 cameras, six training and six held-out test viewpoints, and four manipulation tasks with 25 trials per task per novel viewpoint.The study reports an average success rate over 600 total trials.
  • Results: 13.0%: ICWM improves OOD success rate over the Multi-View BC baseline and consistently outperforms all baselines on seen and unseen viewpoints.The comparison is reported on LIBERO success rates in Fig. 4.
  • Baselines: Controlled comparisons isolate interaction context by contrasting ICWM with Multi-View BC, which lacks context, and Explicit Configuration, which receives ground-truth camera angles as text.Pretrained NORA, π-FAST, and π0.5 models fine-tuned on a single viewpoint provide contextual references.
  • Results: ICWM mitigates viewpoint-induced position offsets and premature gripper closure by continuously grounding actions in system dynamics.These failures are exhibited by standard policies without ICWM.

6 Analysis

The analysis shows that ICWM relies on correctly paired action–observation context, learns identifiable representations, and generalizes across viewpoint, semantic, and morphological shifts. These gains require explicit training, while probing adds modest computational overhead that preserves control stability.

  • Training Requirement: A behavior-cloning policy without in-context supervision collapses to near-zero performance (< 1%), showing that in-context adaptation requires explicit training incentives.The capability does not emerge naturally from imitation learning when interaction context tokens are prepended at evaluation.
  • Context Ablation: Removing actions, images, or all context degrades performance, while false context is actively harmful, scoring 18.9 versus 22.0 without context.The results indicate that visual flow provides a coarse spatial anchor, whereas paired actions and outcomes are needed for full calibration.
  • Learned Representations: t-SNE representations form tight within-viewpoint clusters and clear between-viewpoint separation across six OOD viewpoints, indicating stability and identifiability.The visualization uses 1,024 points per viewpoint.
  • Probing Strategy: All four probing strategies outperform the multi-view baseline by 15–27%, showing that ICWM is robust to how target poses are sampled.The evaluated strategies are Random, XY-only, Z-only, and R-only.
  • OOD Generalization: ICWM maintains its advantage under semantic and morphological shifts, including 14.4% success versus MV’s 5.6% at an 80 mm gripper offset.On WindowX, MV falls from 57% to 28% while ICWM declines from 77% to 62%, widening the margin from 20 to 34 points.
  • Computational Overhead: ICWM increases inference latency from 0.112s to 0.165s with N = 3 and 0.185s with N = 5, without compromising control-loop stability.The overhead mainly comes from the 2N image and N action tokens, while static context hidden states can be pre-computed under fixed configurations.

7 Conclusion · Appendix

ICWM addresses missing system-configuration conditioning in VLA models by using self-generated interaction clips for parameter-free adaptation. Experiments show reduced spatial ambiguity under novel viewpoints, with extension to semantic scene variations and robot morphology changes.

  • Appendix: The appendix compares inference time across different settings.Figure 10 presents the inference-time comparison, but the supplied passage does not provide numerical values or identify a winner.
  • 7 Conclusion: ICWM addresses the underexplored lack of explicit system-configuration conditioning that limits modern VLA generalization.The framework is inspired by human motor adaptation and reframes the VLA policy as an adaptive inference engine.
  • 7 Conclusion: The conclusion characterizes ICWM as transforming a static VLA mapping into an adaptive inference engine.This characterization follows from conditioning on self-generated interactions and inferring system relationships at test time.
  • 7 Conclusion: ICWM conditions the policy on self-generated forward interaction clips to capture sensory-motor relationships at test time without parameter updates.This converts adaptation into in-context inference rather than fine-tuning.
  • 7 Conclusion: Experiments across simulation benchmarks and real-world platforms demonstrate that ICWM substantially reduces spatial ambiguities under novel viewpoints.The conclusion reports this effect across both simulated and physical robot settings.
  • 7 Conclusion: ICWM also extends to semantic scene variations and robot morphological changes.These extensions are reported as additional demonstrated capabilities beyond viewpoint adaptation.

A Proof of Proposition 1 · B Details of Simulated Experiment Setup · B.1 Experimental Configuration

The proof establishes that an interaction history contains strictly more information about system configuration ψ than a single initial observation. The simulated evaluation tests this advantage under out-of-domain camera viewpoints using LIBERO suites and re-rendered, filtered expert trajectories.

  • A Proof of Proposition 1: The proof shows that the interaction history carries strictly more information about system configuration ψ than the single observation o0.The argument applies the chain rule of mutual information and uses d-separation, state-chain information preservation, and the data processing inequality.
  • A Proof of Proposition 1: The proof begins from ψ ⊆ s0 and establishes a stronger mutual-information statement before deriving the proposition through the data processing inequality.Its decomposition analyzes conditional information in future observations given the initial observation and actions.
  • A Proof of Proposition 1: Under the graphical model, action variables are exogenous and paths from the initial state to actions are blocked by collider-based d-separation.Conditioning on o0 activates a relevant path from s0 through the state chain to later observations, while assumption A2 preserves information about s0.
  • B.1 Experimental Configuration: The benchmark distributes 14 azimuthal camera angles into 8 in-domain training angles and 6 withheld out-of-domain testing angles to stress-test viewpoint generalization.The train/test viewpoint allocation is illustrated in Figure 11.
  • B.1 Experimental Configuration: The evaluation uses LIBERO, comprising Spatial, Goal, Object, and Long suites that test spatial reasoning, goal conditioning, semantic grounding, and extended multi-stage manipulation.LIBERO-Long includes 10 complex manipulation sequences requiring temporal consistency and recovery from small execution errors.
  • B.1 Experimental Configuration: Training data replays original expert demonstrations and re-renders them from the 8 in-domain camera poses.Unsuccessful episodes are filtered, and redundant near-zero-action frames with static gripper states are removed to increase learning-signal density.

B.2 Detailed Results · C Details of Real-world Experiments · C.1 Experimental Configuration

ICWM outperforms competing approaches on most viewpoint conditions, including a 25.0% versus 19.8% average OOD success rate on LIBERO-Long, while using in-context calibration to stabilize viewpoint-shifted execution. Real-world evaluation uses a 6-DoF UR5e with 12 cameras, withholding half the viewpoints for zero-shot testing across diverse manipulation tasks.

  • B.2 Detailed Results: The 135° OOD viewpoint challenges all methods, as occlusion and reduced visible workspace can cause manipulation targets to leave the field of view.This limitation is attributed to shared perceptual constraints rather than an ICWM-specific failure.
  • B.2 Detailed Results: ICWM re-aligns its end-effector within the first few task steps after viewpoint-induced spatial offsets, enabling grasps that baselines frequently miss because of depth and coordinate errors.Rollout visualizations show this stability during precise grasping and multi-stage execution without environment-specific fine-tuning.
  • B.2 Detailed Results: 25.0% average OOD success on LIBERO-Long exceeds the Multi-View baseline’s 19.8%, while ICWM outperforms multi-view behavior cloning and explicit configuration approaches in most evaluated conditions.Tables 3 and 4 provide exhaustive success-rate breakdowns for in-domain and out-of-domain viewpoints.
  • C Details of Real-world Experiments: The real-robot platform uses a 6-DoF UR5e manipulator with a Robotiq 2F-85 gripper and 12 cameras spanning varied elevations and azimuthal angles.The camera array provides broad workspace coverage while introducing substantial perspective variation.
  • C.1 Experimental Configuration: Six cameras are used for training and six withheld for testing, making zero-shot success depend on functional system identification rather than memorization of camera-to-robot geometry.The balanced split evaluates adaptation to previously unseen perspectives.
  • C.1 Experimental Configuration: The evaluation covers spatial disambiguation, fine-grained cup stacking, handle-centric basket lifting, and multi-object grounding under clutter and unfamiliar viewpoints.These tasks test depth-sensitive positioning, precision alignment, structural affordance grounding, and instruction-object alignment under occlusion or visual overlap.
  • C.1 Experimental Configuration: Approximately 100–150 high-quality human demonstrations per task are collected via teleoperation to capture precision and coordination across varied initial object layouts.These demonstrations provide task-specific knowledge for the real-world evaluation.

C.2 Details of Generalization Experiments · D Details of Self-Exploration Probing

The generalization experiments test ICWM under semantic scene changes and altered robot morphology across simulated and physical platforms. The physical setup uses a 6-DoF UR5e with a parallel gripper and a 12-camera perception array spanning diverse viewpoints.

  • C.2 Details of Generalization Experiments: The physical platform uses a 6-DoF UR5e manipulator with a parallel gripper and a 12-camera array distributed across elevations and azimuths.
  • C.2 Details of Generalization Experiments: Semantic perturbation tests cover four tasks, four in-domain viewpoints, and ten trials per condition, totaling 160 trials per bar.
  • C.2 Details of Generalization Experiments: The semantic tests vary scenes using either ten task-irrelevant distractor objects or four unseen table textures absent during training.
  • C.2 Details of Generalization Experiments: UR5E morphological tests span four tasks, four viewpoints, ten trials, and three test-time configurations created by rigid spacers of ΔL ∈{20, 40, 80} mm.
  • C.2 Details of Generalization Experiments: WindowX experiments scale link lengths to 100%, 90%, 80%, and 70%, producing four distinct morphological configurations.
  • C.2 Details of Generalization Experiments: For WindowX, the model is trained on the two boundary configurations, 100% and 70% link length.

D.1 Implementation Details · D.2 Case Visualization

ICWM uses safe, task-agnostic random exploration to collect visual-motor transitions that adapt the robot’s internal coordinate system to the current configuration. The supplementary visualizations show these interaction clips and successful real-robot trajectories across manipulation tasks and disturbances.

  • D.1 Implementation Details: The probing phase synchronizes the agent’s internal coordinate system with the current camera viewpoint through safe, task-agnostic self-exploration.This phase is performed before task execution and is designed for real-robot deployment.
  • D.2 Case Visualization: Supplementary real-robot trajectories demonstrate ICWM completing lifting and pick-and-place maneuvers while remaining robust to object disturbances.Figure 15 presents these successful trajectories.
  • D.1 Implementation Details: Random end-effector movements generate diverse visual-motor transitions rather than requiring the robot to reach sampled targets, forming the in-context prefix T.A fixed-step movement policy exposes the action-observation mapping under the current configuration.
  • D.1 Implementation Details: For real-robot experiments, probing targets are sampled uniformly within the reachable workspace, whose bounding box is derived automatically from forward kinematics and joint limits.This removes the need for human annotation or scene-specific calibration.
  • D.2 Case Visualization: Figure 16 visualizes the start and end frames of random interaction clips collected in both the real world and simulation.These clips illustrate the interaction data used to construct the in-context prefix.
  • D.1 Implementation Details: Because the workspace bounds are defined in the robot base frame, the exploration remains invariant to camera viewpoint shifts and semantic scene changes.The random movements are task-agnostic and configuration-agnostic, so the zero-shot claim excludes task-specific demonstrations and parameter updates for novel configurations.
  • D.1 Implementation Details: The deployment performs 20 probing actions in approximately 5–6 seconds, then reuses the recorded context pool throughout task execution.At each inference step, N=5 triplets are sampled from the 20 transitions, with no additional probing required during the task.
Loading 2606.26025v3…