PaperlayerView exact revision on arXiv

Source-linked AI summary

DeltaWorld: Physically Consistent Interactive World Simulators via Action-Conditioned Latent Increment Learning

Boyuan Hou, Xiaoge Cao, Chaofan Zhang, Shuo Wang, Shaowei Cui

arXiv:2610.02691v1cs.RO

TL;DR

World models for robotic manipulation need to predict action consequences despite small changes in largely static scenes. DeltaWorld predicts action-induced latent increments and emphasizes interaction-sensitive regions; evaluations report improved video-prediction metrics, including lower FVD and LPIPS than IWS on the cross-robot dataset.

  • Problem

    Models predicting complete future frames or latent states may miss small action-driven object changes in mostly static scenes, and rollout errors can grow into object drift or implausible deformation.

    Source: §I · p. 1

  • Method

    DeltaWorld predicts action-conditioned latent increments, adds them to the current state, and uses counterfactual interaction regions to emphasize related latent changes during training.

    Source: §I · p. 1, B. Delta Latent Transition Model · p. 3, C. Interaction-aware Latent Alignment · p. 4

  • Results

    FID fell from 44.16 to 36.04, FVD from 264.52 to 231.98, and LPIPS from 0.0661 to 0.0490 versus IWS in average results across tasks and views.

    Source: B. Video Baseline Comparisons · p. 6

  • Takeaways & Limitations

    The increment-based, interaction-aware formulation provides an approach for learning robotic world dynamics, with evaluations reporting improved action-conditioned video prediction across image- and video-level metrics.

    Source: §V · p. 7

Abstract

from arXiv · show

Interactive world simulators can provide scalable environments for robot planning, policy training, and evaluation by predicting action consequences while reducing reliance on repeated physical rollouts. To serve these applications, they must generate future image sequences that respond faithfully to robot actions and preserve the dynamics of robot-object interactions over long horizons. However, existing world models typically predict the entire next latent state and often fail to capture subtle changes induced by robot actions. Such omissions can produce physically implausible outcomes, including object interpenetration and excessive deformation. To address this limitation, we propose DeltaWorld, a physically consistent interactive world simulator for robotic manipulation. Our method introduces the Delta Latent Transition Model (Delta-LTM), which predicts action-induced latent feature changes and adds them to the current latent state to obtain the next state, rather than predicting the next latent state directly. To mitigate object interpenetration and excessive deformation in predicted future frames, Interaction-aware Latent Alignment is introduced to construct counterfactual interaction regions and supervise interaction-related latent changes. DeltaWorld is evaluated on the IWS manipulation benchmark and a self-collected cross-robot dataset covering multiple robot embodiments and manipulation tasks. On the cross-robot dataset, DeltaWorld reduces FVD by 46.6% and LPIPS by 31.1% relative to the IWS baseline. These results highlight the potential of DeltaWorld for long-horizon action-conditioned video prediction in robotic manipulation.

I. INTRODUCTION

World models could reduce the costly physical rollouts needed for robot planning, policy learning, and evaluation. DeltaWorld targets the challenge of capturing small action-driven changes and introduces latent-increment prediction with interaction-aware supervision.

I. INTRODUCTION

  • Physical robot rollouts consume time and hardware resources, limiting the scale and reproducibility of policy training, action exploration, and evaluation.World models can simulate action outcomes from existing interaction data for planning, learning, and evaluation.Source: §I · p. 1
  • Models that predict whole future frames or latents can miss small action-driven object changes against mostly static scenes, with errors growing into object drift or implausible deformation.The central challenge is predicting action-caused changes, especially where robots interact with objects.Source: §I · p. 1
  • DeltaWorld predicts action-induced latent changes with Delta-LTM and adds them to the current state, while Interaction-aware Latent Alignment emphasizes interaction-related changes using counterfactual regions.The method is designed to support physically plausible interactions and stable long-horizon prediction.Source: §I · pp. 1–2
  • Evaluations on the IWS benchmark and a self-collected dataset spanning multiple robot embodiments show improvements over existing world-model baselines across image- and video-level metrics.Ablations are reported to verify the complementary roles of Delta-LTM and Interaction-aware Latent Alignment.Source: §I · p. 2

A. World Models for Robotic Manipulation

Robotic world models are shifting from visual generation toward interactive prediction of action consequences, but stable long-horizon dynamics remain challenging. The related work also describes compact latent dynamics and world models as scalable resources for policy training and evaluation.

A. World Models for Robotic Manipulation

  • Robotic world models predict environment evolution under actions for planning, control, policy steering, and evaluation, moving from compact-state models toward visual dynamics in latent representations.Two directions include leveraging visual priors for future-video generation and learning action-conditioned models from robot interaction data.Source: A. World Models for Robotic Manipulation · p. 2

B. Action-conditioned Latent Dynamics

  • Latent-dynamics methods use compact representations to predict future states, but most target future latents rather than explicitly modeling action-induced transitions in interaction-relevant regions.Examples include DINO-WM's future patch-representation prediction and IWS's action-conditioned dynamics after freezing its visual autoencoder.Source: B. Action-conditioned Latent Dynamics · p. 2

C. World Models for Policy Training and Evaluation

  • World models can support policy training through synthetic trajectories and policy optimization, while learned interactive environments can provide demonstrations without additional real-robot interaction.For policy evaluation, closed-loop world-model interaction can help preserve relative policy or checkpoint rankings while enabling scalable, reproducible screening before deployment.Source: C. World Models for Policy Training… · pp. 2–3

III. METHOD

DeltaWorld learns action-conditioned environment evolution through a two-stage latent-state framework. It predicts latent increments autoregressively, emphasizes interaction-sensitive regions during training, and decodes predicted states into observations for continued rollouts.

III. METHOD

  • Stage 1 encodes observations into compact 2D latents and trains a conditional decoder to reconstruct observations while tolerating small latent perturbations.Small Gaussian noise is injected into encoded states during decoder training; the resulting representation is intended to retain visual information and support stable dynamics modeling.Source: A. Overview · p. 3
  • With the encoder and decoder frozen, Stage 2 predicts action-induced latent increments from recent states and actions, adds each increment to the current state, and repeats the update autoregressively.Carrying largely unchanged scene features forward focuses learning on action-driven object motion and robot-object interactions.Source: A. Overview · p. 3
  • Interaction-aware Latent Alignment gives greater training weight to transitions in regions identified by comparing original latents with a counterfactual scene lacking the interacting robot arm and manipulated object.The method addresses the fact that interaction-related changes occupy a small portion of the latent representation.Source: A. Overview · p. 3
  • At inference, predicted latent states are decoded into RGB observations, which are used as context for subsequent predictions in long-horizon action-conditioned rollouts.The simulator conditions future predictions on observed latent states and a sequence of robot actions.Source: A. Overview · p. 3

B. Delta Latent Transition Model

Delta-LTM makes the action-induced change between latent states its prediction target, then applies that change through a residual state update. Its transition predictions are supervised globally and in interaction-sensitive regions.

B. Delta Latent Transition Model

  • Delta-LTM explicitly predicts the latent increment caused by robot actions rather than the complete next latent state, associating action conditioning with the state change it produces.The residual update carries persistent information from the current state into the next, focusing model capacity on action-conditioned evolution.Source: B. Delta Latent Transition Model · p. 3
  • The predicted increment is added to the current latent state to recover the next state, preserving unchanged scene information through a residual update.The overview depicts this as autoregressive action-conditioned latent prediction.Source: B. Delta Latent Transition Model · p. 3, Figure 1 · p. 4
  • The action-conditioned consistency dynamics model uses 3D convolutions, FiLM action conditioning, and spatial and temporal attention to predict latent transitions.The next latent state is recovered through the residual update.Source: B. Delta Latent Transition Model · p. 4
  • The residual between predicted and ground-truth increments supplies the basis for global dynamics supervision and subsequent interaction-aware local supervision.The overview describes interaction weighting via a mask derived from original and counterfactual scene features.Source: B. Delta Latent Transition Model · p. 4, Figure 1 · p. 4

C. Interaction-aware Latent Alignment

Interaction-aware Latent Alignment builds a mask from counterfactual latent differences to localize robot–object interaction regions. It uses that mask to apply spatially selective supervision to Delta-LTM’s latent increments.

C. Interaction-aware Latent Alignment

  • Large encoded-feature differences indicate latent locations associated with the robot–object interaction.The observations differ mainly in the manipulated object and the robot arm involved.Source: C. Interaction-aware Latent Alignment · p. 4
  • The method averages latent differences across channels, normalizes the map, and thresholds it into a binary interaction-region mask.The resulting mask localizes regions for subsequent supervision of latent state transitions.Source: C. Interaction-aware Latent Alignment · pp. 4–5
  • The interaction mask is derived by encoding the original and interaction-removed observations with the same frozen encoder and comparing their latent features.The counterfactual removes the manipulated object and robot arm involved in the interaction while preserving the remaining scene.Source: C. Interaction-aware Latent Alignment · pp. 4–5
  • The residual between ground-truth and predicted latent increments is extracted within the interaction mask for localized supervision.The spatial mask is broadcast across latent channels for element-wise masking.Source: C. Interaction-aware Latent Alignment · p. 5
  • Interaction-aware Latent Alignment applies spatially selective supervision to the same latent-increment prediction rather than introducing another dynamics target.The interaction mask focuses training on latent changes associated with robot–object interactions.Source: C. Interaction-aware Latent Alignment · pp. 4–5

D. Training and Inference

Training freezes the encoder and decoder while optimizing the dynamics model using teacher-forced latent histories and noisy contexts. At inference, predicted latent increments update the state autoregressively and are decoded into frames.

D. Training and Inference

  • After Stage 1, the encoder and decoder are frozen, and only the dynamics model is optimized.Training uses ground-truth observations for teacher-forced latent histories, with noise to improve robustness to imperfect rollout contexts.Source: D. Training and Inference · p. 5
  • The Stage 2 objective penalizes prediction residuals globally and within the interaction mask.The combined objective is also described as weighting more distant prediction steps and gradually increasing the interaction loss during training.Source: D. Training and Inference · p. 5
  • At inference, each predicted latent increment updates the current state, enters the sliding history, and is decoded into an RGB frame.The model predicts increments from a fixed-length history of latent states and the corresponding action sequence.Source: D. Training and Inference · p. 5

IV. EXPERIMENTS

The experiments assess DeltaWorld’s prediction quality and interaction consistency, cross-embodiment and cross-task effectiveness, and the contribution of its proposed components.

IV. EXPERIMENTS

  • The evaluation compares DeltaWorld with state-of-the-art world models on action-conditioned video prediction quality and long-horizon interaction consistency.These are the first of the three evaluation questions identified for the experiments.Source: §IV · p. 5
  • The experiments test whether the latent-increment formulation remains effective across robot embodiments and manipulation tasks.The cross-platform and cross-task evaluation is a separate stated experimental question.Source: §IV · p. 5
  • Ablation studies examine the contributions of Delta-LTM and Interaction-aware Latent Alignment.The paper presents these studies in Section IV-C.Source: §IV · p. 5

A. World Model Instantiation

The evaluation instantiates DeltaWorld on five IWS tasks and four additional tasks collected across three robot platforms. Models are trained and evaluated per camera view on IWS and generate autoregressive predictions of up to 150 frames.

A. World Model Instantiation

  • The IWS evaluation covers five tasks spanning rigid and deformable objects and single-arm and bimanual manipulation.Each task has approximately 500–600 episodes and two camera views; models are trained and evaluated independently for each view.Source: A. World Model Instantiation · p. 5
  • The self-collected evaluation adds four manipulation tasks across three robot platforms, with datasets of 127, 329, 886, and 893 episodes.The tasks cover different robot embodiments, action spaces, and forms of robot–object interaction.Source: A. World Model Instantiation · p. 5
  • Self-collected interaction trajectories have task-dependent variable lengths, preserving variation in duration across tasks.Evaluation starts from the first observed frame and autoregressively generates sequences of up to 150 frames.Source: A. World Model Instantiation · p. 6

B. Video Baseline Comparisons

DeltaWorld outperforms the IWS baseline on benchmark averages and on a cross-robot dataset, with qualitative predictions that better preserve interactions over longer horizons. The comparisons use consistent data splits and rollout settings across methods.

B. Video Baseline Comparisons

  • The comparison evaluates LeWorld, Cosmos 3, and IWS using FID, FVD, LPIPS, PSNR, SSIM, and MSE under shared data splits and evaluation protocols.LeWorld and IWS are trained from scratch, while Cosmos 3 is fine-tuned using its official robotic-data pipeline.Source: B. Video Baseline Comparisons · p. 6
  • On the IWS benchmark, FID falls from 44.16 to 36.04, FVD from 264.52 to 231.98, and LPIPS from 0.0661 to 0.0490 versus IWS.PSNR, SSIM, and MSE also improve; results average five tasks and two camera views.Source: Table I · p. 6, B. Video Baseline Comparisons · p. 6
  • On the cross-robot dataset, FVD drops 46.6% versus IWS, from 168.98 to 90.31, while LPIPS, PSNR, SSIM, and MSE also improve.The dataset covers four tasks across Imeta, UR5, and Franka.Source: B. Video Baseline Comparisons · p. 6, Table II · p. 7
  • Qualitative predictions more consistently preserve object structure, robot motion, and interaction relationships over longer horizons while reducing visual drift and dynamic inconsistencies.Figure 3 shows representative tasks from both the IWS benchmark and the cross-robot dataset.Source: B. Video Baseline Comparisons · p. 7

C. Ablation Study

Ablations show that both Delta-LTM and Interaction-aware Latent Alignment contribute to DeltaWorld’s performance. The full model performs best overall, with the two components targeting temporal increments and interaction-related changes, respectively.

C. Ablation Study

  • Removing Delta-LTM raises FVD from 191.50 to 228.03 and degrades LPIPS, PSNR, SSIM, and MSE; removing alignment also degrades evaluated metrics.The complete model performs best across all evaluation metrics.Source: C. Ablation Study · p. 7
  • Delta-LTM predicts action-conditioned latent increments and applies them to the current state, while Interaction-aware Latent Alignment emphasizes robot–object interaction-related changes.The components target temporal prediction and spatially focused supervision, respectively.Source: C. Ablation Study · p. 7
  • Together, the components achieve the best overall performance, supporting the importance of modeling what changes and emphasizing which changes matter.The results suggest both contribute to stable, interaction-consistent latent dynamics.Source: C. Ablation Study · p. 7

V. CONCLUSION

DeltaWorld predicts action-conditioned latent increments to update the current state and uses counterfactual supervision to emphasize interaction-sensitive regions. Evaluations across two datasets show improved action-conditioned video prediction, while ablations support the complementary roles of its components.

V. CONCLUSION

  • Delta-LTM estimates action-conditioned latent increments and applies them to update the current state rather than predicting the entire future latent state.This provides an increment-based formulation for learning robotic world dynamics.Source: §V · p. 7
  • Interaction-aware Latent Alignment emphasizes transition errors in robot–object interaction-sensitive regions through counterfactual supervision.Source: §V · p. 7
  • Experiments on the IWS benchmark and a self-collected multi-robot dataset show improved video prediction across image- and video-level metrics, with ablations confirming complementary component roles.Source: §V · p. 7

Related papers

Loading 2610.02691v1…