PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Model

John Won, Kyungmin Lee, Huiwon Jang, Dongyoung Kim, Jinwoo Shin

arXiv:2510.27607v3cs.CVcs.RO

TL;DR

VLAs lack explicit physical-process modeling, making joint prediction of actions and future states difficult because the modalities differ substantially. DUST uses separate, cross-attending modality streams with independent noise and decoupled flow matching, plus asynchronous sampling. It reports gains across simulated and real-world benchmarks and supports transfer learning through video pretraining and heterogeneous joint training.

  • Problem

    VLAs often lack an explicit grasp of how actions transform the environment, while jointly modeling actions and future observations faces a modality gap.

  • Method

    DUST uses a multimodal diffusion transformer with separate action and observation streams, independent modality noise, decoupled flow matching, and asynchronous joint sampling.

  • Results

    DUST consistently outperforms baselines on RoboCasa, GR-1, and Franka Research 3, with average improvements of 18% over GR00T-N1.5 and 5% over FLARE on RoboCasa, and 13% over GR00T-N1.5 and 10.4% over FLARE on Franka Research 3.

  • Takeaways & Limitations

    DUST supports scalable test-time refinement and successful transfer learning through action-free video pretraining and heterogeneous joint training.

Abstract

from arXiv · show

Augmenting vision-language-action models (VLAs) with world models is promising for robotic policy learning but faces challenges in jointly predicting states and actions due to the modality gap. To address this, we propose DUal-STream diffusion (DUST), a world-model augmented VLA framework featuring a multimodal diffusion transformer that maintains separate modality streams while enabling cross-modal knowledge sharing. In addition, DUST utilizes independent noise perturbations and a decoupled flow matching loss to learn cross-modal causal relationships. We further introduce an asynchronous sampling method for action and vision tokens that enhances performance through inference-time scaling. Experimental results on simulated benchmarks like RoboCasa and GR-1 show that DUST achieves up to 6% gains over state-of-the-art VLA and world-modeling baselines, with inference-time scaling providing an additional 2-5% improvement. In real-world tasks using the Franka Research 3, DUST outperforms baselines by 10% in success rate. Finally, we demonstrate that DUST enables effective transfer learning through both pretraining on action-free videos and joint-training with heterogeneous robot and human datasets.

1. Introduction

VLAs provide strong perceptual grounding and instruction following but often lack explicit modeling of how actions transform the environment. DUST addresses the modality conflict in joint action–observation prediction with separate yet cross-attending streams, decoupled training, and asynchronous sampling.

  • VLAs generalize across objects, scenes, and instructions but often struggle to model how actions transform the environment.
  • Unified diffusion places action and visual tokens in one latent space, creating a mismatch between smooth action trajectories and complex visual data.
  • DUST maintains distinct action and observation streams while enabling cross-modal exchange through shared attention in a multimodal diffusion transformer.
  • Independent modality noise and modality-specific flow-matching losses train cross-modal causal relationships according to each modality’s statistical structure.
  • DUST outperforms standard VLA and world-modeling baselines on RoboCasa and GR-1, while asynchronous joint sampling adds further test-time gains.

2. Related Works

Related work augments VLAs with world-modeling objectives, differing in how action–future-state generation is architected and how future states are represented. These approaches range from unified or unidirectional diffusion systems to video-generation-backed policies and embedding-based targets.

  • VLAs adapt internet-scale vision-language models for robotics through autoregressive action generation or diffusion-based action experts.
  • Prior world-modeling methods jointly generate future images and actions through unified diffusion architectures such as PAD and EnerVerse.
  • Other approaches use disjoint architectures with only unidirectional conditioning, including sequential visual-then-action denoising pipelines.
  • World action models repurpose large video-generation backbones for joint video-and-action prediction, whereas DUST augments a VLA with world-modeling objectives.
  • Future states may be represented by directly reconstructed RGB observations or by embeddings from pretrained encoders such as DINO-V2 and Q-Former.

3. Preliminaries

The preliminaries define expert trajectories, chunked action prediction, and the diffusion-based VLA baseline. World-modeling predicts future observation representations rather than pixels so the policy can focus on semantically relevant environmental dynamics.

  • Each demonstration trajectory contains a task instruction, visual and proprioceptive observations, and chunked action sequences; the goal is to predict actions from observations and instructions.
  • A pretrained VLM extracts semantic information from images and instructions, which conditions a diffusion-transformer action expert through cross-attention.
  • Flow matching trains the action expert to predict the velocity field of noisy action sequences formed by interpolating demonstrations with Gaussian noise.
  • World-modeling: World-modeling augments direct observation-to-action policies with prediction of future observations after executing an action chunk.
  • World-modeling: Predicting pixels can waste capacity on textures, lighting, and clutter, whereas learned future-image embeddings capture semantic and structural scene content.
  • DUST architecture: The DUST architecture conditions its diffusion model on VLM representations to generate actions and future observation embeddings.

4. Method

DUST jointly models actions and future observations by preserving separate modality pathways while exchanging information through shared attention. Independent modality noise, decoupled flow-matching losses, and asynchronous sampling address their differing statistical and computational requirements.

  • 4.1. DUST architecture: DUST uses a multimodal diffusion transformer with separate action and vision streams that exchange information through shared cross-modal attention.Each stream retains its own pathway and timestep embedding, while concatenation occurs only in the shared attention layer.
  • 4.2. Joint training algorithm: Independent modality noise exposes the model to varied corruption combinations, training bidirectional dependencies between actions and future observations.Clean observations with noised actions supervise inverse dynamics, while clean actions with noised observations supervise forward dynamics.
  • 4.2. Joint training algorithm: The joint objective sums modality-specific action and world-modeling flow-matching losses, weighted by λWM.Independent noise injection permits decomposition into separate unimodal diffusion losses.
  • 4.3. Vision-action joint sampling: Asynchronous inference updates high-dimensional vision tokens more frequently than low-dimensional action tokens, matching their different denoising requirements.Vision uses No = q × NA steps, whereas actions update every q vision steps with stride ∆τA = 1/NA.
  • 4.3. Vision-action joint sampling: Increasing q provides test-time scaling by trading inference speed for predictive accuracy, while the main experiments use q = 1 and No = NA = 4.The default setting keeps action and vision step counts equal for fair baseline comparison.

5. Experiments

DUST is evaluated across simulated, real-world, transfer-learning, test-time-scaling, efficiency, and ablation settings. It consistently improves task success over VLA and world-modeling baselines while supporting heterogeneous-data transfer and configurable inference cost.

  • 5.1. Main results: 18% over GR00T-N1.5 and 5% over FLARE on RoboCasa with 100 demonstrations per task, with the advantage persisting at larger demonstration scales.On GR-1, DUST also surpasses all baselines at 300 and 1,000 demonstrations across both task categories.
  • 5.1. Main results: 13% over GR00T-N1.5 and 10.4% over FLARE on average across seven Franka Research 3 tasks, with DUST highest on every task.The tasks span pick-and-place, cord insertion, and tool-mediated actions.
  • 5.2. Transfer learning: 64.4% versus 58.5% average success after adding GR-1 and EgoDex data, with DUST showing the largest joint-training gains.The evaluation uses heterogeneous robot and human data mixtures.
  • 5.3. Test-time scaling for joint sampling: 2–3% gains on RoboCasa at 64 vision denoising steps and a 5% gain on GR-1 at 32 steps, while additional steps increase inference time.DUST already outperforms GR00T-N1.5 and FLARE with four steps at approximately 40Hz; 32 steps maintain approximately 8Hz.
  • 5.4. Ablation study: Removing the dual-stream structure drops performance 8%, while removing decoupled noise drops it 12% on RoboCasa.The two components provide complementary gains through structured cross-modal learning and modality-specific causal modeling.
  • 5.4. Ablation study: Performance remains stable for λWM from 0.5 to 2.0 but degrades outside that range, and 12 MMDiT layers produce the best ablation outcome.These settings balance world-modeling and action-modeling objectives while retaining cross-modal processing.

6. Conclusion

DUST is a world-model-augmented VLA that maintains separate, attention-linked action and observation streams. Decoupled training, asynchronous sampling, and transfer-learning setups support gains across simulation, real-world tasks, and heterogeneous data.

  • DUST maintains separate, attention-linked token streams for actions and observations to resolve modality conflicts.
  • Decoupled training with independent noising captures causal dependencies between action and observation modalities.
  • Asynchronous joint sampling enables scalable test-time refinement.
  • Action-free video pretraining and heterogeneous joint training demonstrate successful transfer learning.

A.1. Additional simulation environments

The appendix reports additional comparisons on LIBERO and CALVIN ABC-D against multiple VLA and world-modeling baselines. These evaluations use results from prior papers and, for CALVIN, fair-comparison settings without large-scale pretraining.

  • LIBERO results compare DUST with DreamVLA, WorldVLA, FlowVLA, CoT-VLA, UD-VLA, π0, π0-FAST, GR00T-N1.5, and FLARE.
  • CALVIN ABC-D results compare DUST with UP-VLA, Seer, MDT, GR00T-N1.5, and FLARE.
  • CALVIN comparisons use ablation results for UP-VLA and Seer that exclude large-scale pretraining to match DUST’s evaluation setting.

A.2. Test-time scaling of naive joint sampling

Naively increasing diffusion steps for both modalities performs worse than decoupling vision and action denoising schedules. This supports asynchronous sampling as the key test-time-scaling design choice.

  • Synchronous joint sampling increases vision and action diffusion steps together rather than decoupling their step counts.
  • DUST’s modified MMDiT uses separate timestep embeddings as conditions for each modality.
  • Without decoupling the modality step counts, simply increasing diffusion steps deteriorates performance.The result supports the hypothesis that vision tokens need more diffusion steps than action tokens.

A.3. Implementation and Training Details

DUST builds on the GR00T-N1.5 codebase with modality-specific encoders, decoders, positional encodings, and timestep conditioning. Training uses shared optimization settings but varies compute and batch configurations across datasets and transfer setups.

  • The architecture uses a pretrained Eagle-2 VLM, MLP-based vision encoder and decoder, and linear action encoder-decoder with modality-specific positional encodings.
  • Each modality stream receives conditioning timestep embeddings from an independent source rather than a global timestep embedding.This is a modification of the original MMDiT design.
  • Baselines follow released configurations where available, while FLARE is reimplemented with a shared world-modeling setup because official code and checkpoints are unavailable.
  • RoboCasa training uses 32 global batch samples on 2 A100 GPUs, with convergence ranging from 60k to 600k steps across data scales.
  • GR-1 training uses a 960-sample global batch on 8 H200 GPUs for 60k steps and is sensitive to batch size.
  • Transfer setups include action-free BridgeV2 video pretraining followed by RoboCasa finetuning, plus joint training on RoboCasa, GR-1, and EgoDex trajectories.

A.4. Simulation Benchmarks

The simulation evaluation uses RoboCasa for single-arm kitchen manipulation and GR-1 for humanoid tabletop dexterity.

  • RoboCasa covers 24 household manipulation tasks, including pick-and-place, contraption opening or closing, and miscellaneous activities.The benchmark includes 8 pick-and-place, 6 contraption open/close, and 10 miscellaneous tasks.
  • GR-1 covers 24 dexterous tabletop tasks, combining pick-and-place with articulated tasks that require closing containers.The benchmark contains 16 pick-and-place and 8 articulated tasks.

A.5. Real-World Experiment Details

The real-world evaluation uses a Franka Research 3 across pick-and-place, tool-use, and insertion tasks, with structured trial scoring and analyses of cross-modal guidance, low-latency inference, and failure conditions.

  • Task suite: The real-world suite contains 7 tasks: 4 pick-and-place, 2 tool-using, and 1 insertion task.The setup uses a Franka Research 3 robot with two ZED cameras, one wrist-mounted and one side-facing.
  • Pick-and-place tasks: Pick-and-place evaluation uses four object categories across varied source-target configurations, with partial placement counted as 0.5 successes.Each object-task configuration receives 6 evaluations, totaling 24 trials per task.
  • Insertion task: The insertion task requires picking up a yellow charger and plugging it into a socket, and it has no partial score.The overall evaluation uses 4 configurations with 6 evaluations each, totaling 24 trials.
  • Tool using tasks: Tool-use tasks evaluate erasing board markings and moving six bolts into a dustpan with task-specific partial-credit rules.Erasing earns half credit above 50% removal and full credit above 90%; the brush task awards 1/6 point per bolt.
  • Cross-modal analysis: Action accuracy decreases monotonically as noise on ground-truth future states increases, supporting learned dependency between future observations and actions.Table 12 reports mean squared error of predicted actions under varying future-state noise levels.
  • Inference-time analysis: 47.9% success with zero vision denoising steps exceeds GR00T-N1.5 at 41.7% and FLARE at 44.6%, despite a slight drop from the 4-step default.This provides an ultra-low-latency inference mode without explicit visual reconstruction at test time.
  • Failure analysis: Visual occlusion can undermine anticipatory control, with Task 4 especially affected by self-occlusion from the arm or gripper.The reported rollout had a partially obstructed side camera and a wrist camera that did not view the target objects.
  • Failure analysis: Suggested mitigations include additional camera views, stronger proprioceptive priors, and temporal history, while the current formulation primarily uses the current state.
Loading 2510.27607v3…