PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process

Jiayi Chen, Wenxuan Song, Pengxiang Ding, Ziyang Zhou, Han Zhao, Feilong Tang, Donglin Wang, Haoang Li

arXiv:2511.01718v2cs.ROcs.CV

TL;DR

VLA models must connect language and visual observations to embodied actions, but existing unified approaches either rely on external experts or separate image and action generation. This paper proposes UD-VLA with JD3P, a unified tokenized space and hybrid attention for synchronous joint denoising, and reports state-of-the-art results across multiple benchmarks with faster inference than autoregressive methods.

  • Problem

    Existing unified VLAs either rely on external experts for modality unification or separate image generation from action prediction, limiting direct synergy between the tasks.

  • Method

    UD-VLA uses JD3P to synchronously denoise image and action tokens in a unified multimodal space with hybrid attention, supported by two-stage training and inference-time techniques.

  • Results

    UD-VLA achieves state-of-the-art performance on CALVIN, LIBERO, and SimplerEnv and maintains 4× faster inference than autoregressive methods.

  • Takeaways & Limitations

    Joint denoising provides the paper’s mechanism for coupling future visual generation with action prediction across simulation and real-world evaluations.

  • Takeaways & Limitations

    Generated future images remain aligned with task-level dynamics but lack visual fidelity in fine-grained details because of limited generative pretraining and compressed images with few tokens.

Abstract

from arXiv · show

Vision-language-action (VLA) models aim to understand natural language instructions and visual observations and to execute corresponding actions as an embodied agent. Recent work integrates future images into the understanding-acting loop, yielding unified VLAs that jointly understand, generate, and act -- reading text and images and producing future images and actions. However, these models either rely on external experts for modality unification or treat image generation and action prediction as separate processes, limiting the benefits of direct synergy between these tasks. Our core philosophy is to optimize generation and action jointly through a synchronous denoising process, where the iterative refinement enables actions to evolve from initialization, under constant and sufficient visual guidance. We ground this philosophy in our proposed Unified Diffusion VLA and Joint Discrete Denoising Diffusion Process (JD3P), which is a joint diffusion process that integrates multiple modalities into a single denoising trajectory to serve as the key mechanism enabling understanding, generation, and acting to be intrinsically synergistic. Our model and theory are built on a unified tokenized space of all modalities and a hybrid attention mechanism. We further propose a two-stage training pipeline and several inference-time techniques that optimize performance and efficiency. Our approach achieves state-of-the-art performance on benchmarks such as CALVIN, LIBERO, and SimplerEnv with 4$\times$ faster inference than autoregressive methods, and we demonstrate its effectiveness through in-depth analysis and real-world evaluations. Our project page is available at https://irpn-eai.github.io/UD-VLA.github.io/.

1 INTRODUCTION

Unified Diffusion VLA addresses weak coupling between visual generation and action prediction by jointly refining both modalities in a synchronous denoising process. Its unified token space, hybrid attention, two-stage training, and test-time techniques support strong benchmark performance and faster inference.

  • 1 INTRODUCTION: Existing unified VLA designs either depend on external modality experts or keep image generation and action prediction separate, limiting alignment and visual guidance.The first paradigm introduces modular complexity and weak coupling, while the second restricts direct use of future visual information for action prediction.
  • 1 INTRODUCTION: Iterative denoising repeatedly lets action tokens attend to future image tokens, progressively refining actions under richer visual guidance than one-step autoregressive computation.This coarse-to-fine process transforms latent visual representations into temporally structured actions.
  • 1 INTRODUCTION: UD-VLA jointly refines visual and action tokens through JD3P, integrating understanding, generation, and acting into one synchronous denoising trajectory.The model uses discrete tokenization and hybrid attention to support cross-modal interaction while preserving causal conditioning across modalities.
  • 1 INTRODUCTION: The two-stage pipeline first develops image prediction in a world-model setting, then jointly trains future-image and action generation on robot datasets.The approach also adds inference-time techniques intended to balance performance and efficiency.
  • 1 INTRODUCTION: UD-VLA achieves state-of-the-art results on CALVIN, LIBERO, and SimplerEnv while maintaining 4× faster inference than autoregressive methods.The paper also reports in-depth design analysis and real-world evaluations demonstrating practical utility.

2 RELATED WORKS

Prior work explores visual planning, unified multimodal architectures, and diffusion-based action generation, but differs in how images and actions are tokenized, attended to, and decoded. UD-VLA combines discrete multimodal tokenization with joint diffusion decoding and a two-stage training design.

  • 2 RELATED WORKS: Prior visual-prediction methods decompose manipulation into visual planning and inverse dynamics, or formulate control as text-conditioned video generation.These approaches use predicted visual trajectories or videos to support downstream manipulation.
  • 2 RELATED WORKS: Joint image-action methods such as PAD and UVA use continuous diffusion or separate denoising processes, whereas UD-VLA performs unified generation through its discrete diffusion process.The comparison highlights different mechanisms for coupling future-image and action generation.
  • 2 RELATED WORKS: Existing unified VLAs use auxiliary image objectives, external encoders and decoders, unified token spaces, or causal next-token modeling to connect perception and control.Their architectures differ in whether visual and action tokens are modeled and decoded jointly or separately.
  • 2 RELATED WORKS: Discrete diffusion VLA methods iteratively recover action sequences through BART-style vocabulary substitution or BERT-style masking and prediction.These methods apply discrete denoising strategies specifically to action-token generation.

3 METHODS

UD-VLA unifies language, visual, and action information as discrete tokens and jointly denoises future images and actions. Its hybrid attention, two-stage training, and adaptive inference procedures support multimodal generation with efficient decoding.

  • Unified Tokenization: UD-VLA represents language, current and future images, and actions as concatenated discrete tokens, using VQ visual and FAST action tokenizers.Special boundary tokens mark image and action spans, while text and current-image tokens condition future-image and action outputs.
  • Hybrid Attention Mechanism: Hybrid attention gives input modalities separate causal or bidirectional patterns, enables bidirectional interaction within output blocks, and connects generation to acting causally.This design supports comprehensive within-block interactions while avoiding time-serial dependence among action tokens.
  • Joint Discrete Denoising Diffusion Process: JD3P generates future-image and action tokens in parallel through one synchronous discrete denoising trajectory with a shared mask vocabulary.Tokens are independently masked or retained through transition matrices, and the reverse process reconstructs masked positions until the original signals are recovered.
  • Training: Training uses single-step random masking and computes cross-entropy only on masked positions, with visual-token down-weighting to prevent visual loss dominance.The model is initialized from a pretrained VLM, first trained for future-image generation, then jointly optimized for image and action generation on robot data.
  • Inference: Inference initializes all future-image and action positions as masks, predicts them in parallel, and iteratively refines a shrinking masked set using confidence-guided updates.Prefix KV caching, token pre-filling, modality-restricted decoding, and deterministic action-length termination are used to improve speed and stability.

4 EXPERIMENTS

Experiments evaluate UD-VLA on simulated benchmarks, component ablations, decoding mechanisms, and real-world manipulation. The method achieves strong benchmark and real-world performance, while joint denoising, future-image generation, and hybrid attention provide measurable benefits.

  • Main Results in Simulation: 4.64 average success length on CALVIN ABCD→D, outperforming all baselines in the long-horizon manipulation benchmark.The comparison attributes this performance to explicit visual generation and multi-step diffusion facilitating information exchange between images and actions.
  • Main Results in Simulation: 92.7% average success rate on LIBERO, including 95.7% on Object and 89.6% on Long, establishing SOTA performance.UD-VLA outperforms methods based on extrinsic experts and methods that unify input and output spaces.
  • Main Results in Simulation: 59.4% average success rate on SimplerEnv-WidowX, significantly outperforming all baselines across real-to-sim manipulation tasks.The method also achieves a 24.9% higher success rate than SpatialVLA on the stack block task.
  • In-Depth Analysis: Hybrid attention reaches 4.64 average success length on CALVIN, outperforming purely bidirectional and causal attention schemes.Bidirectional attention supports global image consistency and correlations among action dimensions, while cross-modal bidirectionality can cause information leakage.
  • In-Depth Analysis: Jointly predicting future images and actions performs best because future frames provide temporal cues for anticipating visual dynamics and aligning action planning.Reconstructing current images improves fine-grained perception but remains limited to static scene information.
  • In-Depth Analysis: JD3P achieves 4.64 success length with 4.3× faster decoding, outperforming autoregressive, Jacobi, and independent-diffusion alternatives.The analysis links joint denoising to iterative action refinement using image information from intermediate denoising steps.
  • Real-World Experiment: In real-world tasks, UD-VLA exceeds 80% success across stacking bowls, putting blocks, and flipping towers, outperforming GR00T N1 and UniVLA.The evaluation includes seen and unseen scenes, objects, targets, and backgrounds.
  • Real-World Experiment: Generated future frames follow instructions and capture task-level dynamics while remaining aligned with ground-truth trajectories across simulation and real-world embodiments.These visualizations support the model’s temporal understanding of task logic, while also illustrating the scope and limitations of image generation.

5 CONCLUSION

The paper presents a unified diffusion VLA that jointly refines image and action tokens through a synchronous discrete denoising process, supported by unified multimodal representations and hybrid attention. Experiments report state-of-the-art results in simulation and real-world environments while balancing performance and efficiency.

  • The unified diffusion VLA jointly refines image and action tokens through a single synchronous denoising trajectory.
  • Unified multimodal space, hybrid attention, two-stage training, and test-time techniques support the model’s performance-efficiency trade-off.

REPRODUCIBILITY STATEMENT

The paper documents its experimental setup, evaluation protocols, and implementation details to support re-implementation and verification. It uses publicly available datasets and follows official benchmark splits.

  • The paper details data, tokenization, architectures, training schedules, decoding methods, and selection criteria for reproducible implementation.
  • Evaluations follow official benchmark splits, and all datasets used are publicly available for consistent verification.

A LLM USAGE STATEMENT

Large language models were used only for proofreading and minor language refinement. The authors state that the research ideas, methods, experiments, analyses, code, and substantive writing were authored and verified by them.

  • Large language models were used only for grammar, phrasing, and minor style refinement.
  • The authors state that they conceived and authored the research ideas, methodology, experiments, analyses, code, and substantive writing.

B VISUALIZATION OF FUTURE IMAGE GENERATION

Generated future images follow instructions, capture task-level dynamics, and remain aligned with ground-truth trajectories across simulated and real-world embodiments. Their fine-grained visual fidelity is limited, but they remain informative for downstream action planning.

  • Generated future frames follow instructions and capture task-level dynamics across simulation and real-world embodiments.
  • The images lack fine-grained fidelity in robotic arms and backgrounds because of limited generative pretraining and compressed image tokens.
  • Despite lower pixel-level accuracy, the generated frames convey task progression and remain sufficient for action planning.

C REAL-WORLD SETUP

The real-world setup uses a dexterous robotic arm-hand system with close-range and workspace-wide RGB-D observation. Experiments include representative trajectories and precise-manipulation tasks involving a toy approximately one-fifth the size of other objects.

  • Hardware and sensing: The platform combines a 6-DoF UR5e arm, a 6-DoF Inspire RH56E2 hand, and two cameras for close-range and workspace-wide observations.The wrist-mounted Intel RealSense D435i provides close-range RGB-D views, while a static Gemini 336L records the full workspace.
  • Task visualization: Figure 6 visualizes representative trajectories for each real-world task.
  • Task visualization: Figure 7 focuses on precise manipulation involving a toy with an approximately 1.5 cm radius.The toy is about one-fifth the size of the other objects.

D REAL-WORLD TASKS

The real-world evaluation covers three manipulation tasks, while the loss formulations describe reconstruction, autoregressive, continuous-diffusion, and discrete-diffusion objectives. These objectives respectively support prediction, next-token modeling, iterative denoising, and masked-token recovery.

  • Task Definition: The three real-world tasks are stacking bowls, putting blocks into a box, and flipping towers by grasping and moving them until folded.
  • Loss Formulations: LMSE measures average squared discrepancy between predicted outputs and ground-truth targets for visual and action modalities.The target may be conditioned on instructions or proprioception over an index set of computed elements.
  • Loss Formulations: LDiff-cont predicts injected noise in a noised continuous sample and uses reverse updates to reconstruct the clean target.The objective may use L2 or Smooth-L1 error and can be conditioned on signals such as instructions or proprioception.
  • Loss Formulations: LDiff-disc trains masked-token prediction over discrete sequences containing a special [MASK] token.The model learns to recover ground-truth tokens at masked positions from the remaining sequence context.
  • Loss Formulations: LNTP is a causal next-token objective that maximizes the probability of each token given its preceding context.

F TRAINING DETAILS.

Training for the virtual-environment and real-world experiments used H100 GPUs, with dataset-specific action chunks and training durations.

  • Training Details: The Calvin-ABCD and LIBERO models used action chunks of 10 and trained for about 24 and 30 hours, respectively, across 8 H100 GPUs.LIBERO was jointly trained on four tasks rather than training each task separately.
  • Training Details: SimplerEnv training used the Bridge dataset, while the passage also states that real-world experiments were trained on H100 GPUs.

G BASELINES

The baselines span language-conditioned visuomotor policies, unified vision-action models, visual-prediction methods, and autoregressive world models. They differ in representation, visual planning, action generation, and attention mechanisms.

  • Language-conditioned policies: MCIL, RT-1, and RoboFlamingo represent language-conditioned manipulation through end-to-end visuomotor learning, a single Transformer policy, and decoupled VLM-policy components.These baselines differ in training scale and how visual-language understanding connects to decision making.
  • Generalist VLAs: GR-1 predicts robot actions and future images end-to-end after video-prediction pretraining, while OpenVLA uses a visually conditioned Llama-2 backbone with DINOv2 and SigLIP features.
  • Generalist VLAs: UniVLA plans in a task-centric latent action space extracted from videos, enabling heterogeneous data use and lightweight trajectory decoding.The method reports improvements over OpenVLA with a fraction of the pretraining compute and downstream data.
  • Generalist VLAs: Octo maps multimodal observation and task tokens to actions, while SpatialVLA aligns 3D observations and actions using egocentric position encoding and adaptive action grids.
  • Visual prediction and planning: CoT-VLA generates subgoal images as visual chain-of-thought steps before conditioning actions on observations and generated subgoals.
  • World and action models: WorldVLA unifies image, text, and action generation with separate tokenizers sharing one vocabulary, using visual tokens to support subsequent action generation.Its action-attention masking addresses error propagation and limited action generalization during chunked autoregressive generation.
  • Visual prediction and planning: FlowVLA decomposes next-frame prediction into motion and appearance through an intermediate optical-flow target.Optical flows and camera frames share a VQ tokenizer within one autoregressive Transformer.
  • Visual prediction and planning: DreamVLA predicts dynamic regions, depth maps, and semantic features instead of full frames to provide compact guidance for inverse dynamics.Its block-wise structured attention masks cross-type interactions.
Loading 2511.01718v2…