Source-linked AI summary
MMaDA-VLA: Large Diffusion Vision-Language-Action Model with Unified Multi-Modal Instruction and Generation
Yang Liu, Pengxiang Ding, Tengyue Jiang, Xudong Wang, Wenxuan Song, Minghui Lin, Han Zhao, Hongyin Zhang, Zifeng Zhuang, Wei Zhao, Siteng Huang, Jinkui Shi, Donglin Wang
TL;DR
Existing VLA paradigms face architectural overhead, long-horizon error accumulation, and limited explicit dynamics modeling. MMaDA-VLA addresses these issues with a native discrete diffusion VLA that jointly denoises goal observations and action chunks in a shared token space, achieving strong simulation and real-world results.
Problem
Hierarchical and autoregressive VLA paradigms can add architectural overhead, accumulate long-horizon errors, and generally lack an explicit mechanism for modeling environment dynamics.
Method
MMaDA-VLA uses a shared discrete token space and masked-token denoising to jointly generate a future goal observation and an action chunk in parallel.
Results
MMaDA-VLA achieves state-of-the-art performance across simulation benchmarks and real-world tasks, including a 98.0% average success rate on LIBERO.
Takeaways & Limitations
Joint iterative refinement of visual goals and actions provides a unified framework for multimodal instruction following and generation with strong generalization.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models map visual observations and natural-language instructions to robot actions; however, hierarchical and autoregressive paradigms often incur architectural overhead, accumulate long-horizon errors, and require auxiliary modules to capture environment dynamics. To this end, we present MMaDA-VLA, a fully native, pretrained discrete diffusion VLA that unifies multi-modal understanding and generation. Specifically, MMaDA-VLA uses a shared discrete token space to jointly denoise a future goal observation and an action chunk, grounding actions in predicted visual outcomes without an auxiliary world model. In this way, parallel, order-free refinement improves long-horizon consistency. Extensive experiments and comprehensive analyses demonstrate that MMaDA-VLA achieves an average success rate of 98.0\% on LIBERO and an average successful sequence length of 4.78 on CALVIN, while performing strongly in real-world settings. The project page is available at https://yliu-cs.github.io/MMaDA-VLA.
1 Introduction
MMaDA-VLA addresses architectural complexity, weak temporal consistency, and limited dynamics modeling in conventional VLA paradigms with a native discrete diffusion design. It unifies multimodal tokens and jointly generates future observations and action chunks through parallel denoising.
- Motivation: Conventional VLAs use hierarchical policy heads or autoregressive action generation, creating architectural complexity, training costs, or long-horizon error accumulation.Hierarchical designs may lose information across module boundaries, while action-discretization methods can weaken temporal consistency and global trajectory planning.
- MMaDA-VLA: MMaDA-VLA represents language, images, and continuous robot controls in one discrete token space with a shared masked-token denoising objective.The model is trained end-to-end to jointly generate a future goal observation and an action chunk in parallel.
- MMaDA-VLA: Parallel, order-free iterative denoising repeatedly aligns action chunks with predicted future visual outcomes without auxiliary world models.This design supports global refinement and an implicit representation of task-relevant state evolution.
- Training and evaluation: MMaDA-VLA combines large-scale cross-embodiment pre-training with task-specific fine-tuning in an end-to-end training pipeline.The authors report extensive simulation and real-world experiments against prior state-of-the-art VLAs.
2 Related Work
Related work spans discrete and hierarchical VLA models, diffusion-style generation for discrete tokens, and world models that predict future observations for action selection. These directions motivate unified approaches that connect visual generation, dynamics modeling, and action prediction.
- Discrete diffusion: Autoregressive language generation can hinder bidirectional reasoning and cause systematic failures on reversal tasks.These limitations motivate adapting diffusion-style generation to discrete language tokens.
- Discrete diffusion: Diffusion models have demonstrated scalability and sample fidelity in continuous domains such as images and audio, motivating discrete-token adaptations.These adaptations seek alternatives to left-to-right factorization for language generation.
- VLA paradigms: Discrete VLA models extend VLM vocabularies to generate action tokens, whereas hierarchical continuous VLAs add policy heads or action experts.The two families differ in whether action prediction is cast as token generation or delegated to an added continuous-control module.
- World models: World-model research predicts future observations to model environment dynamics and uses them for action selection through inverse dynamics prediction.Recent joint frameworks generate future frames with corresponding actions to improve temporal consistency and policy learning.
3 Methodology
MMaDA-VLA unifies multimodal instruction following and generation in a native discrete diffusion VLA, jointly predicting future observations and action chunks. Its shared tokenization, hybrid attention, masked-token training, and iterative denoising support parallel action refinement in a closed-loop control pipeline.
- Multi-Modal Sequence Modeling: The model jointly predicts a future goal observation and an action chunk to introduce task-relevant state evolution into action generation.The goal observation is at t′ = t + k, where k is the action chunk size.
- Data Tokenization: MMaDA-VLA represents language, images, and continuous robot actions in one discrete token space with a shared masked-token objective.It uses text, image, and action tokenizers, with each action dimension discretized into 256 bins.
- Hybrid Attention Mechanism: Hybrid attention applies bidirectional full attention within modalities and causal attention across modalities, decoupling the two generation targets.This enables global within-modality exchange while enforcing directed cross-modal information flow.
- Learning Objective: Masked-token diffusion training corrupts generation tokens at sampled diffusion steps and learns a single objective across the multimodal sequence.Tokens are replaced with [M] according to a cosine mask schedule during pre-training and fine-tuning.
- Iterative Denoising: At inference, MMaDA-VLA iteratively denoises masked goal-observation and action tokens, decodes them into outputs, and executes actions in a closed loop.Confidence-based remasking updates intermediate states across D denoising steps before tokenizers recover the observation and action sequences.
- Key-Value Cache: A training-free key-value cache reuses stable instruction representations and reduces repeated computation during iterative denoising.The generation representations evolve sparsely across denoising steps, while the instruction part remains fixed.
4 Experiments
MMaDA-VLA is evaluated on LIBERO, CALVIN, and real-world manipulation tasks, covering multi-task generalization, long-horizon execution, and robustness to environmental changes. It achieves the strongest reported performance across both simulation benchmarks and all four real-world tasks.
- Evaluation settings: The evaluation spans LIBERO, CALVIN, and real-world tasks to assess multi-task generalization and long-horizon performance.LIBERO includes spatial, object, goal, and long-horizon suites; CALVIN evaluates five-subtask sequences.
- Evaluation settings: The real-world setup uses an AgileX PiPER 6-DoF arm with a 1-DoF gripper, third-person RGB observations, and wrist-view observations.The robot is equipped with a RealSense D435 and a gripper-mounted DX200-2.8mm camera.
- Evaluation settings: The model is pretrained on large-scale cross-embodiment manipulation data and uses vertically concatenated third-person and wrist-view images with textual proprioceptive state.MMaDA-VLA uses MMaDA-8B-Base as its 8-billion-parameter backbone.
- Main results: MMaDA-VLA achieves the highest reported performance among compared methods on both simulation benchmarks and all four real-world tasks.The comparison is reported in Table 1 and Figure 3.
- Main results: 98.0% average success rate on LIBERO surpasses VLA-Adapter’s 97.3% by 0.7 percentage points.The result is associated with improved generalization of goal prediction and action generation.
- Main results: 4.78 average successful execution length on CALVIN exceeds DreamVLA’s 4.44 by 0.34, while fifth-task success reaches 89.7% versus 78.1%.The fifth-task improvement is 11.6 percentage points.
- Main results: 83.3%–93.3% success across four real-world tasks exceeds GR00T N1.6 by 23.3–26.6 percentage points.Reported failures mainly involve imprecise grasps, insufficient drawer displacement, and unstable tall-cup stacking, with occasional recovery through corrective actions.
5 Analysis
The analysis includes CALVIN ablations that remove world-model prediction or parallel denoising, alongside an assessment of pre-training effectiveness. These studies examine the roles of MMaDA-VLA’s design components and pre-training.
- Ablation studies: CALVIN ablations compare the full model with variants without goal-image prediction, without parallel denoising, or with causal attention.All ablations are evaluated without pre-training and compare models fine-tuned from scratch.
- Ablation studies: The ablation study is designed to isolate the effects of world-model prediction, parallel denoising, and attention design on CALVIN performance.The comparison uses MMaDA-VLA fine-tuned from scratch as the reference configuration.
- Pre-training analysis: Table 5 evaluates the effectiveness of pre-training.The supplied passage identifies the evaluation but does not report its numerical findings.
Method LIBERO CALVIN
The analyses examine ablations, pre-training, generated goal observations, and inference settings, showing benefits from dynamics modeling, large-scale pre-training, and calibrated iterative denoising. Generated observations remain broadly aligned with task dynamics, while cache refresh improves speed with performance trade-offs.
- Ablation analysis: Removing environment-dynamics modeling reduces the average execution length by 0.48.
- Pre-training: Pre-training increases LIBERO multi-task generalization from 94.5% to 98.0% and CALVIN average execution length from 4.56 to 4.78.These gains are attributed to large-scale, diverse data spanning cross-embodiment robot manipulation skills and egocentric human demonstrations.
- Goal-observation generation: Generated goal images generally match instructions, preserve high-level task dynamics, and remain broadly aligned with ground-truth trajectories.Figure 5 visualizes composite frames combining third-person and wrist-mounted views at each timestep.
- Inference efficiency: Key-value caching reduces average inference latency from 2.0 s to 1.4 s under identical settings.
- Inference efficiency: Performance peaks at 24 denoising steps with an average length of 4.65, while additional cache-refresh steps improve speed slightly but cause considerable performance fluctuations.The selected configuration uses 24 denoising steps and 6 cache-refresh steps to balance speed and performance.
6 Conclusion
MMaDA-VLA unifies multimodal understanding and generation through a fully native discrete diffusion VLA. With cross-embodiment pre-training and downstream fine-tuning, it achieves strong simulation and real-world performance, accurate execution, and generalization.
- MMaDA-VLA is a fully native discrete diffusion VLA that unifies multimodal understanding and generation.
- Iterative denoising jointly refines goal observations and action tokens within a unified training and inference framework.
- Cross-embodiment pre-training followed by downstream fine-tuning supports strong performance on simulation benchmarks and real-world tasks.