PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoning

Junjie Wen, Minjie Zhu, Yichen Zhu, Zhibin Tang, Jinming Li, Zhongyi Zhou, Chengmeng Li, Xiaoyu Liu, Yaxin Peng, Chaomin Shen, Feifei Feng

arXiv:2412.03293v3cs.ROcs.CV

TL;DR

Robot VLAs must combine reasoning with precise, robust, and efficient action generation. DiffusionVLA addresses this by joining autoregressive reasoning, diffusion control, and direct reasoning injection, achieving broad real-world generalization and fast inference. Its main boundaries are degraded low-precision performance and reduced grasping precision for novel instructions.

  • Problem

    Existing autoregressive VLAs have limitations in precise, coherent, and efficient action generation, while diffusion policies lack reasoning capabilities for complex tasks.

    Source: §1 · p. 2

  • Method

    DiffusionVLA combines next-token reasoning from a pretrained VLM with diffusion-based action generation and FiLM-based reasoning injection.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    DiVLA demonstrates generalization across tasks, environments, instructions, and embodiments, with performance improving as model size scales from 2B to 72B parameters.

    Source: §1 · p. 3, §5 · p. 8

  • Takeaways & Limitations

    The unified framework supports interpretable robot actions, failure analysis, visual generalization, conversational interaction, and real-time inference.

    Source: §1 · p. 3, §5 · p. 8

    Performance degrades substantially at 8-bit and 4-bit precision, and grasping precision decreases for novel instructions.

    Source: §8.2 · p. 14, §8.3 · p. 15

Abstract

from arXiv · show

In this paper, we present DiffusionVLA, a novel framework that seamlessly combines the autoregression model with the diffusion model for learning visuomotor policy. Central to our approach is a next-token prediction objective, enabling the model to reason effectively over the user's query in the context of current observations. Subsequently, a diffusion model is attached to generate robust action outputs. To enhance policy learning through self-reasoning, we introduce a novel reasoning injection module that integrates reasoning phrases directly into the policy learning process. The whole framework is simple and flexible, making it easy to deploy and upgrade. We conduct extensive experiments using multiple real robots to validate the effectiveness of DiffusionVLA. Our tests include a challenging factory sorting task, where DiffusionVLA successfully categorizes objects, including those not seen during training. We observe that the reasoning module makes the model interpretable. It allows observers to understand the model thought process and identify potential causes of policy failures. Additionally, we test DiffusionVLA on a zero-shot bin-picking task, achieving 63.7\% accuracy on 102 previously unseen objects. Our method demonstrates robustness to visual changes, such as distractors and new backgrounds, and easily adapts to new embodiments. Furthermore, DiffusionVLA can follow novel instructions and retain conversational ability. Notably, DiffusionVLA is data-efficient and fast at inference; our smallest DiffusionVLA-2B runs 82Hz on a single A6000 GPU and can train from scratch on less than 50 demonstrations for a complex task. Finally, we scale the model from 2B to 72B parameters, showcasing improved generalization capabilities with increased model size.

1. Introduction

Existing VLA approaches trade off precise action generation and efficient control against reasoning capability. DiffusionVLA unifies autoregressive reasoning with diffusion-based actions through reasoning injection, supporting generalization, interpretability, adaptability, fast inference, and scaling.

  • Next-token VLA models can disrupt continuous-action coherence and precision through discretization, while also being inefficient for real-time action generation.Source: §1 · p. 2
  • Diffusion policies generate multimodal action sequences efficiently but lack the reasoning capabilities needed for complex robotic tasks.Source: §1 · p. 2
  • DiffusionVLA combines a pretrained VLM’s autoregressive reasoning with a diffusion model for robust robotic action generation.Source: §1 · p. 2
  • Reasoning injection embeds self-generated reasoning into policy learning, helping connect logical reasoning with executable actions.Source: §1 · p. 2
  • The end-to-end framework generalizes to novel objects and provides interpretable actions, remains unaffected by visual distractions or novel backgrounds, and scales from 2B to 72B with improved generalization and performance.Source: §1 · p. 3

2. Related Works

Related work spans autoregressive VLAs, diffusion policies, robot foundation models, and unified multimodal generators. DiffusionVLA differs by combining next-token reasoning with diffusion-based robotic action generation.

  • RT-2 and OpenVLA apply next-token prediction to robot learning, but discretized actions and embodiment adaptation limit this approach for robotic control.Source: §2 · p. 3
  • Diffusion Policy and subsequent variants model multimodal robotic actions through diffusion, while TinyVLA and π0 explore lightweight or flow-matching alternatives.Source: §2 · p. 3
  • Robot foundation models commonly separate multimodal understanding from embodied control or fine-tune pretrained VLMs directly on robotic data.Source: §2 · p. 3
  • Unlike prior unified multimodal generators, DiffusionVLA combines next-token prediction with diffusion specifically to add reasoning to vision-language-action models.Source: §2 · p. 3

3. Methodology

DiffusionVLA integrates visual-language reasoning, diffusion action generation, and direct reasoning injection in a flexible architecture. Its training combines diffusion and next-token objectives with scalable pretrained data and embodiment-specific adaptation.

3. Methodology

  • The methodology targets an efficient architecture that integrates autoregressive reasoning with diffusion action generation while avoiding additional inference computation overhead.Source: §3 · p. 3

3.1. Architecture

  • Visual inputs are encoded with SigLIP across camera views, transformed into visual embeddings, and concatenated before vision-language processing.Source: §3.1 · pp. 3–4
  • Factory-sorting evaluation uses four object categories, real-robot multi-camera setups, and multi-task evaluation includes in-distribution and out-of-distribution success rates.Source: Figure 2 · p. 4, Table 1 · p. 4
  • Qwen2-VL provides a replaceable pretrained vision-language backbone, while projected action tokens connect its outputs to a Diffusion Policy action decoder.Source: §3.1 · p. 4
  • Figure 3 compares DiVLA with Diffusion Policy, Octo, TinyVLA, and OpenVLA on factory sorting.Source: Figure 3 · p. 5
  • The reasoning injection module uses FiLM to inject the reasoning component’s final token embedding into policy layers, adding contextual reasoning without dominating action-specific processing.Source: §3.1 · p. 5

3.2. Model Design Choices

  • Training minimizes a combined loss, L = Ldiff + αLntp, balancing diffusion action learning with next-token prediction; α is typically set to 10.Source: §3.2 · p. 5
  • Pretraining uses Droid for DiVLA-2B and DiVLA-7B, and OXE plus Droid for DiVLA-72B, with GPT-4o-generated reasoning added to robotic data.Source: §3.2 · p. 5

4. Experiments

DiVLA combines multi-task robot evaluations with tests of visual, object, reasoning, and embodiment generalization. Across these settings, it maintains strong performance, including 63.7% success on 102 unseen objects and up to 70.8% on mixed seen–unseen bimanual tasks.

4. Experiments

  • The evaluation uses factory sorting, bin picking, multi-task learning, and table bussing across Franka and bimanual AgileX robots.The experiments include both single-arm and bimanual real-world settings.Source: §4 · pp. 5–6, §4.1 · p. 6

4.2. Real-World Multi-Task Learning

  • Across five tasks, DiVLA maintains the highest average success rate under distractors, changed backgrounds, and colorful lighting, without visual data augmentation.All methods decline under these visual changes, but DiVLA remains the most robust.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3. End-to-End Sorting on Real Robot

  • DiVLA achieves a 66.2% average success rate in factory sorting and maintains 60% success in highly cluttered mixed scenarios.Diffusion Policy declines to 9.2% in the highly cluttered mixed setting, while DiVLA is evaluated on both seen and unseen objects.Source: §4.3 · pp. 6–7

4.4. Behavior Analysis of Robot Foundation Model

  • Self-generated reasoning supports interpretable generalization: the model analogizes unseen objects by semantic similarity and dynamically redirects actions when the target changes.A screwdriver is categorized as a hex key, while reasoning can shift from grabbing a toy car to grabbing a hex key after intervention.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5. Zero-Shot Bin Picking of Unseen Objects

  • 63.7% success on 102 unseen objects exceeds Diffusion Policy, Octo, TinyVLA, and OpenVLA, which achieve 8.9%, 19.6%, 23.5%, and 28.4%, respectively.The objects vary in size, color, texture, and deformability, testing instance generalization without training examples.Source: Figure 4 · p. 6, §4.5 · p. 8

4.6. Adapt to Real-World Bimanual Robot

  • 72.9% success on seen objects and up to 70.8% with seen and unseen objects show bimanual adaptation and generalization to different colors and shapes.Diffusion Policy and OpenVLA achieve 45.8% and 0% on seen objects, respectively.Source: §4.6 · p. 8, Table 2 · p. 8

5. Conclusion

DiVLA unifies next-token prediction for task reasoning with diffusion-based action prediction and reuses reasoning during action generation. Evaluations across simulations and multiple real-world embodiments show strong performance and generalization to new instructions, tasks, and environments.

5. Conclusion

  • DiVLA combines next-token prediction for task reasoning with diffusion models for action prediction, using a reasoning reuse module to enhance action generation.Source: §5 · p. 8
  • Across simulations and multiple real-world embodiments, DiVLA outperforms several state-of-the-art robot models and generalizes to new instructions, tasks, and environments.Source: §5 · p. 8

6. Evaluation Tasks and Detailed Results

The evaluation spans language-guided manipulation, spatial and commonsense reasoning, multi-step operations, and visual generalization. DiVLA-2B shows superior performance across most tasks while adapting to environmental changes.

6.1. Evaluation Tasks

  • The multi-task benchmark contains five tasks and evaluates 77 multi-task trials plus 45 visual-generalization trials.Source: §6.1 · p. 12
  • Object selection tests whether the robot follows varied user instructions to identify and pick the intended object.Source: §6.1 · p. 12
  • The benchmark tests commonsense orientation, multi-step manipulation, spatial precision, and instruction-based box selection.Tasks include flipping a vertically placed pot, opening a box before placing a cube, placing a cup on a shelf plate, and selecting a yellow or blue box.Source: §6.1 · p. 12
  • Visual generalization evaluates manipulation under added distractors, altered backgrounds, and colorful lighting.These conditions test whether the robot can maintain focus while adapting to changed visual environments.Source: §6.1 · p. 12, Figure 10 · p. 13

6.2. Detailed Results

  • DiVLA-2B consistently demonstrates superior performance across the majority of Franka and bimanual AgileX tasks.The complete results are reported for both robots in Table 4.Source: §6.2 · pp. 12–13
  • DiVLA-2B demonstrates robust visual generalization by adapting effectively to changes in the surrounding environment.Source: §6.2 · p. 13

7. Implementation Details

The baselines are trained or adapted under settings intended to make comparisons consistent, including matching OpenVLA’s three camera views. A single-view OpenVLA comparison shows a substantial sorting-performance decrease.

7. Implementation Details

  • OpenVLA is extended from one camera view to three, matching DiVLA’s visual input before fine-tuning with pretrained weights.Each view uses the same visual encoder, and the outputs are concatenated.Source: §7 · p. 13
  • 45.3% sorting success with three views falls to 12.7% with a single-view OpenVLA baseline.The comparison highlights the importance of the multi-view setup for this evaluation.Source: §7 · p. 13

8. More Experiments

Experiments show DiVLA generalizes across changed viewpoints and novel instructions, runs at high control frequencies, benefits from reasoning injection and scaling, and retains useful but imperfect visual-question-answering abilities.

8.1. View Shifting Generalization

  • 60% success rate under shifted camera views demonstrates DiVLA-2B’s robustness in factory sorting.The comparison evaluates DP, OpenVLA, and DiVLA-2B with completely different camera positions.Source: §8.1 · p. 13, Figure 12 · p. 14, Table 6 · p. 15

8.2. Efficient Inference

  • 82 Hz on an A6000 makes DiVLA-2B substantially faster than OpenVLA, while DiVLA-7B reaches 42 Hz and remains 8× faster at the same size.Without vLLM, DiVLA-2B and DiVLA-7B still reach 74 Hz and 30 Hz, respectively, or 6× OpenVLA’s speed.Source: §8.2 · p. 14, Table 5 · p. 15
  • Low-precision inference at 8-bit and 4-bit causes significant performance degradation, limiting straightforward quantization-based acceleration.The authors suggest that VLA-specific quantization methods may be needed to preserve performance.Source: §8.2 · p. 14

8.3. Following Novel Instruction

  • DiVLA correctly follows novel multi-step instructions, whereas OpenVLA handles basic picks but fails to interpret sequential commands accurately.The evaluation uses unseen commands involving watermelon, lemonade, blue paper trash, and red pepper.Source: §8.3 · p. 15, Table 9 · p. 16
  • Novel instructions reduce grasping precision, indicating that instruction novelty adds execution complexity despite successful sequential-task following.This limitation qualifies the model’s otherwise stronger performance on complex unseen commands.Source: §8.3 · p. 15

8.4. Ablation Study on Reasoning Injection Module

  • Removing reasoning injection causes a significant performance drop, supporting its role in task decomposition and more fine-grained action generation.The ablation compares the baseline against a model trained with the same methodology but without the reasoning injection module.Source: §8.4 · pp. 15–16, Table 8 · p. 15

8.5. Model Scaling

  • Scaling from DiVLA-2B to DiVLA-7B improves factory-sorting and bin-picking success rates by 8.7% and 3.0%, while 72B models generalize better out of distribution.Larger models also improve in-distribution performance and handle unseen object properties and dynamic conditions more effectively.Source: §8.5 · p. 16, Table 10 · p. 16

8.6. DiVLA can do Visual-Question-Answering

  • DiVLA answers some visual questions and spatial-relations prompts, but it often misidentifies objects and relies strongly on color cues.Examples include confusing a toy dragon with a toy tiger and a football with a regular ball.Source: §8.6 · pp. 16–17, Table 11 · p. 16

Related papers

Loading 2412.03293v3…