PaperlayerView exact revision on arXiv

Source-linked AI summary

Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policy

Zhi Hou, Tianyi Zhang, Yuwen Xiong, Haonan Duan, Hengjun Pu, Ronglei Tong, Chengyang Zhao, Xizhou Zhu, Yu Qiao, Jifeng Dai, Yuntao Chen

arXiv:2503.19757v2cs.ROcs.CV

TL;DR

Generalist robot policies must adapt across heterogeneous embodiments and action spaces, but compact action heads limit continuous-action denoising. Dita directly denoises continuous action sequences with Transformer-based in-context conditioning, achieving strong simulation performance and robust real-world adaptation after 10-shot finetuning using third-person images.

  • Problem

    Compact diffusion heads struggle to denoise continuous actions across the diverse camera views, embodiments, and action spaces found in cross-embodiment datasets.

  • Method

    Dita uses a causal Transformer to directly denoise continuous action chunks while conditioning in context on language tokens, image features, timesteps, and noisy actions.

  • Results

    Dita achieves state-of-the-art or competitive simulation results and succeeds on novel real-world long-horizon tasks after 10-shot finetuning with a single third-person camera.

  • Takeaways & Limitations

    Dita provides a lightweight, open-source baseline for generalist robot policy learning across cross-embodiment datasets and real-world setups.

Abstract

from arXiv · show

While recent vision-language-action models trained on diverse robot datasets exhibit promising generalization capabilities with limited in-domain data, their reliance on compact action heads to predict discretized or continuous actions constrains adaptability to heterogeneous action spaces. We present Dita, a scalable framework that leverages Transformer architectures to directly denoise continuous action sequences through a unified multimodal diffusion process. Departing from prior methods that condition denoising on fused embeddings via shallow networks, Dita employs in-context conditioning -- enabling fine-grained alignment between denoised actions and raw visual tokens from historical observations. This design explicitly models action deltas and environmental nuances. By scaling the diffusion action denoiser alongside the Transformer's scalability, Dita effectively integrates cross-embodiment datasets across diverse camera perspectives, observation scenes, tasks, and action spaces. Such synergy enhances robustness against various variances and facilitates the successful execution of long-horizon tasks. Evaluations across extensive benchmarks demonstrate state-of-the-art or comparative performance in simulation. Notably, Dita achieves robust real-world adaptation to environmental variances and complex long-horizon tasks through 10-shot finetuning, using only third-person camera inputs. The architecture establishes a versatile, lightweight and open-source baseline for generalist robot policy learning. Project Page: https://robodita.github.io.

1. Introduction

Generalist robot policies aim to transfer across tasks and environments using heterogeneous robot data, but compact diffusion heads struggle with diverse embodiments and action spaces. Dita addresses this with scalable in-context action denoising and reports strong simulation and few-shot real-world results.

  • Large-scale robot data for generalized tasks is costly to collect, motivating policies pretrained on heterogeneous data and adapted with minimal supervision.
  • Recent VLA models map visual observations and language instructions to actions, demonstrating zero-shot or few-shot generalization in novel environments.
  • Dita’s architecture contrasts discretized-action Transformer policies and compact diffusion action heads with inherent in-context action denoising.
  • Compact diffusion heads face difficulty denoising continuous actions across diverse camera views, embodiments, and action spaces.
  • Dita directly conditions continuous action-sequence denoising on image tokens through an in-context causal Transformer, modeling subtle action deltas in historical observations.The model is presented as a 334M-parameter lightweight, open-source baseline.
  • 10-shot finetuning with only a third-person camera supports generalization to novel environments and long-horizon real-world tasks.

2. Related Work

Related work spans language-conditioned generalist policies and diffusion-based action generation. Existing approaches improve generalization and visual reasoning, while Dita focuses on scalable diffusion-based action generation with a causal Transformer.

  • Diffusion-based manipulation policies commonly use U-Nets, shallow cross-attention networks, or compact MLP diffusers, limiting scalability to multimodal applications.
  • Generalist policies increasingly use foundation multimodal models for navigation and manipulation, with scalable VLA models as a dominant framework.
  • Some approaches use internet-scale video backbones for temporal visual reasoning, whereas Dita concentrates on expressive diffusion-based action generation.
  • Dita adopts a LLaMA-style causal Transformer rather than PaliGemma for policy learning, emphasizing simplicity and scalability.

3. Method

Dita is a Transformer-based diffusion policy that tokenizes language, images, timesteps, and noisy actions into one sequence for direct action-chunk denoising. Its training and inference use DDPM and DDIM objectives with multi-frame observations and continuous action vectors.

  • 3.1. Architecture: Language instructions are encoded with pretrained CLIP, while DINOv2 extracts image features and a Q-Former reduces each image representation to 32 tokens with FiLM conditioning.
  • 3.1. Architecture: The end-effector action is represented as a continuous 7D vector containing translation, rotation, and gripper position, with noise applied only to those action dimensions.
  • 3.1. Architecture: Dita uses a scalable DiT with in-context conditioning, concatenating language tokens, image features, timestep embeddings, and noisy action tokens in a causal Transformer.Direct image-token conditioning is intended to capture nuanced action changes across historical observations.
  • 3.2. Training Objective: During training, Gaussian noise is added to actions at sampled timesteps, and the causal Transformer predicts that noise using an MSE objective.
  • 3.2. Training Objective: Inference iteratively denoises actions with scheduler parameters α, γ, and σ over Neval evaluation steps.
  • 3.3. Data and Implementation: Pretraining uses OXE datasets with normalized and filtered actions following prior dataset-selection and weighting procedures.
  • 3.3. Data and Implementation: The implementation trains with DDPM for 1000 timesteps and uses DDIM with 20 evaluation timesteps, accelerating zero-shot inference.

4. Simulation Experiments

Dita is evaluated across diverse simulation benchmarks for zero-shot generalization, multitask transfer, long-horizon control, camera-view variation, and architectural factors. Across these settings, it shows strong or comparative performance, including improvements on LIBERO, CALVIN, and ManiSkill2 tasks.

  • Benchmark setup: Dita is evaluated on SimplerEnv, LIBERO, CALVIN, and ManiSkill2 to assess zero-shot adaptation, multitask transfer, long-horizon performance, and camera-view generalization.The experiments cover both benchmark comparisons and an ablation study of observation and trajectory lengths.
  • SimplerEnv: Dita achieves strong zero-shot generalization across background, texture, object, and spatial-position variations using only third-person images on SimplerEnv.The comparison includes RT-1-X, Octo-Base, and OpenVLA-7B under match and variant scenarios.
  • LIBERO: Dita increases average LIBERO success rate by nearly 6% and improves LIBERO-LONG performance by 10%, indicating strong transfer to long-horizon tasks.The LIBERO-LONG subset contains 10 long-horizon tasks with diverse object interactions and motor skills.
  • CALVIN: On CALVIN, Dita performs comparably to single-RGB-camera methods, surpasses its non-pretrained variant by 1.23, and exceeds a similar diffusion-head variant by 0.45 points.The results also attribute effective transfer to discerning subtle visual nuances in long-horizon tasks across diverse environments.
  • ManiSkill2: Dita outperforms the diffusion-head baseline by 20% on PickSingleYCB and 12% on PickClutterYCB in novel camera-view scenarios.The results are reported on ManiSkill2, where Dita is compared with discretization and diffusion action heads.
  • Ablation study: Two-frame observations improve performance, while longer trajectory lengths substantially benefit complex tasks such as PickClutterYCB.Longer trajectories help the model anticipate target positions and future states; simpler tasks maintain high performance once trajectory length exceeds 4.

5. Real-Robot Experiments

Dita is evaluated for few-shot adaptation to unseen real-robot environments using 10-shot finetuning and a third-person RGB-D camera. It completes complex long-horizon tasks and remains robust to environmental variations, outperforming comparison policies in reported experiments.

  • Experimental Setup: 10-shot finetuning evaluates Dita’s adaptation to complex, long-horizon, multimodal tasks in unseen robot environments.The real-robot experiments use a 7-DoF Franka arm, Robotiq gripper, and third-person camera setup.
  • Experimental Setup: Dita is evaluated on pick-and-place tasks with banana and kiwifruit targets, including position variations during evaluation.Ten samples are collected for each task.
  • Baselines: The real-robot comparison uses Dita, Octo, OpenVLA, and a multimodal diffusion-policy baseline finetuned with LoRA.The baseline is implemented with a causal Transformer and a diffusion head for comparison.
  • Quantitative Results: Dita’s long-horizon tasks show the second stage contributing nearly half of overall success, indicating substantial completion beyond the first stage.Figure 5 separates first-stage success from the second-stage contribution in stacked bars.
  • Results: Dita consistently outperforms Octo and OpenVLA on complex tasks and remains resilient to background, object-arrangement, and lighting variations.The reported variance tests include changes to backgrounds, non-target-object arrangements, and lighting conditions.
  • Qualitative Comparison: Under 10-shot finetuning, baseline methods often miss grasp positions or misunderstand lifting, whereas Dita completes complex tasks involving extreme 3D rotations.The qualitative comparison includes insertion, grasping, and pouring examples.

6. Conclusion

Dita is presented as a scalable generalist robot-learning architecture that combines Transformer-based diffusion with in-context conditioning. The paper reports unified simulation generalization and few-shot transfer to novel real-world setups and long-horizon tasks using a single third-person camera.

  • Architecture: Dita denoises continuous action sequences with a Transformer-based diffusion model and in-context conditioning.The architecture is designed for generalist robot learning.
  • Generalization: Dita models diverse robot behaviors across cross-embodiment datasets and generalizes across multiple simulation benchmarks within one framework.The conclusion attributes this scope to Transformer scalability.
  • Real-World Adaptation: Dita transfers to novel real-world robot setups and long-horizon tasks with minimal in-domain samples.The conclusion characterizes this as few-shot adaptation.
  • Practical Scope: The model is described as clean, lightweight, open-source, and capable of promising performance using only a single third-person camera input.The architecture also permits investigation of additional modalities.

A. Model and Training Scheme

Dita combines frozen language encoding, finetuned visual encoding, and a learned Q-Former to produce compact multimodal inputs for the model.

  • Language and Vision Encoding: A pretrained CLIP encoder encodes language instructions and remains frozen during training.The language representation is used as part of the multimodal model input.
  • Language and Vision Encoding: A pretrained DINOv2 model processes 224 × 224 input images, with all of its parameters finetuned.The visual encoder supplies image features to subsequent processing.
  • Feature Compression: A four-layer Q-Former trained from scratch reduces image features to a sequence of 32 tokens.Text tokens are injected as FiLM conditions within each block.

B. Simulation Benchmarks

Dita is assessed across simulation benchmarks covering real-to-sim transfer, long-horizon task sequences, camera-view generalization, and multiple manipulation settings. The experiments use visual inputs and compare Dita with diffusion and generalist-policy baselines.

  • LIBERO: LIBERO covers spatial, object, goal, and long-horizon subtasks that test layout, object, task, and extended-sequence generalization.The benchmark is used to assess different model capabilities across its four sub-datasets.
  • CALVIN: Dita is evaluated on CALVIN using only a single static RGB camera to predict translation, rotation, and gripper actions.The action representation contains three translation, three Euler-rotation, and one gripper dimension.
  • ManiSkill2: ManiSkill2 provides 20 manipulation task families and over 4M demonstration frames for generalized manipulation evaluation.Dita is evaluated with a camera-view generalization benchmark built from five selected tasks and a 300K-camera pool.
  • ManiSkill2: The ManiSkill2 setup uses more than 40K rendered trajectories and evaluates robustness under background, object-arrangement, and lighting changes.ColorJitter augmentation is also applied during optimization.

C.2. Details of Real-Robot Tasks

The real-robot evaluation uses complex multitask and long-horizon tasks, with full finetuning improving robustness beyond the limited capacity of LoRA finetuning.

  • Task Design: The real-robot suite includes pick-and-place, pouring, stacking, insertion, and multi-step drawer and box manipulation tasks.Tasks are decomposed into sequential subtasks, while several extended-horizon tasks are demonstrated because previous approaches fail to complete them.
  • Finetuning and Robustness: Full finetuning achieves a 20% success rate under extreme variances, whereas LoRA finetuning fails to complete tasks.The paper attributes the difference to LoRA leaving only approximately 11M trainable parameters, which is inadequate for accommodating image augmentations.
  • Baseline Comparison: OpenVLA performs better on pickup tasks, whereas Octo performs better on rotational operations such as opening a box.The paper relates this difference to continuous actions being less sensitive to action normalization than discretized actions.

D.2. Convergence Analysis

The convergence analysis compares Dita with a diffusion-head baseline and reports faster convergence for Dita, alongside representative execution failures involving grasp positioning and unstable pouring.

  • Convergence: Dita achieves clearly faster convergence than the diffusion-head baseline EDiffθ.Figure 11 compares the two policies on the OXE dataset using the same number of parameters.
  • Failure Analysis: A representative failure involves approaching an incorrect position for a successful grasp, even though the model can recover and retry.This indicates a remaining execution weakness in precise pickup positioning.
  • Failure Analysis: During pouring while in motion, the model tends to spill contents because of unstable handling.The failure analysis identifies this as a separate representative failure case.

D.4. More Ablations

The ablations examine learning-rate scheduling, action designs, execution steps, shuffle-buffer size, and denoising steps, with several settings affecting generalization and execution accuracy.

  • Optimization: A standard learning-rate scheduler produces a slight performance improvement over a fixed learning rate in the Calvin benchmark.The scheduler decays the learning rate during experiments rather than retaining 1e-4 throughout.
  • Denoising: The ablations also evaluate the number of DDIM denoising steps during inference on Google Robot Simulation.The reported table examines inference denoising steps after training with 1000 DDPM denoising steps.
  • Execution Steps: Shorter execution steps yield slightly better results than longer steps, while two-frame observations still support reliable action trajectories.The paper attributes the decline with longer horizons to lower accuracy farther from the current frame.
  • Action Designs: Dita achieves better generalization on Calvin ABC→D than the compared diffusion-head strategies without pretraining.The comparison includes multiple action designs and diffusion-head variants.
  • Data Loading: Shuffle-buffer size significantly influences performance in the reported SimplerEnv ablation.The experiment evaluates both matched and variant Google Robot results.
Loading 2503.19757v2…