PaperlayerView exact revision on arXiv

Source-linked AI summary

TBD-VLA: Temporal Block Diffusion Vision Language Action Model

Sung-Wook Lee, Xuhui Kang, Yen-Ling Kuo

arXiv:2606.07895v1cs.CVcs.RO

TL;DR

Discrete VLA action generation is slowed by autoregressive decoding and lacks explicit modeling of temporal dependencies in parallel approaches. TBD-VLA applies masked discrete diffusion within temporal blocks and autoregressive generation across blocks, achieving strong results across simulation and real-world manipulation tasks. Its temporal modeling also supports Real-Time Chunking and competitive latency.

  • Problem

    Discrete VLA models face high inference latency from autoregressive action decoding, while parallel decoding provides limited mechanisms for modeling temporal dependencies.

  • Method

    TBD-VLA partitions discretized action sequences into temporal blocks, applying masked discrete diffusion within blocks and autoregressive generation across blocks.

  • Results

    TBD-VLA achieves strong generalization and robustness across simulated and real-world manipulation tasks with competitive latency.

  • Takeaways & Limitations

    Temporal block diffusion is a promising direction for temporally aware, low-latency, discrete VLA models compatible with Real-Time Chunking.

  • Takeaways & Limitations

    TBD-VLA can still fail under certain out-of-distribution conditions, including modified camera viewpoints in the “transfer the liquid” task.

Abstract

from arXiv · show

Discrete Vision-Language-Action (VLA) models typically formulate action generation as next-token prediction over discretized action spaces, conditioning each token autoregressively on prior context. While effective, this paradigm incurs high inference latency and largely ignores the temporal structure inherent in action trajectories. Recent efforts introduce parallel decoding to improve efficiency, enabling faster inference, but lack explicit mechanisms for modeling token dependencies. We introduce TBD-VLA, a discrete token-based VLA framework that incorporates block diffusion to enable temporal action generation. We partition action sequences into temporal blocks and perform masked discrete diffusion within each block, while maintaining autoregressive generation across blocks. This design unifies temporal autoregression and parallel action decoding, achieving both strong temporal coherence and improved inference speed. In addition, the explicit temporal modeling enables asynchronous execution of action chunks (e.g., Real-Time Chunking) via temporal in-painting. TBD-VLA significantly outperforms prior VLA approaches in both simulation and real-world manipulation tasks, offering a scalable path toward fast, temporally aware, discrete VLA models. Project webpage: https://tbd-vla.github.io/

1 Introduction

Discrete token-based VLA models face a latency challenge, while parallel decoding provides limited temporal dependency modeling. TBD-VLA addresses this by combining blockwise discrete diffusion with autoregressive generation across temporal blocks.

  • Autoregressive generation of long discrete action chunks is prohibitively slow for closed-loop, high-frequency robot control.
  • Existing parallel decoding approaches improve latency but provide limited mechanisms for modeling temporal dependencies across actions.
  • TBD-VLA partitions action sequences into temporal blocks, decoding tokens in parallel within each block and generating blocks autoregressively.
  • The framework combines parallel action decoding with temporal-level autoregression for temporally coherent action generation and faster inference.
  • TBD-VLA supports Real-Time Chunking through training with inpainting, aligning the model to complete partially committed action chunks.
  • The model is evaluated across multiple simulation and real-world benchmarks under varied perturbation scenarios.

2 Related Work

Discrete diffusion and block diffusion improve the efficiency of token generation, while existing discrete VLA methods use compact representations or faster decoding. TBD-VLA introduces temporal block structure to retain timestep-level tokens and temporal autoregression while decoding within-block tokens in parallel.

  • Masked diffusion refines many discrete tokens in parallel through iterative masked-token prediction.
  • Recent block diffusion combines parallel denoising within blocks with autoregressive generation across blocks.
  • Discrete VLA methods address efficiency through compact action representations or faster decoding.
  • TBD-VLA performs masked discrete diffusion within temporal blocks while generating blocks autoregressively.
  • Unlike compressed-token methods, TBD-VLA preserves timestep-level action tokens and retains temporal autoregression across blocks.

3 Problem Statement

The problem formulation maps visual and proprioceptive observations plus a task specification to a future sequence of robot actions. Actions are discretized into tokens and their likelihood is factorized over temporal blocks.

  • The policy maps visual inputs, proprioceptive state, and a task specification to a sequence of future robot actions.
  • Each action feature is discretized into one of Nb bins, with corresponding tokens drawn from a vocabulary of size |V| = Nb.
  • An action chunk has token length Lt = Hp · Da, where Hp is the prediction horizon and Da is the action dimension.
  • The action-sequence likelihood is factorized over temporal blocks, with block size m and K = Hp/m blocks.

4 Method

TBD-VLA generates discrete action sequences through masked diffusion within temporal blocks and autoregressive conditioning across blocks. Its training and inference pipeline combines temporal modeling with parallel decoding, selective execution, caching, and asynchronous chunking.

  • Temporal-level Token Shift: Temporal-level token shift trains each current action block to predict the next block, aligning diffusion with the VLM’s autoregressive objective.The shifted predictor uses an anchor block for the first action block and clean preceding blocks thereafter.
  • Discrete Block Diffusion: TBD-VLA partitions tokenized action sequences into temporal blocks and applies masked discrete diffusion within each block.Each clean action block is independently corrupted by masking tokens before reverse prediction.
  • Block-level Attention Masking: A doubled-layout input and custom attention mask process clean and corrupted action sequences in parallel while supporting inter-block autoregression.The clean and noised sequences share RoPE positions during block-level attention masking.
  • Decoding as Needed: At inference, TBD-VLA sequentially generates only the action blocks required for the execution horizon, refining masked positions through discrete diffusion steps.For rollout horizon Ha, it generates Kexec = ⌈Ha/m⌉ blocks rather than all K = Hp/m blocks, requiring Kexec · nd denoising steps.
  • Prefix KV Cache: Prefix key–value caching avoids recomputing unchanged visual, prompt, and previously generated action-block context during denoising.This caching strategy improves inference efficiency by reusing context across diffusion steps.
  • Real-Time Chunking: Real-Time Chunking asynchronously generates future actions while executing current actions by freezing and reusing the previously generated action tail as in-painting context.The hard in-painting strategy uses the inference-latency window to provide context for early future action blocks.

5 Experiments

The experiments evaluate TBD-VLA across simulation, real-to-sim transfer, and real-world manipulation, including perturbations and inference-latency settings. Results show strong benchmark performance and benefits from temporal modeling, RTC, and block-diffusion design choices.

  • Experimental setup: The evaluation spans diverse simulation, transfer, and real-world settings, including perturbations, RTC, and inference-efficiency analyses.Experiments cover LIBERO, LIBERO-Plus, SimplerEnv, and three real-world tabletop tasks.
  • Simulation results: 97.6% average success rate makes TBD-VLA state of the art on LIBERO test suites.At four simulation-step inference delay, TBD-VLA with RTC retains 93.2% success, 3.4 percentage points above π0.5 with RTC.
  • SimplerEnv results: TBD-VLA reaches 66.8% on SimplerEnv Widow-X and 91.0% and 86.3% on Google Robot visually matching and variant aggregation tasks.It ranks second on Widow-X behind UniVLA at 69.8% and outperforms baselines on the Google Robot tasks.
  • Real-world results: 67.1% average real-world success across three tasks exceeds π0.5 at 50.0%, while removing RTC lowers TBD-VLA to 60.0%.Evaluation includes one in-distribution and three perturbation settings.
  • Ablations and efficiency: The study varies temporal block size, diffusion refinement steps, and action sampling while measuring success rate, inference time, and VLM forward passes.Decode-as-needed, KV caching, and VLM compilation are evaluated as inference optimizations.

6 Limitations

The paper identifies remaining robustness and training-strategy limitations for TBD-VLA. Certain out-of-distribution conditions can still cause failure, and alternative co-training strategies remain unexplored.

  • Robustness: TBD-VLA can fail under some out-of-distribution conditions, including a modified camera viewpoint in the “transfer the liquid” task.The authors associate this failure with the task’s need for accurate visual fidelity.
  • Future work: Alternative training strategies, including co-training with auxiliary VLM objectives, are left for future work.The paper also leaves deeper interpretation of VLM-only action decoding for future study.

7 Conclusion

TBD-VLA combines temporal autoregression with parallel action decoding through block discrete diffusion. Across simulated and real-world manipulation tasks, it achieves strong generalization, robustness, competitive latency, and compatibility with Real-Time Chunking.

  • TBD-VLA denoises tokens within each temporal block in parallel while generating blocks autoregressively, explicitly modeling temporal dependencies.This preserves VLM-compatible action generation while combining parallel decoding with temporal autoregression.
  • Across simulated and real-world manipulation tasks, TBD-VLA achieves strong generalization, robustness, and competitive latency.
  • TBD-VLA is compatible with Real-Time Chunking for temporally aware, low-latency discrete VLA models.
  • Training and deployment use the LeRobot framework across the simulated and real-world benchmarks.All models are trained using 4 NVIDIA A100 GPUs.

A.1 Pre-training

TBD-VLA is pre-trained on a broad mixture of robot manipulation datasets before fine-tuning on target benchmarks. The supplied protocol specifies dataset scale, training stages, hardware, evaluation codebases, and inference configurations.

  • The pre-training mixture contains 160,268 robot demonstration episodes and 32,351,396 training samples across multiple datasets, domains, embodiments, and camera views.Datasets include DROID, BC-Z, RoboMind, RoboSet, MolmoAct, RH20T, and Open-X Embodiment subsets.
  • Table 7 reports the pre-training datasets together with robot demonstration episodes and training samples.
  • After pre-training, the policy is fine-tuned on target datasets for SimplerEnv, LIBERO, and LIBERO-Plus benchmarks.Fine-tuning requires approximately 40–120 GPU hours depending on the benchmark.
  • Training uses shared and stage-specific hyperparameter configurations summarized in Tables 8–10.Table 10 covers inference-time settings for fine-tuned policies.
  • Evaluations use official LIBERO, LIBERO-Plus, ManiSkill2, and SimplerEnv implementations with the LeRobot evaluation wrapper where specified.

B.2 LIBERO Results under Inference Latency

TBD-VLA remains effective under inference latency, and Real-Time Chunking becomes increasingly beneficial as latency rises. On LIBERO-Plus, it also achieves strong robustness across perturbation settings, with pre-training providing especially large gains for several shifts.

  • LIBERO Results under Inference Latency: 97.7% overall success is achieved at zero latency, while performance without RTC falls to 72.3% at Latency L = 4.
  • LIBERO Results under Inference Latency: +20.9 percentage points is the RTC improvement at Latency L = 4, reaching 93.2% overall success.The table reports absolute changes of w/ RTC relative to w/o RTC for L > 0.
  • LIBERO-Plus Full Results: 83.49% average success is achieved across all LIBERO-Plus suites and perturbation types.Table 12 provides the full robustness comparison with and without pre-training.
  • LIBERO-Plus Full Results: +58.38% under camera-viewpoint, +28.29% under sensor-noise, and +25.24% under language-instruction perturbations are the larger pre-training gains reported.Figure 6 visualizes the benefits of pre-training across seven perturbation settings.
  • Real-World Setup: The real-world setup uses a Franka Research 3 robot with UMI grippers and width commands for precise manipulation in Transfer the Liquid.

C.3 Evaluation Protocol

Real-world evaluation covers three tabletop manipulation tasks under one in-distribution and three out-of-distribution perturbation settings. Each task and setting is evaluated with 20 rollouts, with success defined by task completion.

  • Each method is evaluated in one in-distribution setting and three out-of-distribution settings: camera viewpoint, language instruction, and background/lighting.The background/lighting shift combines a gray table cover with a spotlight.
  • Twenty rollouts are run for each task and evaluation setting.
  • Table 13 reports real-world success counts out of total rollouts and average success rates for each task and perturbation setting.

D Real-World Results

TBD-VLA achieves strong real-world performance in-distribution and remains effective under several perturbations, while camera viewpoint shifts expose a task-specific failure. Real-Time Chunking further improves average success.

  • 86.67% success rate across three tasks in the in-distribution setting.The camera view, language instruction, and background match the training data.
  • 51.67%, 50.00%, and 80.00% success rates under modified camera view, language instructions, and background shifts, respectively.
  • 7.08% average success-rate improvement from enabling Real-Time Chunking.The improvement is attributed to temporal modeling with asynchronous inference.
  • Modified camera viewpoint causes zero success on “Transfer the Liquid,” where the robot cannot approach the dropper.The authors identify visual-consistency requirements and under-representation of similar tasks in pre-training as likely factors.
Loading 2606.07895v1…