Source-linked AI summary
Training-Time Action Conditioning for Efficient Real-Time Chunking
Kevin Black, Allen Z. Ren, Michael Equi, Sergey Levine
TL;DR
Inference-time inpainting makes real-time chunking costly and less effective at high delays. The paper simulates inference delay during training and conditions directly on action prefixes, achieving strong simulated and real-world parity without inference-time computational overhead. Training-time RTC is therefore presented as a practical drop-in replacement, with reduced flexibility and dependence on the simulated-delay distribution.
Problem
Inference-time RTC inpainting adds computational overhead and is fundamentally limited at high inference delays, complicating reactive robot control.
Method
The method simulates inference delay during training and learns postfix actions conditioned on ground-truth action prefixes.
Results
Training-time RTC outperforms inference-time RTC at higher simulated delays and maintains performance and speed parity in real-world experiments without computational overhead.
Takeaways & Limitations
Training-time action conditioning provides a practical drop-in replacement for inference-time RTC in real-time robot control.
Takeaways & Limitations
Training-time RTC only supports hard prefixes tied to simulated inference delays and requires choosing the training delay distribution carefully.
Abstract
from arXiv · showhide
Real-time chunking (RTC) enables vision-language-action models (VLAs) to generate smooth, reactive robot trajectories by asynchronously predicting action chunks and conditioning on previously committed actions via inference-time inpainting. However, this inpainting method introduces computational overhead that increases inference latency. In this work, we propose a simple alternative: simulating inference delay at training time and conditioning on action prefixes directly, eliminating any inference-time overhead. Our method requires no modifications to the model architecture or robot runtime, and can be implemented with only a few additional lines of code. In simulated experiments, we find that training-time RTC outperforms inference-time RTC at higher inference delays. In real-world experiments on box building and espresso making tasks with the $π_{0.6}$ VLA, we demonstrate that training-time RTC maintains both task performance and speed parity with inference-time RTC while being computationally cheaper. Our results suggest that training-time action conditioning is a practical drop-in replacement for inference-time inpainting in real-time robot control.
I. INTRODUCTION
Real-time chunking addresses the need for reactive robot control despite substantial VLA inference latency, but inference-time inpainting adds overhead. The paper proposes simulating delay during training to provide a drop-in alternative without inference-time overhead.
- VLAs must produce smooth, reactive trajectories while operating at high frequencies despite inference latency of tens to hundreds of milliseconds.
- Inference-time RTC asynchronously predicts chunks and conditions each new chunk on previously predicted actions, but its inpainting adds latency and struggles with high delays.
- The proposed method simulates inference delay during training, eliminating inference-time computational overhead.
- Training-time RTC requires no model-architecture or robot-runtime modifications and can be implemented with a few additional lines of code.
- In simulation, training-time RTC outperforms inference-time RTC at higher delays, while real-world experiments show improved performance on box building and espresso making.
II. RELATED WORK
Related work improves VLA efficiency and real-time execution through faster architectures, hierarchical designs, and asynchronous control. These approaches differ in their treatment of inter-chunk continuity and architectural changes.
- Hierarchical VLAs separate heavyweight high-level planning from lightweight low-level action generation to support high-frequency control.
- MiniVLA and SmolVLA pursue faster, more efficient VLA architectures that make edge inference more feasible.
- RTC provides an asynchronous execution framework, while SmolVLA leaves inter-chunk discontinuities that can produce out-of-distribution jerks.
- A2C2 and VLASH address discontinuity using a lightweight correction head or conditioning on a single feature.
III. PRELIMINARIES
The paper formulates chunked action prediction around horizon, execution, and inference-delay variables. When delay is sufficiently short, overlapping actions from the previous chunk form an executable action prefix.
- Figure 1 distinguishes the d-action prefix from the full H − s overlap used by inference-time RTC and the prefix-only conditioning used by training-time RTC.
- An action chunk At contains H future actions conditioned on observation ot, and the controller executes each chunk for s ≤ H timesteps.
- Inference delay d measures controller timesteps between inference start and chunk availability, making the first d actions unavailable for direct execution.
- When d ≤ H − s, actions from the previous chunk cover the delay and constitute the action prefix.
- Conditional flow matching trains a neural network vθ over timestep τ and integrates it from τ = 0 to 1 to sample actions from p(At|ot).
IV. TRAINING-TIME ACTION CONDITIONING
Inference-time RTC uses flexible inpainting over overlapping actions but incurs a vector-Jacobian-product cost during denoising. The proposed training-time method instead learns postfix generation from ground-truth prefixes while simulating delay.
- Inference-time RTC uses pseudoinverse guidance to condition on action prefixes and soft masking to incorporate additional overlapping actions.
- Pseudoinverse guidance requires a vector-Jacobian product during every denoising step, creating inference-time computational overhead.
- Training-time conditioning learns p(At+d:H|ot, At:t+d) from a ground-truth action prefix and corresponding postfix while simulating inference delay.
- The conditioning architecture feeds non-noisy ground-truth prefix actions and denoises postfix actions with token-dependent flow-matching timesteps.
- The resulting generator accepts an action prefix and delay and outputs an action postfix, matching inference-time RTC’s interface.
V. EXPERIMENTS
The experiments compare training-time RTC with inference-time RTC and synchronous or asynchronous baselines on simulated and real-world robot-control tasks.
- V. EXPERIMENTS: Experiments compare training-time RTC with inference-time RTC, naive synchronous inference, and asynchronous baselines.The simulated benchmark is dynamic Kinetix, while real-world evaluation uses box building and espresso making.
- V. EXPERIMENTS: Simulated results measure solve rate against inference delay using a fixed execution horizon s = max(d, 1).Each data point represents 2048 trials, with 95% Wilson score intervals shown.
A. Simulated Results
On the dynamic Kinetix benchmark, training-time RTC is fine-tuned with action-prefix conditioning and outperforms inference-time RTC at delays of 2 or higher.
- A. Simulated Results: The benchmark uses prediction horizon H = 8, a 4-layer MLP-Mixer, 2048 rollouts per data point, and delays from 0 to 4.Binary success rates are reported on the dynamic Kinetix benchmark.
- A. Simulated Results: Training-time RTC is fine-tuned for 8 epochs after epoch 24, sampling delays from {0, 1, 2, 3, 4} with exponentially decreasing weights.This matches methods in training compute, while separate checkpoints for each delay could use more training compute.
- A. Simulated Results: Training-time RTC outperforms inference-time RTC at inference delays of 2 and higher, with the gap widening as delay increases.It is marginally worse at delays 1 and 0, likely because some actions receive less training supervision.
B. Real-World Results
In real-world box building and espresso-making evaluations, training-time RTC matches inference-time RTC in performance and speed while both outperform synchronous inference in speed.
- B. Real-World Results: The real-world tasks are espresso making and box building using the π0.6 base model.Espresso making includes grinding, tamping, extracting, and pouring.
- B. Real-World Results: Training samples delays uniformly from 0 to 10, supporting a maximum latency of 200ms on a 50Hz robot.Inference uses five denoising steps on a remote H100 server, averaging 108ms of end-to-end latency.
- B. Real-World Results: The evaluation reports success rate and duration for espresso making and box building.Error bars show 68% Wilson score intervals for success rate and ±1 SEM for duration.
- B. Real-World Results: Training-time RTC maintains performance and speed parity with inference-time RTC without computational overhead.Both RTC variants improve speed over the synchronous inference baseline.
- B. Real-World Results: Both RTC variants improve speed over synchronous inference, whose execution exhibits visible pauses between chunks.The synchronous baseline uses the same checkpoint as inference-time RTC.
VI. DISCUSSION AND FUTURE WORK
The paper presents training-time RTC as a drop-in replacement that removes inference-time computational overhead, while identifying reduced flexibility and delay-distribution selection as limitations.
- VI. DISCUSSION AND FUTURE WORK: Training-time RTC removes inference-time computational overhead by adding a small amount of additional training compute.The method requires no model-architecture or robot-runtime modifications and only a few additional lines of code.
- VI. DISCUSSION AND FUTURE WORK: Simulated experiments favor training-time RTC at higher delays, while real-world experiments show parity with inference-time RTC in performance and speed.The real-world parity is achieved without computational overhead.
- VI. DISCUSSION AND FUTURE WORK: Training-time RTC is less flexible because it supports only a hard prefix tied to inference delay, whereas inference-time RTC can softly incorporate additional actions.Training-time RTC also requires choosing the training delay distribution based on expected inference latency.