PaperlayerView exact revision on arXiv

Source-linked AI summary

Leave No Observation Behind: Real-time Correction for VLA Action Chunks

Kohei Sendai, Maxime Alvarez, Tatsuya Matsushima, Yutaka Matsuo, Yusuke Iwasawa

arXiv:2509.23224v1cs.ROcs.AIcs.CVcs.SYeess.SY

TL;DR

Large VLA models incur inference latency, leaving chunked actions less responsive to new observations. A2C2 adds a lightweight correction head that uses the latest observation to adjust action chunks. It improves reported success rates over RTC across delay and long-horizon settings, with gains on Kinetix and results across Kinetix and LIBERO Spatial.

  • Problem

    Large VLA inference latency and open-loop action chunks based on previous observations limit reactive behavior when environments change during inference.

    Source: §1 · p. 1

  • Method

    A2C2 augments a base VLA policy with a lightweight correction head that integrates the latest observation into its action chunks.

    Source: §1 · p. 2

  • Results

    23% point increase in Kinetix success rate over RTC in the presence of delay, with A2C2 maintaining high success across Kinetix and LIBERO Spatial where naive and RTC degraded.

    Source: §1 · p. 3, §7 · p. 9

  • Takeaways & Limitations

    A2C2 is presented as a lightweight add-on that can in principle be applied to off-the-shelf VLAs to help preserve reactivity under delay and long execution horizons.

    Source: §1 · p. 2, §7 · p. 9

    Further validation is needed for richer language instructions, out-of-distribution settings, and more dynamic tasks beyond LIBERO Spatial.

    Source: §7 · p. 9

Abstract

from arXiv · show

To improve efficiency and temporal coherence, Vision-Language-Action (VLA) models often predict action chunks; however, this action chunking harms reactivity under inference delay and long horizons. We introduce Asynchronous Action Chunk Correction (A2C2), which is a lightweight real-time chunk correction head that runs every control step and adds a time-aware correction to any off-the-shelf VLA's action chunk. The module combines the latest observation, the predicted action from VLA (base action), a positional feature that encodes the index of the base action within the chunk, and some features from the base policy, then outputs a per-step correction. This preserves the base model's competence while restoring closed-loop responsiveness. The approach requires no retraining of the base policy and is orthogonal to asynchronous execution schemes such as Real Time Chunking (RTC). On the dynamic Kinetix task suite (12 tasks) and LIBERO Spatial, our method yields consistent success rate improvements across increasing delays and execution horizons (+23% point and +7% point respectively, compared to RTC), and also improves robustness for long horizons even with zero injected delay. Since the correction head is small and fast, there is minimal overhead compared to the inference of large VLA models. These results indicate that A2C2 is an effective, plug-in mechanism for deploying high-capacity chunking policies in real-time control.

1 INTRODUCTION

Large VLA inference latency makes chunked actions less responsive to changing observations, while existing chunking and hierarchical approaches leave reactivity limited. A2C2 adds per-timestep correction, with reported success-rate gains over naive execution and RTC on Kinetix.

1 INTRODUCTION

  • Large VLA models incur high inference latency, and open-loop chunks based on previous observations can accumulate errors in rapidly changing environments.This can lower success rates and sometimes lead to task failure.Source: §1 · pp. 1–2
  • Action chunking reduces expensive inference frequency, while synchronous execution and RTC address waiting or chunk continuity; fixed-length horizons still limit reactivity to new sensory input.Hierarchical approaches also retain latency because the low-level executor waits for high-level predictions.Source: §1 · p. 2
  • A2C2 is a lightweight correction head that incorporates the latest observation into VLA action chunks to maintain responsiveness under inference delays and long horizons.It complements rather than replaces the base policy.Source: §1 · p. 2
  • 35% point increase in Kinetix success rate over naive execution and 23% point increase over RTC in the presence of delay; long execution horizons yield gains of 12% and 7% points, respectively.The reported comparisons are against naive execution and RTC, respectively.Source: §1 · p. 3
  • The paper formulates inference delays for VLA-generated action chunks and introduces A2C2 as an architecture-independent add-on for improving reactivity across dynamic tasks and manipulation benchmarks.The contribution summary describes improvements under varied inference delays.Source: §1 · p. 3

2 PROBLEM FORMULATION

The formulation describes an imitation-learning policy that generates action chunks from observations and language instructions. It defines execution horizon and inference delay, and characterizes the resulting constraint on action availability and observation freshness.

2 PROBLEM FORMULATION

  • The imitation-learning policy maps observation o_t and language instruction l to an action chunk A_t of horizon H.H is the training sequence length of the policy.Source: §2 · p. 3
  • Execution horizon e is the number of chunk actions used before the policy predicts another chunk, while delay d is the control-step gap between receiving an observation and obtaining its corresponding chunk.The delay combines inference and communication time, measured relative to a control-step duration.Source: §2 · p. 3
  • The execution horizon must satisfy d ≤ e ≤ H − d to avoid waiting during inference or exhausting the current chunk before the next arrives.The policy server is assumed to handle only one inference at a time.Source: §2 · p. 3
  • Each executed action uses a past observation at least d steps old and, in the worst case, one d + e steps old.The formulation identifies this stale-observation exposure as a feature of delayed chunk execution.Source: §2 · p. 3

3 METHOD

A2C2 adds a lightweight correction head to a chunk-based policy, refining actions with recent observations and within-chunk position. It is trained to predict residuals from expert demonstrations while leaving the base policy unchanged.

3.1 OVERVIEW

  • A2C2 refines each predicted chunk action using the latest observation, base-policy features, and a temporal position feature, without retraining the base policy.The correction head outputs a residual that is added to the corresponding base action.Source: §3.1 · pp. 3–4
  • The base policy infers a chunk every e steps with delay d, while the correction head is assumed fast enough to run within one control step.The head is intended to incorporate new observations at each step.Source: §3.1 · p. 4

3.2 MODEL TRAINING PROCEDURE

  • The correction head uses the same demonstration datasets as the base VLA policy and does not require reinforcement-learning fine-tuning.Its training examples are assembled into a correction dataset from base-policy inference results.Source: §3.1 · p. 4, §3.2 · p. 5
  • Correction training targets the difference between expert-demonstration actions and the base-policy output, using mean squared error.The expert action is the target action from the demonstration dataset.Source: §3.2 · p. 5

4 EXPERIMENTAL SETUP

Experiments use dynamic Kinetix tasks and the LIBERO Spatial manipulation benchmark, with separate correction-head implementations for each setting. The Kinetix head is an MLP, while the LIBERO head combines a transformer encoder with an MLP.

4.1.1 KINETIX

  • Kinetix tests responsiveness in 12 dynamic tasks spanning locomotion, grasping, and game-like environments, where delayed or inconsistent actions can quickly lead to failure.The study uses expert-policy demonstrations and evaluates Kinetix as a setting for inference-time methods such as RTC.Source: §4.1.1 · p. 5

4.1.2 LIBERO

  • LIBERO Spatial is a widely used benchmark for robot manipulation emphasizing spatial reasoning; for 3D manipulation benchmarking, it provides 432 episodes and 52,970 frames across 10 tasks.The dataset includes top- and wrist-camera RGB images at 256 × 256, an 8-dimensional state, and language instructions.Source: §4.1.2 · p. 5

4.2 MODEL TRAINING

  • The Kinetix correction head is a 3-layer MLP that combines state, base action, and a 2-dimensional sinusoidal chunk-position feature to predict a residual action.It has 512-unit hidden layers and 0.31M parameters; language instructions and base-policy latent representations are not used.Source: §4.2 · pp. 5–6, Figure 3 · p. 6
  • For LIBERO, SmolVLA supplies the base action chunk, while a transformer encoder processes visual, language, state, action, and time features for residual prediction.The encoder has 6 layers, and the full correction head has 32M parameters.Source: §4.2 · pp. 6–7, Figure 4 · p. 6

5 RESULTS

Across Kinetix and LIBERO Spatial, action-chunk correction improves success rates under inference delay and long execution horizons. The reported LIBERO results also show gains at a long horizon with no injected delay.

5.1 KINETIX

  • At delay d = 4, correction achieves nearly 35% higher success than the naïve baseline, and remains above 85% for horizons H = 7 in Kinetix.Across 12 tasks, correction maintains higher success rates across settings where naïve async and RTC degrade as delay or horizon increases.Source: §5.1 · p. 7

5.2 LIBERO SPATIAL

  • On LIBERO Spatial, A2C2 reaches 84% success versus 67% for the naïve baseline at H = 40 and d = 10, and 81.6% versus 72.2% at H = 50 and d = 0.The figure varies inference delay at fixed horizon and execution horizon with no delay; the table reports success rate.Source: §5.2 · pp. 7–8, Figure 6 · p. 8, Table 1 · p. 8

6 RELATED WORK

Prior work uses action chunks for faster, more coherent policy execution, but large VLA models and asynchronous chunk execution leave latency and chunk-mismatch challenges. Related approaches include hierarchical executors, RTC, and inference-speed optimizations.

Imitation learning and VLAs

  • Action-chunk policies, including diffusion, transformer-based ACT, and flow-based approaches, generate coherent sequences while supporting faster inference.VLA models extend multimodal task generalization but their larger computational footprint exacerbates latency in real-time control.Source: §6 · p. 8

Asynchronous chunk execution

  • Asynchronous server-client execution can preserve continuity by continuing the previous chunk during inference, but mismatches between chunks can cause jerky motion and performance degradation.The server returns a chunk after a delay, and the client executes its actions sequentially.Source: §6 · p. 9
  • RTC frames chunk switching as inpainting, freezing actions guaranteed to execute and filling in the remaining actions while the current chunk runs.Source: §6 · p. 9

Reducing inference latency

  • Faster inference methods and model or memory optimizations can reduce latency, but do not resolve the challenge when action generation remains slower than the control step.Source: §6 · p. 9

7 CONCLUSION

A2C2 augments large base policies with a lightweight correction head intended to preserve reactivity under inference delays and long execution horizons. Experiments in Kinetix and LIBERO Spatial report high success where naive execution or RTC degrades, while broader applicability remains to be validated.

7 CONCLUSION

  • A2C2 adds a lightweight correction head to large base policies to address reduced reactivity under inference delays and long execution horizons.The paper says the head is trained on the base policy's dataset and can, in principle, be added to off-the-shelf VLAs.Source: §7 · p. 9
  • Kinetix and LIBERO Spatial experiments show A2C2 maintaining high success rates where naive execution or RTC degrades significantly.Source: §7 · p. 9
  • Further validation is needed for richer language instructions, out-of-distribution settings, and more dynamic tasks beyond LIBERO Spatial.Source: §7 · p. 9
  • The authors frame the correction mechanism as a step toward scaled VLAs operating in real time without sacrificing responsiveness.Source: §7 · p. 9
  • The framework treats communication delay as inference latency and is described as extending to client–server architectures with remotely executed VLAs.Source: §7 · p. 9

ETHICS STATEMENT

The authors report that the work used no human subjects, personally identifiable information, or sensitive data, and that all experiments used publicly available datasets.

ETHICS STATEMENT

  • The work involved no human subjects, personally identifiable information, or sensitive data; experiments used publicly available datasets.Source: ETHICS STATEMENT · p. 10

A.1.1 ENVIRONMENT

The appendix describes the Kinetix and LIBERO Spatial environments, their training and evaluation procedures, and computational resources. It reports that A2C2 improves over naive execution in Kinetix and runs substantially faster per step than SmolVLA.

A.1.1 ENVIRONMENT

  • Kinetix uses 12 tasks, a 2722-dimensional non-image observation space, and a 6-dimensional action space.The action dimensions represent four motor controls and two thruster controls; unused entity and actuator entries are zero-padded.Source: §A.1.1 · p. 13, Figure 7 · p. 13

A.1.2 DATASET GENERATION AND TRAINING DETAIL

  • For Kinetix, experts were trained with RPO across eight seeds per task, then generated one million environment steps per task for flow-policy training.The correction head was trained using base-policy action chunks and the dataset; both training stages used AdamW.Source: §A.1.2 · pp. 13–14

A.1.3 EVALUATION DETAILS

  • Kinetix evaluation used 2048 rollouts per task and measured success across compatible combinations of inference delay and execution horizon.Source: §A.1.3 · p. 14

A.2.1 ENVIRONMENT

  • LIBERO Spatial has 10 tasks, and the appendix lists language instructions for those tasks.Source: §A.2.1 · p. 14

A.1.3 EVALUATION DETAILS

  • A2C2 consistently improves over the naive baseline in Kinetix; the table covers success rates across execution horizons and inference delays.Source: Table 4 · p. 15

A.2.2 DATASET AND TRAINING DETAIL

  • For LIBERO, SmolVLA was trained from scratch because its pretraining embodiment differed from the Franka arm; the correction head used a prepared dataset with base-policy outputs and VLM latents.The correction head used a 1e-5 constant learning rate, and AdamW was used for both training stages.Source: §A.2.2 · p. 15

A.2.3 EVALUATION DETAIL

  • LIBERO evaluation first tested delay–horizon combinations with 10 rollouts per task, then used 50 rollouts per task for three selected pairs: (0,10), (10,40), and (0,50).The appendix also states that experiment source code was released and reports training on NVIDIA RTX A6000 and H200 GPUs.Source: §A.2.3 · p. 15, §A.3 · p. 16, §A.4 · p. 16

A.5 INFERENCE TIME COMPARISON

  • 0.0047s versus SmolVLA’s 0.101s: the correction head’s average inference time per step is about 20× faster.Measurements used 100 trials per model on an NVIDIA RTX 5080 laptop GPU with 16GB VRAM.Source: §A.5 · p. 16

Related papers

Loading 2509.23224v1…