PaperlayerView exact revision on arXiv

Source-linked AI summary

VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference

Jiaming Tang, Yufei Sun, Yilong Zhao, Shang Yang, Yujun Lin, Zhuoyang Zhang, James Hou, Yao Lu, Zhijian Liu, Song Han

arXiv:2512.01031v2cs.ROcs.AIcs.LG

TL;DR

VLAs deployed synchronously stall while waiting for inference, whereas asynchronous inference can misalign predictions with the robot’s later execution state. VLASH bridges this gap by rolling the robot state forward through the previous action chunk and conditioning on that future state. Across simulated and real-world evaluations, it improves asynchronous accuracy and reduces reaction latency by up to 11.8×, while action quantization provides additional speedups with minimal accuracy loss.

  • Problem

    Synchronous VLA deployment introduces action stalls and high reaction latency, while asynchronous inference suffers prediction-execution misalignment that can destabilize control.

  • Method

    VLASH estimates the robot’s execution-time state by rolling it forward through actions issued during inference, then conditions the VLA on that future state.

  • Results

    VLASH consistently outperforms asynchronous baselines in accuracy and achieves up to 11.8× reaction speedup over synchronous inference across simulated and real-world evaluations.

  • Takeaways & Limitations

    Action quantization further delivers 1.5–2.0× task completion speedup with minimal accuracy loss, while VLASH enables dynamic tasks such as human ping-pong rallies and whack-a-mole.

Abstract

from arXiv · show

Vision-Language-Action models (VLAs) are becoming increasingly capable across diverse robotic tasks. However, these models are typically deployed under synchronous inference, where the robot waits for model inference to complete before acting, and cannot perceive or respond to environmental changes during action execution. This not only introduces noticeable action stalls, but also significantly increases reaction latency, fundamentally limiting the applicability of VLAs to dynamic, real-time tasks. Asynchronous inference offers a promising solution to achieve continuous and low-latency control by enabling robots to execute actions and perform inference simultaneously. However, because the robot and environment continue to evolve during inference, a temporal misalignment arises between the prediction and execution intervals. This leads to significant action instability, while existing asynchronous methods either degrade accuracy or introduce runtime overhead to mitigate it. We propose VLASH, a simple yet effective method for asynchronous VLA inference that delivers smooth, accurate, and fast reaction control without architectural changes or additional runtime overhead. VLASH leverages the future execution-time state by rolling the robot state forward with the previous action chunk, thereby bridging the gap between prediction and execution. Experiments show that VLASH reduces reaction latency by up to 11.8x compared to synchronous inference and consistently outperforms all asynchronous baselines in accuracy. With action quantization, it further achieves 1.5-2.0x task completion speedup with minimal accuracy loss. Moreover, it empowers state-of-the-art VLAs such as $π_{0.5}$ to handle fast-reaction, high-precision tasks including playing ping-pong and playing whack-a-mole, where traditional synchronous inference fails. Code is available at https://github.com/mit-han-lab/vlash

I. INTRODUCTION

Synchronous VLA inference causes stalls and delayed reactions, while asynchronous inference removes stalls but introduces prediction-execution misalignment. VLASH addresses this gap through future-state-aware inference and achieves faster, accurate control across benchmarks and dynamic tasks.

  • Synchronous inference stalls the robot and prevents perception during execution, increasing reaction latency for fast, dynamic interactions.
  • Asynchronous inference executes the current action chunk while inferring the next, enabling continuous motion and more prompt reactions to environmental changes.
  • Temporal misalignment between prediction and execution causes unstable, inaccurate control because the robot and environment evolve during inference.
  • VLASH conditions inference on an estimated execution-time robot state obtained by rolling the state forward with the previous action chunk.
  • 30.5% accuracy improvement over naive asynchronous inference and up to 11.8× reaction speedup over synchronous inference are reported across simulation and real-world evaluations.With action quantization, VLASH further provides 1.5–2.0× task completion speedup with minimal accuracy loss.

III. BACKGROUND AND MOTIVATION

Action chunks are predicted over a horizon but only an execution prefix is applied before the next inference. Asynchronous execution shifts the application interval by inference latency, so actions generated for one state may reach a changed state and scene.

  • Action chunking policy: An action-chunking policy generates H future actions from the environment observation and robot state at controller timestep t.H is the prediction horizon.
  • Prediction and execution intervals: Only the first K actions, where K≤H, are executed before the next inference to preserve control accuracy.K is the execution horizon.
  • Prediction and execution intervals: Inference latency Δ>0 delays execution of the planned action prefix relative to its prediction interval.
  • Asynchronous inference and interval misalignment: Asynchronous inference runs the next policy computation while the previous action chunk continues executing.
  • Asynchronous inference and interval misalignment: The shifted execution interval applies the same actions after the robot and environment have changed, producing unstable and discontinuous behavior.

IV. VLASH

VLASH makes asynchronous VLA inference future-state-aware by estimating the robot state when the next action chunk will begin. It rolls the state forward through actions executed during inference and conditions the VLA on that estimate.

  • Future State Awareness: VLASH conditions the policy on the robot state at the beginning of the next execution interval rather than the current state.
  • Future State Awareness: The future execution-time state can be estimated because the actions executed during inference are already known.
  • Future State Awareness: VLASH rolls the current state forward under the remaining previous-chunk actions and feeds the rolled-forward state with the current observation into the VLA.For example, the method computes s3 = s1 + a1 + a2 in the illustrated case.

B. Fine-tuning with Offsets to States and Actions

VLASH fine-tuning offsets robot states and action targets while holding observations fixed, teaching VLAs to use future-state information. An efficient shared-observation implementation packs multiple offsets into one sequence and reuses observation tokens.

  • Offset training: Current VLAs may under-utilize robot state, although identical visual observations can correspond to different robot states and required actions.The method addresses this limitation by making future robot state meaningful for action selection.
  • Offset training: Temporal-offset augmentation pairs one observation with future state s_t+δ and the corresponding future action chunk.Offsets are sampled from a predefined range, while the visual input remains o_t.
  • Offset training: The same observation can map to different target actions across offsets, forcing the VLA to attend to state inputs rather than visual features alone.Training over multiple offsets supports deployment across varying inference delays.
  • Shared-observation implementation: The efficient implementation packs one shared observation with all offset branches and masks cross-offset attention while preserving observation-to-branch attention.This makes branches equivalent to separate examples while encoding the observation once.
  • Shared-observation implementation: Packing five offsets increases π0.5 sequence length by about 31%, from 650 to 850 tokens, while making effective training trajectories 5× larger.The approach reuses each observation across multiple offset targets in a single pass.

D. Action Quantization

Action quantization accelerates physical robot motion by grouping adjacent fine-grained actions into coarser macro-actions. Each macro-action summarizes q consecutive actions, with interpolation used for non-integer q.

  • Action quantization: Action quantization groups adjacent fine-grained actions into coarser macro-actions to accelerate robot motion.Asynchronous inference hides model inference behind execution, so physical speed becomes the limiting factor for dynamic tasks.
  • Action quantization: For quantization factor q, each macro-action summarizes q consecutive actions, such as â_0 = a_0 + a_1 + a_2 for q=3.For non-integer q, the trajectory is re-sampled using linear interpolation.

V. EXPERIMENTS

VLASH is evaluated on simulated and real-world benchmarks spanning general manipulation and dynamic interaction. Across these settings, it maintains strong accuracy under asynchronous inference and supports fast-reaction tasks.

  • Experimental scope: The evaluation covers general manipulation in LIBERO and real-world tasks, plus dynamic interaction in Kinetix and real-world ping-pong and whack-a-mole.The benchmark suite tests both accuracy preservation and fast reaction capabilities.
  • Kinetix: Kinetix evaluates success rates across 12 dynamic tasks, including throwing, catching, and balancing, under simulated delays from 0 to 4 steps.Each data point averages 1,024 rollouts, and future states are estimated by rolling the simulator forward with generated actions.
  • Kinetix: At a four-step inference delay, VLASH reaches 81.7% success versus 51.2% for Naive Async, a 30.5% accuracy improvement.VLASH remains robust as inference delay increases, while RTC degrades and Naive Async collapses under larger delays.
  • LIBERO: LIBERO evaluates π0.5 and SmolVLA across Spatial, Object, Goal, and Long sub-benchmarks under inference delays of 0 to 3 steps.The benchmark tests whether VLASH preserves accuracy across different VLA architectures and large delays.
  • LIBERO: On LIBERO, VLASH reaches 97.2% and 97.1% average SR for π0.5 at small delays versus 96.8% for Sync, and 79.1% for SmolVLA at delay 3 versus 79.0% for Sync.Accuracy decreases only slightly at larger delays while remaining competitive.

B. Real-World Evaluation

Real-world experiments deploy π0.5 and GR00T N1.6 on two robotic platforms across manipulation tasks and reaction-speed settings. VLASH achieves the highest success rates and preserves accuracy while accelerating execution through action quantization.

  • Experimental setup: The real-world evaluation deploys π0.5 and GR00T N1.6 on the Galaxea R1 Lite and LeRobot SO-101 platforms.Experiments measure success rate, the speed-accuracy trade-off under quantization, and reaction speed.
  • Experimental setup: The three manipulation tasks are pick and place, stacking, and sorting, with varying object positions across episodes.Each method is evaluated over 20 rollouts with up to three attempts per rollout.
  • Success rate and robustness: VLASH achieves the highest success rate across all tasks and quantization factors, outperforming synchronous, naive asynchronous, and RTC baselines.It is also more robust to action quantization than the baselines.
  • Success rate and robustness: On pick-and-place and sorting, q=2 matches or exceeds synchronous inference at q=1, enabling 2× speedup without accuracy loss.On stacking, VLASH at q=1.5 maintains accuracy comparable to Sync at q=1.
  • Success rate and robustness: Quantization factors q∈[1.5, 2] achieve 1.5–2.0× speedup without degrading accuracy compared with Sync.The largest accuracy-preserving q can be selected according to task precision requirements.

2) Reaction Speed:

Asynchronous inference reduces reaction latency by allowing action execution and inference to overlap, enabling VLASH to handle dynamic interactive tasks where synchronous inference largely fails.

  • Experimental Setup: Action quantization is set to q=2 for all methods because it enables faster robot motion suited to both dynamic tasks.The selected factor yields the best performance across all baselines.
  • Reaction Speed: 11.8× reaction-speedup is achieved by asynchronous inference over synchronous inference.The comparison uses maximum reaction latency, defined as the worst-case delay from an environmental change to the first informed executed action.
  • Dynamic Interactive Tasks: VLASH achieves a 55% single-return hit rate in ping-pong and an average whack-a-mole score of 28.8.Ping-pong results are reported over single-return trials, while whack-a-mole reports average successful hits per round.
  • Dynamic Interactive Tasks: Synchronous inference largely fails on ping-pong and whack-a-mole because of excessive reaction latency.Both tasks require rapid detection and response to dynamic targets or incoming balls.

C. Fine-tuning Efficiency

VLASH’s shared-observation fine-tuning is slower initially but reaches comparable accuracy while making each training step substantially faster.

  • C. Fine-tuning Efficiency: 3.26× per-step speedup results from encoding the shared observation once and reusing it across temporal offsets.This efficiency comparison uses the same effective batch size as standard fine-tuning.
  • C. Fine-tuning Efficiency: VLASH ultimately achieves comparable accuracy to standard fine-tuning despite slower early convergence.Both methods are evaluated under synchronous inference.
  • C. Fine-tuning Efficiency: VLASH does not hurt the model’s original synchronous performance in these experiments.The conclusion follows from evaluating both fine-tuning methods under synchronous inference.

Appendix I Experimental Details

The experiments use consistent fine-tuning settings and compare VLASH with synchronous and naive asynchronous inference across manipulation tasks, including fast-response games.

  • Experimental Details: The same fine-tuning hyperparameters are used for π0.5 and SmolVLA on LIBERO and real-world tasks.The settings are intended to balance training stability and convergence speed while preventing downstream overfitting.
  • Task Details: Ping-pong tests rapid reaction by requiring the robot to track and strike a human-served ball.The task demonstrates interactive rallies with a human player.
  • Task Details: Whack-a-mole tests fast detection and precise striking of randomly activated targets.Each round lasts up to 30 seconds and ends early after three missed strikes.
  • Experimental Details: The demonstrations compare synchronous inference, naive asynchronous inference, and VLASH across four real-world manipulation tasks.The tasks are ping-pong, whack-a-mole, pick-and-place, and folding clothes.
  • Demonstration Results: VLASH produces smoother motions and faster task completion than synchronous and naive asynchronous baselines in the video demonstrations.Synchronous inference shows action-stall stuttering, whereas naive asynchronous inference shows erratic movements from prediction-execution misalignment.

Appendix III Architectural Modifications

VLASH applies to existing VLA architectures without architectural modifications by offsetting state information during fine-tuning to learn temporal alignment.

  • Architectural Modifications: VLASH requires no architectural modifications because current VLA models accept robot state inputs.State information is offset during fine-tuning to account for inference delay.
  • Architectural Modifications: For π0 and SmolVLA, VLASH integrates through their state projection layers without changing the transformer architecture.These models embed proprioceptive state vectors into continuous representations before transformer processing.
  • Architectural Modifications: VLASH also works directly with π0.5 despite its text-token encoding of numerical state values.That encoding disrupts the inherent numerical structure of state values and makes state learning more challenging.
Loading 2512.01031v2…