PaperlayerView exact revision on arXiv

Source-linked AI summary

FASTER: Rethinking Real-Time Flow VLAs

Yuxiang Lu, Zhe Liu, Xianzhe Fan, Zhenya Yang, Jinghua Hou, Junyi Li, Kaixin Ding, Hengshuang Zhao

arXiv:2603.19199v3cs.ROcs.CV

TL;DR

Real-time VLA systems need to react quickly to environmental changes, but existing asynchronous methods primarily optimize smoothness and constant flow schedules delay all movement until sampling completes. The paper analyzes reaction timing and proposes FASTER, whose horizon-aware sampling and streaming execution reduce first-action latency while preserving trajectory quality.

  • Problem

    Existing real-time VLA methods primarily improve trajectory smoothness while overlooking latency in reacting to environmental changes.

  • Method

    FASTER combines a Horizon-Aware Schedule that prioritizes near-term actions with a streaming client-server interface for early action dispatch.

  • Results

    Real-robot experiments demonstrate improved reaction capability and reduced TTFA, including on resource-constrained GPUs.

  • Takeaways & Limitations

    FASTER provides a plug-and-play path toward real-time embodied intelligence by enabling single-step immediate-action generation while preserving overall trajectory quality.

  • Takeaways & Limitations

    The analysis models total inference latency as constant, and mixed-schedule fine-tuning can introduce a fine-tuning gap and distribution shift.

Abstract

from arXiv · show

Real-time execution is crucial for deploying Vision-Language-Action (VLA) models in the physical world. Existing asynchronous inference methods primarily optimize trajectory smoothness, but neglect the critical latency in reacting to environmental changes. By rethinking the notion of reaction in action chunking policies, this paper presents a systematic analysis of the factors governing reaction time. We show that reaction time follows a uniform distribution determined jointly by the Time to First Action (TTFA) and the execution horizon. Moreover, we reveal that the standard practice of applying a constant schedule in flow-based VLAs can be inefficient and forces the system to complete all sampling steps before any movement can start, forming the bottleneck in reaction latency. To overcome this issue, we propose Fast Action Sampling for ImmediaTE Reaction (FASTER). By introducing a Horizon-Aware Schedule, FASTER adaptively prioritizes near-term actions during flow sampling, compressing the denoising of the immediate reaction by tenfold (e.g., in $π_{0.5}$ and X-VLA) into a single step, while preserving the quality of long-horizon trajectory. Coupled with a streaming client-server pipeline, FASTER substantially reduces the effective reaction latency on real robots, especially when deployed on consumer-grade GPUs. Real-world experiments, including a highly dynamic table tennis task, prove that FASTER unlocks substantially improved real-time responsiveness for generalist policies, enabling rapid generation of accurate and smooth trajectories.

1 Introduction

Real-time VLA deployment requires addressing reaction latency in addition to trajectory smoothness. FASTER targets this bottleneck by accelerating near-term action sampling and streaming completed actions to the robot.

  • Existing asynchronous methods improve motion continuity but largely overlook reaction to environmental changes.
  • Reaction time is modeled as a uniform random variable governed by inference latency and inference-execution timing.
  • FASTER introduces a Horizon-Aware Schedule that allocates more aggressive sampling to near-term actions while preserving slower long-horizon denoising.
  • FASTER can produce the immediate action with one-step sampling without architectural modifications or additional training cost.
  • A streaming client-server interface dispatches early actions immediately while subsequent actions are refined in parallel.
  • Real-robot experiments report improved reaction capability and lower latency across consumer-grade GPU platforms.

2 Analysis on Action Chunking Policy Inference

The analysis separates reaction from trajectory smoothness and shows that action-chunking responsiveness depends on inference timing, execution horizon, and the time needed to produce the first action. It motivates TTFA as the key latency measure and identifies limits of asynchronous inference alone.

  • Action chunking predicts H future actions but commonly executes only s actions before triggering new inference.
  • Inference latency includes model, communication, preprocessing, postprocessing, memory I/O, and other overheads, modeled as constant for analysis.
  • Reaction time depends jointly on inference latency and cycle frequency rather than being a fixed constant.
  • Asynchronous inference can reduce expected reaction time by only 0.5 ∗∆tinfer relative to synchronous inference.
  • Reducing the execution horizon to smin increases inference frequency while ensuring the execution interval exceeds inference latency.
  • TTFA measures the earliest moment the robot can begin moving and isolates first-action generation as the direct responsiveness bottleneck.

3 Methodology

FASTER identifies why constant flow-sampling schedules delay action dispatch and introduces Horizon-Aware Scheduling to prioritize near-term actions while preserving later-horizon refinement.

  • Preliminaries: Flow-based VLAs initialize action chunks from Gaussian noise and progressively integrate a learned velocity field toward the predicted actions.Inference uses an ODE solver such as Euler with N sampling steps, typically N=10.
  • Constant Flow Sampling: Constant scheduling forces every action in a chunk to undergo the same denoising process before any action can be issued.This treats the immediate action like the most distant future action and creates a dominant latency bottleneck.
  • Pilot Study: Early actions exhibit lower straightness and smaller denoising variation, indicating that they can be integrated accurately with fewer sampling steps.The straightness metric assigns S(A)=0 to a perfectly straight path; smaller values correspond to paths closer to linear interpolation.
  • Horizon-Aware Schedule: Horizon-Aware Scheduling assigns index-dependent hit times, accelerating near-term actions while allocating more denoising to uncertain future actions.The schedule uses global progress ρj and action-specific hit times ui, with α controlling how hit times vary across the horizon.
  • Horizon-Aware Schedule: Setting u0=(N−1)/N makes the first action ready after one sampling step while later actions continue refining progressively.The first completed action can be dispatched immediately, enabling streaming output.
  • Fine-tuning: Mixed-schedule fine-tuning combines HAS and the original constant schedule to improve robustness to schedule variation across the action horizon.Each training sample uses HAS with probability p and the constant schedule with probability 1−p.
  • Reaction Capability: FASTER shortens TTFA from approximately ΔtVLM+NΔtAE to ΔtVLM+ΔtAE because the first action requires only one action-expert sampling step.This schedule is paired with action conditioning, whose denoised prefixes provide additional constraints for early actions.

4 Experiments

Experiments evaluate FASTER’s reaction speed and task performance on two GPUs, including dynamic table tennis and less reaction-critical manipulation tasks. FASTER accelerates reaction, improves or preserves task outcomes, and reduces completion duration relative to synchronous inference.

  • Reaction-speed analysis: FASTER achieves substantial reaction-performance acceleration across scenarios, including a 3× TTFA boost for X-VLA on RTX 4060.The authors attribute additional responsiveness to early stopping, which decreases the feasible inference interval through a smaller smin.
  • Reaction-speed analysis: FASTER surpasses both baselines in probabilistic reaction-time comparisons, with a larger advantage under resource constraints.For X-VLA, its reaction-time upper bound is lower than the baselines’ lower bound, establishing strict dominance.
  • Real-world table tennis: In table tennis, synchronous inference fails to respond to incoming balls, while FASTER reacts faster than all baselines at contact.Training-time RTC improves racket-swing smoothness and obtains moderately higher scores than naive asynchronous inference.
  • Additional tasks: Across Pick Beverage and Fold Towel, asynchronous methods outperform Sync, while FASTER achieves better or comparable completion scores.These results indicate that task performance depends on real-time physical interaction as well as action accuracy.
  • Additional tasks: FASTER and Training-time RTC reduce completion time relative to synchronous inference, which suffers frequent inter-chunk pauses.The shorter durations improve efficiency for downstream applications.

5 Conclusion

The paper identifies constant timestep scheduling as a bottleneck for real-time responsiveness and introduces FASTER to accelerate immediate actions. Horizon-aware sampling and streaming execution reduce reaction delays while preserving trajectory quality.

  • Conclusion: FASTER identifies the constant timestep schedule in flow-based VLAs as a key bottleneck of real-time responsiveness.The schedule forces uniform treatment of actions despite their different temporal roles.
  • Conclusion: FASTER uses a Horizon-Aware Schedule to generate the immediate action in one sampling step without compromising overall trajectory quality.The method adaptively accelerates action sampling rather than modifying the model architecture.
  • Conclusion: A streaming client-server interface with early stopping jointly reduces TTFA and speeds up closed-loop control.The approach is presented as a robust, general, and plug-and-play path toward real-time embodied intelligence, particularly on edge devices.

A Related Work

Related work spans VLA architectures, inference-efficiency techniques, asynchronous execution, diffusion acceleration, and streaming policies. FASTER differs by explicitly targeting immediate-action responsiveness without architectural modifications and complements other efficiency methods.

  • Vision-Language-Action Models: VLAs map multimodal observations and language instructions directly to low-level motor commands, supporting dexterous robotic manipulation.Early token-based methods were followed by continuous diffusion- and flow-matching-based action generation.
  • Real-Time VLAs: Real-time VLA research reduces latency through smaller backbones, backbone compression, faster decoding, and visual-token pruning.These approaches primarily target computational efficiency in model inference.
  • Real-Time VLAs: Asynchronous execution removes inter-chunk pauses but can introduce discontinuities and jerky motion when chunks switch.Prior methods therefore emphasize trajectory continuity and smoothness within asynchronous pipelines.
  • FASTER’s distinction: FASTER explicitly targets responsiveness by accelerating immediate-action sampling and requires no architectural modifications.This makes it orthogonal and complementary to techniques such as LLM compression and token pruning.
  • Diffusion Acceleration in VLAs: Diffusion acceleration methods distill or directly train one-step models, while streaming policies produce immediate actions in one step but repeatedly update observations.Repeated observation updates require a VLM-backbone forward pass at every step, creating substantial inference burden for VLAs.
  • Asynchronous pipeline analysis: Inference timing is quantized by the controller period, with the minimum feasible execution horizon defined as smin = ⌈∆tinfer/∆tctrl⌉.Because inference completion rarely aligns with controller boundaries, the next inference starts at a subsequent controller timestep.

C Additional Results in Pilot Study

The pilot study tests shorter prediction horizons and multiple models and datasets to examine whether action indices differ in sampling difficulty. It extends analysis beyond the default π0.5 setting.

  • Pilot-study scope: The pilot study compares H = 30 against the default H = 50 for π0.5 and evaluates X-VLA on LIBERO and CALVIN ABC.The additional settings test generalization across models and data sources.
  • Pilot-study findings: The study reports straightness and intermediate-clean-estimate differences across these settings, with larger variance in some straightness metrics.These results provide additional pilot-study evidence concerning sampling difficulty across action indices.

D Additional Methodological Details

The appendix details action-prefix conditioning, streaming execution, and real-robot evaluation procedures for FASTER. It specifies how actions are trained, sampled, transmitted, and assessed across three tasks.

  • Horizon-Aware Schedule with Action Conditioning: Action-prefix conditioning sets prefix timesteps to zero and shifts action indices so the immediate valid action reaches its hit time in one sampling step.During inference, the prefix represents overlapping actions from the previous chunk, with its length set to the real inference delay.
  • Training: Training samples prefix lengths uniformly up to dmax and masks the loss so it is computed only over suffix actions.The prefix length simulates varying TTFA across devices, while the mask excludes already-conditioned actions.
  • Streaming Interface: FASTER streams smaller action packets progressively, allowing later-action communication to overlap with execution of preceding actions.The controller avoids stalls when the next action arrives within one control period, although transmitting a full chunk can take longer.
  • Algorithms: The inference procedure computes horizon-aware local timesteps, updates actions with Euler integration, streams finalized actions, and stops after the execution horizon is complete.The training and inference algorithms explicitly implement schedule selection, masked learning, streaming, and early stopping.
  • Real-Robot Evaluation: Real-robot evaluation uses an AgileX Cobot Magic platform with four 6-DoF Piper arms, three cameras, and Table Tennis, Pick Beverage, and Fold Towel tasks.The evaluation uses 15, 35, and 10 trials respectively, with fine-grained scores averaged across sub-steps and rollouts.

E.2 Simulation Benchmarks

The real-world benchmark setup evaluates FASTER across VLA models, hardware platforms, and manipulation tasks, while measuring reaction capability through TTFA-based distributions. Results compare task performance and duration under controlled rollout protocols.

  • Benchmark Setup: LIBERO contains four suites of ten tasks, while CALVIN contains 34 tasks evaluated through 1,000 five-task instruction chains.A single policy is trained across LIBERO suites, and CALVIN performance is measured by average successfully completed tasks per chain.
  • Benchmark Setup: Kinetix provides 12 dynamic simulation tasks suited to varying-delay evaluation, but uses a 4-layer MLP rather than a VLA model.Consequently, Kinetix is treated as supplementary rather than the primary VLA evaluation.
  • Implementation: The real-world setup uses π0.5 and X-VLA with default configurations, fine-tuning on eight NVIDIA A800 GPUs and setting HAS parameters by model and benchmark.Both VLAs use ten flow-matching sampling steps, with u0 selected so the immediate action requires one step.
  • Implementation: Real-robot inference uses a 30Hz controller, defines delay as floor(∆tinfer/∆tctrl), and sets the minimum execution horizon as ceil(∆tinfer/∆tctrl).Table Tennis uses the minimum horizon for reaction capability, whereas other tasks use longer horizons for smoother trajectories.
  • Real-World Results: FASTER’s reaction-time distribution is uniformly bounded, with upper bounds of 111.5ms versus 113.7ms on RTX 4090 and 329.2ms versus 399.5ms on RTX 4060.The comparison is between FASTER and the baselines’ lower-bound-separated reaction ranges.
  • Real-World Results: On Pick Beverage and Fold Towel, FASTER achieves better or comparable completion scores and durations than X-VLA despite X-VLA’s lower baseline performance.Naive Async often fails because of unstable actions caused by inter-chunk discontinuities.

F.3 Simulation Benchmarks

Simulation benchmarks test whether FASTER preserves task performance while reducing reaction latency. HAS remains competitive on LIBERO and CALVIN, and FASTER outperforms real-time baselines on Kinetix under a matched inference budget.

  • LIBERO and CALVIN: HAS maintains competitive LIBERO and CALVIN performance with limited degradation despite aggressive action sampling.These benchmarks provide controlled tests of whether the original model performance is preserved without real-world inference-latency effects.
  • Kinetix: 5× reduction in immediate-action inference latency is achieved on Kinetix, where FASTER is compared with baselines under the same wall-clock inference budget.The benchmark uses d = 4 for baselines and d = 1 for FASTER with the same execution horizon.
  • Kinetix: FASTER outperforms all included Kinetix baselines by a clear margin under matched inference conditions.The comparison includes BID, Inference-time RTC, VLASH, and REMAC.

F.4 Ablation Study

Ablations examine the Horizon-Aware Schedule parameter, mixed training schedule, and prediction-error trade-off. The results support robust α choices, while showing that mixing HAS with the constant schedule is important for long-horizon performance.

  • Horizon-Aware Schedule: HAS remains robust across α values, with the largest Avg. Len difference only 0.18 except at α = 1.0.Smaller α values make hit times decay faster and allocate more denoising steps to future actions.
  • Mixed Schedule: Small mixing probabilities p degrade performance because the inference-time schedule appears less frequently during training.Training primarily with HAS can also harm long-horizon performance, motivating the default mixed schedule p = 0.5.
  • Real-World α Study: Real-world experiments show no clear performance differences across α values for two models, suggesting task-specific HAS tuning is unnecessary in these settings.The reported default configuration generalizes across the tested settings.
  • Error Analysis: Prediction error generally increases with action index, while FASTER produces moderately higher errors on some dimensions distributed more uniformly across chunk indices.The analysis characterizes an accuracy-latency trade-off introduced by accelerated sampling.

G Limitations and Future Work

FASTER remains bounded by model, hardware, timing-model, and responsiveness–accuracy limitations. Its controlled-setting results also leave deployment safety and broader-impact risks requiring further assessment.

  • Method scope: FASTER applies primarily to flow- or diffusion-based VLAs with iterative sampling, and practical latency reduction depends on implementation and hardware.In JAX, forward-pass runtime is not always proportional to iteration count, which may limit speedup.
  • Inherited model limitations: FASTER does not resolve perception failures, language-grounding errors, or inaccurate actions inherited from the underlying VLA model.The robot may still fail when the base policy produces incorrect task actions.
  • Timing assumptions: The uniform reaction-time model is a first-order characterization that assumes sensing, communication, preprocessing, inference, and dispatch share one effective latency.Actual latency can vary with CPU/GPU scheduling, network jitter, memory contention, and operating-system overhead.
  • Future work: Horizon-Aware Scheduling trades responsiveness against action accuracy, especially for tasks requiring precise long-horizon coordination.Aggressive early sampling may slightly perturb the original generation trajectory; uncertainty- or feedback-dependent schedules are proposed as future improvements.
  • Broader impacts: More reactive robots may amplify failures from incorrect predictions, misinterpreted instructions, or out-of-distribution observations, while broader automation raises misuse and labor-displacement concerns.Experiments were conducted in controlled research settings, and safety-critical or unsupervised human-facing deployment requires additional assessment, monitoring, and safeguards.
Loading 2603.19199v3…