PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Real-Time VLAs: Stage-Aware Two-Step Flow Denoising and System-Level Evaluation

Di Wu, Rongtian Shen, Ping Liu, Yan Shen, Zhenhan Yin, Shun Zuo, Xuhua Chen, He Zheng, Lingfeng Zhang, Jianglin Zhang, Tao Zhang

arXiv:2609.39822v1cs.RO

TL;DR

VLA inference and robot execution operate at mismatched time scales, with timing shaped by both model computation and system delays. The paper measures these sources, proposes two-step non-uniform denoising, and builds a distributed real-time evaluation framework. In garment folding, Legato and Temporal Smoothing lead their respective method groups, while combining them with the denoising strategy reduces inference cost.

  • Problem

    VLA models update more slowly than high-rate robot control, while perception, communication, scheduling, and physical response also contribute to end-to-end timing.

    Source: §1 · p. 3, §6 · p. 15

  • Method

    The paper measures model and robot-side latency, introduces two-stage non-uniform Flow Matching denoising, and develops a distributed framework with independently configurable inference, action-publication, and control rates.

    Source: §1 · pp. 3–4

  • Results

    On long-horizon garment folding, Legato and Temporal Smoothing provide the strongest overall performance among training-based and training-free methods, respectively; combining them with two-stage sampling yields inference speedups of 1.676× and 2.804×, respectively.

    Source: §7 · p. 19

  • Takeaways & Limitations

    Real-time VLA performance is jointly constrained by model computation and system latency, motivating joint optimization of inference and execution scheduling.

    Source: §7 · p. 19

    The study focuses on VLA; extending the non-uniform Flow sampling approach to Flow Matching-based world–action models remains prospective.

    Source: §7 · p. 19

Abstract

from arXiv · show

Vision-language-action (VLA) models face a timing gap between low-rate inference and high-rate robot execution. We characterize this gap through end-to-end latency measurements of model inference and the robot execution chain. Repeated Flow Matching denoising contributes substantially to inference cost, while robot-side delays mainly arise from perception acquisition, communication scheduling, and physical response. Analysis of the velocity field shows relatively stable magnitude and direction in early integration, followed by stronger directional correction near the terminal steps. Based on this stage heterogeneity, we propose two-stage non-uniform denoising, reducing the number of steps from 10 to 2 and model-inference time from 61.557 ms to 21.956 ms. We also develop a distributed real-time VLA framework with independent inference, action-publication, and robot-control rates, modular observation acquisition, and action-provenance logging. Using {\pi}0.5 as the baseline, we evaluate six real-time execution methods on a long-horizon physical garment-folding task. Legato performs best overall among training-based methods, while Temporal Smoothing leads among training-free methods; both perform strongly in task success, completion time, action continuity, and acceleration smoothness. Combining two-step denoising with representative execution methods substantially reduces inference cost with a small reduction in task performance. These results motivate joint optimization of model-inference efficiency and robot-system timing.

1 Introduction

VLA systems face a timing mismatch between low-rate inference and high-rate robot control, compounded by delays across perception, communication, scheduling, and physical response. The paper characterizes these sources, proposes two-step non-uniform Flow Matching denoising, and develops a distributed framework for evaluating real-time execution methods.

1 Introduction

  • Prior work addresses generation efficiency, action continuity and state alignment, and joint optimization of perception, computation, and execution.Approaches include fewer sampling steps, action-prefix conditioning, observation-based revision, and sensor timing calibration.Source: §1 · p. 3
  • The paper identifies inconsistent evaluation settings, insufficient characterization of Flow Matching integration stages, and a lack of unified end-to-end latency measurement as remaining gaps.Without unified latency measurement, it is difficult to distinguish the effects of different latency sources.Source: §1 · p. 3
  • The paper measures model-side and robot-side latency, analyzes Flow Matching stages, and evaluates six execution methods on long-horizon bimanual garment folding using π0.5.The evaluation uses common training data, a shared robot platform, and standardized runtime configurations.Source: §1 · p. 3
  • The two-stage non-uniform strategy reduces Flow Matching denoising from 10 steps to 2 and mean inference time from 61.557 ms to 21.956 ms, a 2.804× speedup.It combines standard Flow long-interval generation with SnapFlow short-interval terminal refinement.Source: §1 · p. 3
  • The distributed framework decouples observation acquisition, inference, action publication, temporal alignment, and local control, with independent inference, publication, and robot-control rates.It also provides runtime logging for traceable deployment and evaluation.Source: §1 · p. 4

2 Sources of End-to-End Latency and Model-Inference Bottlenecks

The paper treats real-time VLA timing as a combination of model computation and delays across the robot execution chain. It measures non-model latencies and decomposes inference time into fixed and repeated sampling costs.

2 Sources of End-to-End Latency and Model-Inference Bottlenecks

  • Real-time VLA performance is jointly constrained by model computation and multiple sources of system latency.The latency analysis separates delays in perception, communication, scheduling, and physical execution from visual-language computation and Flow Matching generation.Source: §2 · p. 4

2.1 End-to-End Timeline and Non-Model Latency

  • The closed-loop timeline runs from image formation and state change through readout, feedback, preprocessing, transmission, inference, action publication, and physical response.The host-receipt-to-new-action interval includes preprocessing, request upload, model inference, download, and queueing; it excludes earlier perception latency and later physical response.Source: §2.1 · p. 4
  • A calibration procedure uses visual time codes, marker trajectories, joint commands, and state feedback to estimate image-readout, motion-response, and proprioceptive-feedback delays.Differences between host receipt and camera timestamps estimate readout latency; phase or correlation relationships separate motion response from feedback latency.Source: §2.1 · p. 4, Figure 2 · p. 5
  • 17.690 ms camera timestamp offset, 32.817 ms image-readout latency, 26.4±6.2 ms proprioceptive-feedback latency, and 15.3±5.4 ms command-to-motion response latency were measured on the physical robot system.The paper concludes that observations available to the controller are temporally misaligned with physical robot state, so faster inference alone cannot eliminate the mismatch.Source: Table 1 · p. 4, §2.1 · p. 5

2.2 Model-Inference Time Decomposition

  • Inference time combines fixed costs independent of NFE with repeated Flow sampling costs that increase with NFE.The repeated step cost includes action-head evaluation and numerical integration, while conditioning and output transformation are also part of the inference path.Source: §2.2 · p. 5
  • 61.557 ms was the average time for standard 10-NFE Flow across 620 inference calls on an RTX 4090 D in JAX, excluding the first JIT compilation.The paper uses this model-side measurement to motivate analysis of sampling stages and reduced-NFE integration.Source: §2.2 · p. 5

3 Two-Stage Non-Uniform Denoising

The section identifies stage-dependent behavior in Flow Matching denoising and uses it to motivate a two-stage schedule that pairs long-interval generation with terminal refinement. It describes the standard sampling process, the non-uniform partition, and training that combines Flow Matching with SnapFlow refinement.

3.1 Stage Heterogeneity of the Denoising Velocity Field

  • Near the endpoint, velocity magnitudes change more and all six trajectories show larger directional corrections, motivating long-interval generation followed by short-interval refinement.Early and intermediate stages have broadly stable velocity RMS and small angles between adjacent velocity fields.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 Flow Matching Action Generation and Standard Sampling

  • Standard Euler sampling moves a conditioned action chunk through a decreasing time grid, requiring one action-expert evaluation per integration step.The condition comprises three images, a language instruction, and robot state.Source: §3.2 · pp. 6–7

3.3 Two-Stage Non-Uniform Partition

  • With τ = 0.3, the schedule uses a 0.7 interval for coarse generation and a 0.3 terminal interval for refinement; fivefold fewer evaluations do not imply fivefold faster execution.The first stage takes a large Euler step, and the second predicts mean velocity to the endpoint; inference uses two action-expert evaluations.Source: §3.3 · p. 7, Figure 4 · p. 7

3.4 SnapFlow Terminal Refinement and Joint Training

  • The method retains standard Flow Matching for long-interval generation and confines SnapFlow [28] refinement to the short terminal interval rather than compressing the full interval.The resulting schedule is 1 →τ →0, with τ = 0.3 in the current implementation.Source: §3.4 · pp. 7–9
  • Training selects standard Flow Matching or the two-stage branch with equal probability; stop-gradient prevents the terminal-refinement loss from backpropagating through the first-stage prediction.The terminal target is constructed from shortcut supervision, and the SnapFlow terminal-refinement loss weight is λsc = 0.1.Source: §3.4 · p. 8

4 Distributed Real-Time Inference and Execution Framework

The framework separates observation acquisition and robot control from GPU policy inference, while supporting pluggable execution strategies and runtime tracing. Its logs record action provenance and timing to reconstruct handovers and help localize errors by system layer.

4.1 Overall Architecture

  • A robot-side Collector/RobotIO continuously caches observations, while a configured-rate inference thread sends requests to a GPU policy server and receives action chunks with latency metadata.Cross-host requests use TCP or WebSocket; same-host deployment can use shared memory.Source: §4.1 · p. 9, Figure 5 · p. 9

4.2 Pluggable Strategy Integration and Runtime Tracing

  • The framework integrates six execution strategies through shared observation, communication, buffering, and control paths, while logs capture action provenance and timing for handover reconstruction and error localization.The experiments demonstrate physical execution and trajectory traceability for all six methods.Source: §4.2 · p. 9, Table 2 · p. 10

5 Experimental Evaluation and Analysis

The experiments compare six real-time execution methods on physical bimanual garment folding and evaluate their task performance, action continuity, and inference efficiency. Legato and Temporal Smoothing lead in their respective method groups, while two-stage denoising substantially reduces inference time with some closed-loop performance loss.

5.1 Distributed Inference and Execution Experiments

  • The comparison uses π0.5 on a long-horizon bimanual T-shirt-folding task, with 30 trials per method across three garment conditions and metrics spanning task performance, continuity, latency, and tracking.Six methods share a common software and hardware setup; task success criteria are consistent across methods.Source: §5.1 · p. 10, §5.1.1 · p. 10, §5.1.2 · p. 10

5.1.4 Comparative Analysis

  • Legato leads all three task metrics, while Temporal Smoothing is strongest among training-free methods on garment folding.Legato achieves 96.7% success, 73.56 s mean completion time, and 47.31 h−1 throughput; Temporal Smoothing reaches 76.7% success and 29.38 h−1 throughput.Source: Table 3 · p. 12, §5.1.4 · p. 12

5.1.5 Action-Chunk Continuity

  • Legato has the lowest Tail Gap and Switch Gap, while Temporal Smoothing has the lowest maximum acceleration, showing that continuity and local smoothness differ.Legato's gaps are 0.0193 rad and 0.0062 rad; Temporal Smoothing's maximum acceleration is 25.6 rad/s2.Source: Table 4 · p. 12, §5.1.5 · pp. 12–13

5.1.6 Local Action-Chunk Trajectories

  • Local trajectories reveal distinct handover impulses: Naive Asynchronous jumps sharply, while insufficient prefix or delay alignment also produces discontinuities in both RTC methods.Naive Asynchronous reaches 744.09 rad/s2 maximum acceleration; Inference-time RTC and Training-time RTC reach 65.55 and 197.16 rad/s2, respectively.Source: §5.1.6 · pp. 13–14

5.2.1 Experimental Analysis

  • 21.956 ms versus 61.557 ms: two-stage denoising reduces inference time by 64.33% while cutting NFE from 10 to 2.The measured speedup is 2.804×.Source: Table 5 · p. 14, §5.2.1 · p. 14
  • Under fixed offline replay on two validation sequences, 2-NFE sampling retains action errors at the same order of magnitude as standard 10-NFE Flow.Joint-space and bimanual TCP errors trade advantages across sequences and metrics.Source: Table 6 · p. 14, §5.2.1 · p. 14

5.2.2 Combination with Real-Time Execution Mechanisms

  • Both Legato and Temporal Smoothing combinations reach 21.956 ms inference time, but success rates fall and continuity metrics increase relative to their baselines.Success falls from 96.7% to 86.7% and from 76.7% to 70.0%; all three continuity metrics rise by approximately 20%.Source: §5.2.2 · p. 14, Table 7 · p. 15

6 Literature Review

The review organizes real-time VLA research around six technical areas, from asynchronous alignment to system-level deployment. Across these areas, methods modify different components of generation, execution, representation, and correction.

6.1 Asynchronous Inference and Execution-Time Alignment

  • Asynchronous methods align action indices, predict handover conditions, or differentiate generation within a chunk, but prefix alignment alone does not compensate for robot-state changes during inference.RTC constrains committed prefixes; VLASH and FutureRTC predict handover conditions, while REMAC and AsyncVLA address within-chunk generation.Source: §6.1 · pp. 15–16

6.2 Inter-Chunk Continuity and Action Priors

  • Inter-chunk continuity methods range from selecting or guiding chunks to learning continuity in generation and using action-history priors.History-based priors can propagate errors after a faulty chunk or abrupt environmental change.Source: §6.2 · pp. 16–17

6.3 Adaptive Execution Horizons and Multi-Rate Control

  • Adaptive-horizon methods use uncertainty, motion phases, generation stability, or attention to set replanning boundaries, while multi-rate methods coordinate action horizons and control rates.Contact-oriented approaches further coordinate low-rate vision with higher-rate tactile or force feedback.Source: §6.3 · p. 17

6.4 Fast Action Generation and Streaming Inference

  • Fast-generation research reduces sampling steps or releases useful near-term actions earlier through horizon-aware scheduling, streaming, or speculative inference.These approaches target different goals, from completing a chunk faster to exposing the next useful action sooner.Source: §6.4 · p. 17

6.5 Continuous-Time Action Representations and Physical Executability

  • Continuous-time representations compress or structure action outputs, while geometry- and dynamics-aware methods also consider whether robots can execute actions at higher rates and speeds.Examples include spline-based actions, robot-state manifolds, and reachability-constrained chunk execution.Source: §6.5 · p. 18

6.6 Execution-Time Correction and System-Level Deployment

  • Execution-time correction adds feedback or verification after generation, and deployment work coordinates model computation with sensing, communication, and robot execution.Because methods modify different components, the paper's physical comparison focuses on six methods sharing the same base policy, action representation, and robot interface.Source: §6.6 · pp. 18–19

7 Conclusion

The paper finds that real-time VLA performance is constrained by model computation and system latency. It introduces stage-aware two-stage sampling and a distributed execution framework, then evaluates both on physical garment folding.

7 Conclusion

  • Real-time VLA performance is jointly constrained by model computation and multiple sources of system latency.The analysis covers model computation and the robot execution chain.Source: §7 · p. 19
  • 21.956 ms versus 61.557 ms: two-stage non-uniform sampling reduces Flow Matching inference time by 2.804×, using stable early and intermediate fields followed by stronger terminal directional correction.The schedule 1 →0.3 →0 reduces NFE from 10 to 2.Source: §7 · p. 19
  • The distributed framework independently configures inference, action-publication, and robot-control rates and supports six execution methods on physical robots.It coordinates observation acquisition, inference, action buffering, and runtime records.Source: §7 · p. 19
  • Legato and Temporal Smoothing perform best among training-based and training-free methods, respectively; combining them with two-stage sampling yields speedups of 1.676× and 2.804×.The study focuses on VLA, while future work will investigate joint optimization of inference and execution scheduling.Source: §7 · p. 19

A Task Performance Across Garment Conditions

The appendix breaks down the 180 physical trials by garment condition to show how execution methods behave across objects.

A Task Performance Across Garment Conditions

  • Temporal Smoothing completes 6/10–9/10 tasks across conditions, with throughput of 22.86–34.24 h−1.The appendix reports successful trials, mean completion time, and successful-task throughput for each garment condition.Source: A Task Performance Across Garment… · p. 24
  • Legato and VLASH complete 9/10–10/10 tasks across all three garment conditions, while both RTC methods complete only 4/10 tasks on the medium red garment.Legato maintains throughput of at least 41.01 h−1; VLASH reaches 63.83 h−1 on the small purple garment.Source: Table S1 · p. 24, A Task Performance Across Garment… · p. 24

B Training Configuration

The appendix identifies the training settings retained for reproducibility and compares predicted Flow Matching velocities with their ground-truth scale.

B Training Configuration

  • Table S2 retains training and reproducibility settings while omitting platform-management parameters.Logging frequency, checkpoint intervals, and runtime paths are omitted.Source: B Training Configuration · p. 24, Table S2 · p. 24
  • Table S2 presents the training configuration for two-stage non-uniform denoising.Source: Table S2 · p. 24

C Scale Consistency of the Flow Matching Velocity Field

  • The scale-consistency analysis compares predicted and Flow Matching ground-truth velocity RMS using the component-wise RMS defined in Eq. (3).It complements the main-text analysis of stage-dependent velocity magnitude and directional correction.Source: C Scale Consistency of the Flow… · p. 25

C.1 Scale Comparison with Flow Matching GT Velocity

The appendix compares predicted velocity scale with Flow Matching supervision across denoising steps and adds offline action-accuracy material. Predicted and ground-truth velocity RMS remain closely matched, with terminal updates mainly involving small velocity adjustments.

C.1 Scale Comparison with Flow Matching GT Velocity

  • The velocity target is evaluated at three progress positions using five fixed noise seeds per position on one representative validation sequence.For a fixed action–noise pair, the target is unchanged across denoising steps and is a sample-level supervision target, not dataset-action magnitude.Source: §C.1 · p. 25
  • 0.775% mean absolute relative deviation across 15 trajectories and 10 steps indicates comparable predicted and ground-truth velocity RMS throughout integration.At step 10, 14 trajectories are slightly below ground truth; mean signed deviation is −1.813%, with values from −4.941% to 0.127%.Source: §C.1 · p. 25

D Offline Action Accuracy of Two-Stage Non-Uniform Denoising

D.1 Comparison and Evaluation Protocol

The offline comparison evaluates two-stage denoising against standard 10-NFE inference on a fixed validation sequence. It reports faster inference, lower two-stage P95 and maximum absolute errors, and broadly retained recorded-action trends.

D.1 Comparison and Evaluation Protocol

  • The fixed validation sequence has 376 frames over 12.5 s, with each frame independently using its recorded observation.Errors are aligned between the model’s first 14-dimensional action and the recorded action; predicted state is not recursively passed forward.Source: §D.1 · p. 25
  • 61.571 ms versus 21.923 ms mean inference time on 375 samples gives two-stage 2-NFE a 2.808× speedup and 64.39% time reduction over standard 10-NFE.Source: §D.1 · p. 26

D.2 Per-Joint Error and Frame-Wise Variation

  • 0.018704 rad P95 and 0.052449 rad maximum absolute error for the two-stage model are both lower than the standard model’s.Figure S2 reports per-joint and gripper MAE as well as frame-wise mean absolute error over 12 rotational joints.Source: §D.2 · p. 26, Figure S2 · p. 26

D.3 Fourteen-Dimensional Per-Joint Action Trajectories

  • Two-stage predictions broadly retain the recorded-action trends across the 14 joint and gripper dimensions.Figure S3 distinguishes recorded actions from standard 10-NFE and two-stage 2-NFE predictions and reports per-dimension MAE for both models.Source: §D.3 · p. 26, Figure S3 · p. 27

D.4 Bimanual TCP Trajectories and Pose Errors

The appendix compares bimanual TCP pose errors for two-stage and standard denoising, and supplies additional execution-mode and local-trajectory figures.

D.4 Bimanual TCP Trajectories and Pose Errors

  • Two-stage denoising lowers left-arm TCP rotation MAE from 1.047° to 0.911°, while raising right-arm TCP rotation MAE from 0.902° to 1.169°.Figure S4 also presents bimanual trajectories, translation errors, and SO(3) geodesic orientation errors.Source: §D.4 · p. 28, Figure S4 · p. 28
  • The bimanual comparison uses three-dimensional left and right TCP trajectories, translation error, and orientation error panels.Figure S4 distinguishes recorded-action forward-kinematics trajectories from the standard and two-stage models.Source: Figure S4 · p. 28

E Additional Execution Figures

  • Supplementary figures show reference execution modes, representative delay-aware methods, and local left_j3 trajectories for four execution methods.The local trajectories are shown for Temporal Smoothing, Inference-time RTC, Training-time RTC, and VLASH.Source: Figure S5 · p. 29, Figure S6 · p. 29, Figure S7 · p. 30

Related papers

Loading 2609.39822v1…