PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Execution of Action Chunking Flow Policies

Kevin Black, Manuel Y. Galliker, Sergey Levine

arXiv:2506.07339v2cs.ROcs.AIcs.LG

TL;DR

Large VLAs create latency challenges for real-time physical control, and action chunking does not eliminate pauses or discontinuities at chunk boundaries. RTC performs asynchronous inference-time inpainting by freezing actions guaranteed to execute and generating the remainder, improving throughput and robustness to delay across simulation and real-world evaluations.

  • Problem

    Large VLAs have high inference latency, while action chunking only partially addresses real-time control because delayed chunk transitions can become discontinuous and reduce reactivity.

  • Method

    RTC generates the next action chunk during execution of the current one, freezes delay-guaranteed actions, and inpaints the remaining actions for diffusion- or flow-based VLAs without retraining.

  • Results

    Across 12 dynamic simulation tasks and six real-world bimanual tasks, RTC is fast, performant, robust to inference latency, and improves task throughput.

  • Takeaways & Limitations

    RTC supports smooth asynchronous execution for action chunking policies across simulated and real-world control settings, including precise tasks under significant latency.

  • Takeaways & Limitations

    RTC adds significant computational overhead, applies only to diffusion- and flow-based policies, and lacks real-world evaluation in more dynamic settings such as legged locomotion.

Abstract

from arXiv · show

Modern AI systems, especially those interacting with the physical world, increasingly require real-time performance. However, the high latency of state-of-the-art generalist models, including recent vision-language action models (VLAs), poses a significant challenge. While action chunking has enabled temporal consistency in high-frequency control tasks, it does not fully address the latency problem, leading to pauses or out-of-distribution jerky movements at chunk boundaries. This paper presents a novel inference-time algorithm that enables smooth asynchronous execution of action chunking policies. Our method, real-time chunking (RTC), is applicable to any diffusion- or flow-based VLA out of the box with no re-training. It generates the next action chunk while executing the current one, "freezing" actions guaranteed to execute and "inpainting" the rest. To test RTC, we introduce a new benchmark of 12 highly dynamic tasks in the Kinetix simulator, as well as evaluate 6 challenging real-world bimanual manipulation tasks. Results demonstrate that RTC is fast, performant, and uniquely robust to inference delay, significantly improving task throughput and enabling high success rates in precise tasks $\unicode{x2013}$ such as lighting a match $\unicode{x2013}$ even in the presence of significant latency. See https://pi.website/research/real_time_chunking for videos.

1 Introduction

Real-time control is difficult because large VLAs incur high latency while action chunking only partially mitigates delayed, discontinuous behavior. RTC addresses this by generating future chunks asynchronously, freezing guaranteed actions and inpainting the remainder, with strong simulation and real-world results.

  • Motivation: Real-time interaction matters more for robots because physical environments evolve during model computation, making latency directly affect control outcomes.The paper contrasts latency in conversational systems with consequences such as spilling hot coffee in robotic control.
  • Motivation: Large VLAs are slow, parameter-heavy, and difficult to deploy on edge devices, while growing robot datasets may further increase model size.The paper identifies billions of parameters, heavy hardware requirements, and remote-inference overhead as sources of difficulty.
  • Motivation: Action chunking improves temporal consistency but sacrifices reactivity and can produce discontinuities or jerky behavior at chunk boundaries.Long execution horizons reduce responsiveness, whereas short horizons increase mode-jumping and discontinuous behavior.
  • Contribution: RTC generates the next action chunk while executing the previous one, freezing guaranteed actions and inpainting the remainder without retraining.The method applies at inference time to diffusion- or flow-based VLAs and requires no changes to existing training recipes.
  • Evaluation: The evaluation introduces 12 highly dynamic Kinetix tasks and tests six challenging real-world bimanual manipulation tasks.Using π0.5 as the base policy in the real world, the paper reports improved throughput and robustness to inference latency, including precise tasks such as lighting a match.

2 Preliminaries and Motivation

Action chunking trades responsiveness for temporal consistency, while delayed asynchronous transitions can create discontinuous, out-of-distribution actions. The paper motivates RTC’s overlap-aware freezing and inpainting strategy for maintaining continuity under inference delay.

  • Action chunking: An action chunk contains H future actions, but rollout executes only the first s actions, called the execution horizon.The policy conditions chunk At on observation ot, and s is typically shorter than H but greater than 1.
  • Action chunking: Long execution horizons reduce responsiveness, whereas short horizons increase mode-jumping and jerky behavior from discontinuities between chunks.Chunked execution provides temporal consistency at the expense of reactivity.
  • Inference delay: Real-time control requires a response within one controller timestep, but modern VLAs often take too long to generate an entire action chunk.The paper defines Δt as the controller sampling period and δ as chunk-generation time; real-time operation is trivial only when δ ≤ Δt.
  • Inference delay: Starting asynchronous inference before the execution horizon ends can guarantee availability, but the new chunk may switch strategies at the transition and create severe discontinuities.The problem worsens as inference delay increases, producing jerky behavior and out-of-distribution accelerations.
  • Inference-time inpainting: Figure 3 assigns full guidance to guaranteed actions, exponentially decreasing guidance to updateable overlap, and fresh generation to the final s actions.With d = 4 and s = 5, the regions are a0:3, a4:10, and the final five actions, subject to d ≤ s ≤ H − d.

3 Real-Time Chunking via Inpainting

RTC frames asynchronous action-chunk execution as an inpainting problem: it preserves the already committed prefix while generating a compatible continuation from new observations. Soft masking extends continuity beyond the strictly frozen actions, and a background inference loop adapts execution to observed delays.

  • Inpainting formulation: RTC generates the next action chunk while the current one executes, freezing actions guaranteed to occur and inpainting the remaining actions for compatibility.The method is designed to incorporate new observations without disrupting continuity between chunks.
  • Inpainting formulation: The inpainting correction adds gradient-based guidance so the generated chunk matches a target formed from the previous chunk’s overlapping actions.The correction is applied during iterative diffusion or flow denoising and is computed as a vector-Jacobian product.
  • Soft masking: Soft masking assigns weight 1 to the first d actions, weight 0 to the final s non-overlapping actions, and exponentially decaying weights between them.The decay treats actions farther in the future as more uncertain and uses all H −s overlapping actions for stronger cross-chunk continuity.
  • Soft masking: Hard masking can mismatch the frozen region and produce faster direction changes, motivating the proposed soft masking extension.The comparison is reported in Figure 4.
  • Real-time system: Algorithm 1 runs inference in a background thread, conservatively estimates delay from a buffer, and swaps in the new chunk when it becomes available.The execution horizon is set to max(d, smin), while GETACTION continues returning actions at each controller interval.

4 Experiments

The experiments evaluate RTC against asynchronous and synchronous baselines in dynamic simulation and real-world bimanual manipulation, including increasing inference delays. RTC shows strong robustness to delay, improves simulated solve rates with shorter execution horizons, and achieves the best real-world throughput across tested delays.

  • Experimental Setup: The evaluation covers 12 highly dynamic Kinetix environments and 6 challenging real-world bimanual manipulation tasks using π0.5 VLA.The simulated benchmark uses 2048 rollouts per data point, while the real-world evaluation spans 480 episodes and 28 hours of robot execution.
  • Simulated Benchmark: RTC shows the most robustness to simulated inference delays, outperforming BID with a widening gap as delay increases.BID uses significantly more compute by sampling batches of 64 action chunks.
  • Simulated Benchmark: RTC performs better with decreasing execution horizons, indicating that continuity across chunks lets it exploit more frequent closed-loop corrections.Figure 5 reports strictly increasing performance with decreasing execution horizon for RTC and BID.
  • Simulated Benchmark: Soft masking improves performance at lower inference delays and execution horizons, while hard masking somewhat underperforms it.The difference is particularly apparent at smaller simulated delays.
  • Real-World Results: Neither TE variant runs with +100ms or +200ms injected latency because oscillations trigger the robot’s protective stop.The real-world setup measures RTC latency at 97ms versus 76ms for baselines before added network latency.
  • Real-World Results: RTC achieves the best average task throughput at every real-world inference delay, with statistically significant results at +100ms and +200ms.Average throughput combines the proportion of each task completed with episode duration.
  • Real-World Results: RTC completes real-world tasks faster than synchronous inference even after inference pauses are removed, and improves final score especially for light candle and bed making.Light candle is the most precision-sensitive task and the only task without retrying.

5 Related Work

Related work addresses action chunking, inference acceleration, inpainting, real-time planning, hierarchical VLAs, and bidirectional decoding. RTC differs by applying guidance-based inpainting to asynchronous action-chunk execution, while remaining complementary to some approaches.

  • Action Chunking, VLAs, and Cascade Control: Action chunking is a standard approach for visuomotor imitation learning, supported by expressive models including diffusion and flow matching.These models generate sequences of future actions rather than single controller actions.
  • Reducing Inference Latency: Inference-acceleration methods reduce computation but cannot lower inference cost below one forward pass, leaving a gap when that pass exceeds the controller’s sampling period.The paper positions asynchronous execution as necessary in that regime.
  • Inpainting and Guidance: RTC adapts pretrained diffusion or flow inpainting to real-time execution with soft masking and guidance weight clipping.Diffuser also uses diffusion inpainting for sequential decision-making, but without guidance-based inpainting.
  • Real-Time Execution: Prior real-time approaches parallelize execution and computation, but are not always applicable to imitation learning and do not leverage action chunking.Hierarchical VLA designs are described as orthogonal to RTC and introduce their own tradeoffs.
  • Bidirectional Decoding: BID enables closed-loop control with pretrained action-chunking policies through rejection sampling, but does not consider inference delay directly.The experiments report that BID underperforms RTC while using significantly more compute.

6 Discussion and Future Work

RTC demonstrates speed and performance under significant inference delays, but its scope and computational cost constrain where it can be applied.

  • Limitations: RTC adds significant computational overhead compared with methods that sample directly from the base policy.This is a practical cost of the inference-time framework.
  • Limitations: RTC applies only to diffusion- and flow-based policies, limiting its supported policy classes.The limitation concerns applicability rather than the reported experimental performance.
  • Future Work: The real-world evaluation covers challenging manipulation tasks but omits more dynamic settings such as legged locomotion.Legged locomotion is included in the simulated benchmark but not the real-world results.

NeurIPS Paper Checklist

The checklist responses indicate that the paper’s claims, experimental reproducibility, training details, statistical reporting, and compute documentation are addressed, with supporting materials carrying key implementation details.

  • The paper’s introduction and abstract accurately reflect its contributions, experiments, and stated limitations.
  • The paper discusses limitations involving policy scope, computational efficiency, and experimental shortcomings.
  • The paper provides a reproducible algorithm and supplemental code for simulated benchmark results, while real-robot runtime code and π0.5 training data remain proprietary.
  • All training and testing hyperparameters necessary to understand the results are provided in the supplemental material.
  • Error bars are reported and described wherever the experiments report means over multiple data points.
  • The paper provides compute details in the supplemental material for reproducing its experiments.

10. Broader impacts

The paper includes broader-impact and asset-documentation material, credits existing resources, and provides simulation reproducibility code, while not releasing certain proprietary data or runtime code.

  • The paper includes a broader-impacts statement addressing both positive and negative societal impacts.
  • The paper does not release data or models requiring misuse safeguards.
  • The authors credit π0.5 and Kinetix, and identify the Kinetix software license as MIT.
  • The full code for reproducing the simulated benchmark and its results is provided in the supplemental material.
  • The paper states that it does not involve crowdsourcing, human-subject research, or LLMs as important components of its core method.

A.1 Broader Impacts

The paper frames RTC as technology aimed primarily at household robots, with potential benefits from automating dangerous work and assisting disabled and elderly people, alongside possible harms.

  • The experiments primarily concern household robots, while the stated goal is to improve learned-policy speed and performance for control tasks.
  • The technology could automate dangerous and difficult jobs or assist disabled and elderly people.
  • Potential harms include military applications and displacement of physical labor.

A.2 The Necessity of Guidance Weight Clipping (β)

The guidance-weight clipping value β is introduced to stabilize inpainting for control settings with few denoising steps, where large guidance weights can cause divergence.

  • β clips the guidance weight and is necessary to keep it finite at τ = 0.
  • With few denoising steps, high guidance weights can produce diverging action chunks.
  • The authors note an alternative of starting denoising from τ > 0 but did not try it.

A.3 Latency Measurements

The latency measurements characterize RTC's computational overhead and compare its implementation against alternative inference-time methods under specified hardware and profiling conditions. Additional ablations identify design and deployment choices affecting latency.

  • Measurement protocol: Table 1 measures on-GPU neural-network inference over 10 calls after 5 warmup calls using bfloat16 precision and n = 5 denoising steps.The measurements use an NVIDIA RTX 4090 GPU.
  • Method comparison: RTC slows inference through backpropagation at each denoising step, while BID slows inference by sampling action batches.BID variants reduce overhead by removing forward contrast or sharing the π0 VLM backbone.
  • System components: The latency breakdown includes robot-computer image resizing and separates it from model inference on a workstation GPU.The mobile setup uses an Intel NUC, while the non-mobile setup uses an AMD Ryzen 9 7950X workstation.
  • Model latency: RTC incurs a 2.5x latency increase per denoising step compared with vanilla π0.5.The comparison comes from a single profiling trace on an RTX 4090 GPU.
  • Simulation workload: In simulation, training and evaluation use multiple H100 configurations, including 2048 trials per environment evaluated in approximately 5 minutes on 6xH100s.Expert-policy training uses 6 seeds across 12 environments and takes approximately 4 hours on 4xH100s.
  • Deployment: Real-world inference runs on a single NVIDIA RTX 4090 GPU in a workstation located in the same building as the robots.The policies are fine-tuned from π0.5, with each fine-tuning run taking approximately 24 hours on 8xH100s.
Loading 2506.07339v2…