Source-linked AI summary
Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency
Brian Zhu, Momen Khalil, E Harrison, Emanuele Poggi, Philipp Schmitt, Bernd Kast, Philine Meister, Pranav Atreya, Qiyang Li, Finn Ferchau, Cesar Colmenero, Yash Shahapurkar, Gokul Narayanan, Melih Erdogan, Kai Wurm, Georg von Wichert, Oier Mees, Eugen Solowjow, Andrew Wagenmaker, Sergey Levine
TL;DR
Large generalist policies can require substantial inference time, making latency a challenge for RL improvement and reactive deployment. ARLI combines asynchronous inference with committed-action and intermediate-state conditioning to support finetuning under these delays. It achieves effective improvement in simulated and real-world manipulation tasks, while assuming a compatible pretrained-policy decomposition and recognizing that RTC may reduce reactivity.
Problem
Large generalist policies can require substantial inference time, causing pauses or jerky movements and disrupting RL under latency.
Method
ARLI combines asynchronous inference with RL state augmentation using intermediate actions and a more up-to-date intermediate observation.
Results
ARLI enables effective generalist-policy improvement under practical latency constraints in simulated tasks and three challenging real-world bimanual tasks.
Takeaways & Limitations
Asynchronous inference with properly augmented RL observations can support policy learning while mitigating inference-latency effects.
Takeaways & Limitations
ARLI assumes the pretrained policy can be split into VLM-backbone and action-expert components, and RTC may reduce reactivity by forcing old actions to play.
Abstract
from arXiv · showhide
While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses a fundamental obstacle to effective RL improvement. In particular, their severe inference latency---which can lead to pauses or jerky movements---can alter the effective environment dynamics and, if not correctly accounted for, break the Markov assumption that RL relies on, causing standard RL algorithms to fail completely. In this work, we introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays. Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL by providing a low-latency RL policy design that maximizes reactivity within the inference window through two contributions: state augmentations that restore near-Markovian structure by incorporating committed actions and a mid-inference observation. We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely, even matching or exceeding the performance of standard RL in idealized no-latency settings.
1 Introduction
Generalist policies offer opportunities for online RL improvement, but their inference latency can disrupt reactive control and invalidate standard RL assumptions. ARLI addresses this challenge by augmenting asynchronous inference with intermediate information, enabling improvement under practical delays in simulation and real-world manipulation.
- Generalist policies enable online improvement and adaptation beyond offline training, but existing RL results remain concentrated in constrained laboratory settings.
- 100 milliseconds to over 300 milliseconds per action generation can cause pauses or jerky movements, altering effective environment dynamics and harming reactive performance.
- Asynchronous inference hides latency by computing the next action while the current action executes, but creates a mismatch between the policy’s conditioning state and the state when actions are played.
- ARLI restores near-Markovian information by conditioning RL on committed intermediate actions and a more up-to-date intermediate observation.
- ARLI improves generalist policies under latency in simulation and on three real-world bimanual UR5e tasks, where naive asynchronous RL and synchronous inference fail.
2 Related Work
Prior work addresses latency in robot deployment, delayed-information RL, and generalist-policy improvement, but does not jointly solve RL finetuning for modern action-chunked asynchronous inference.
- Generalist foundation models routinely exceed controller sampling periods, making inference latency a central deployment concern.
- Action chunking amortizes inference cost but leaves synchronous pauses and can produce discontinuities under naive asynchronous execution.
- Delayed-control RL research studies delayed information, actions, rewards, stochastic delays, and concurrent control, largely in generic settings or from-scratch training.
- Generalist-policy RL improvement includes diffusion-noise steering, but prior approaches do not explicitly address finetuning under latency constraints.
3 Preliminaries
The paper formulates RL for action-chunked diffusion or flow policies whose large-model inference takes multiple environment steps, and reviews asynchronous execution, RTC, and DSRL as the operating framework.
- RL seeks a policy maximizing expected discounted reward in a Markov decision process defined by states, actions, transitions, rewards, initialization, and discounting.
- The pretrained policy predicts k-step action chunks, executes n actions before recomputing, and uses sampled noise because it is a diffusion or flow policy.
- Inference requires tdelay environment steps, motivating asynchronous computation of a fresh chunk while the previous chunk is still executing.
- Real-Time Chunking incorporates unplayed actions from the previous chunk into the next inference call to improve temporal consistency and reduce jerky transitions.
- DSRL steers a pretrained diffusion or flow policy by training a lightweight RL policy to select its denoising input noise.
4 Efficient RL Finetuning Under Inference Delays
Naively applying DSRL during asynchronous inference fails because actions are selected from stale information. ARLI addresses this with committed-action and intermediate-state conditioning, while its delayed-observation analysis characterizes the resulting suboptimality.
- 4 Efficient RL Finetuning Under Inference Delays: Synchronous inference introduces problematic pauses, so the proposed finetuning framework focuses on efficient online improvement with asynchronous inference.
- 4 Efficient RL Finetuning Under Inference Delays: Naive DSRL conditions on the state available when inference begins, although the resulting action is played tdelay steps later, making effective finetuning unsuccessful.
- ARLI actions: Conditioning πrl on intermediate actions: ARLI conditions the RL policy on the initial state and committed intermediate actions, giving it predictive information about the future state at action execution.
- ARLI state: Conditioning πrl on intermediate state: Intermediate actions alone cannot capture disturbances occurring during inference, so ARLI additionally uses a later intermediate state for greater reactivity.
- Delayed Oracle Optimality Gap: The delayed-observation analysis bounds learned-policy loss by the oracle delayed optimality gap scaled by 1/(1−γ^k−tdelay), supporting Q-learning when delay has limited oracle impact.
5 Experimental Results
ARLI is evaluated against DSRL, residual RL, and policy variants in simulated and real-world tasks under asynchronous inference. The experiments assess learning speed, final success, throughput, state augmentations, and robustness to inference delays.
- Simulated Experiments: ARLI significantly outperforms naive asynchronous DSRL on four simulated tasks, reaching higher final success rates with less training time.DSRL with RTC improves some tasks but remains insufficient for fully learning the desired behavior; ARLI’s state augmentations are required.
- Simulated Experiments: ARLI enables effective finetuning under asynchronous delays in high-reactivity simulations and with large-scale VLAs.These results cover Kinetix tasks and AlohaTransferCube, which uses a 3.3B-parameter VLA with larger latency.
- Real-World Experiments: ARLI reaches near 100% success on all three real-world tasks after 100–125 training episodes, starting from approximately 40%.DSRL variants struggle to reach 80% on Assembly and Bag-Placement in the same time and take nearly twice as long to match ARLI on Shoe-in-Bag.
- Real-World Experiments: ARLI shows significantly higher successes-per-hour throughput across all three real-world tasks than the compared methods.Throughput is estimated from success durations, with failure declared after twice the average success duration.
- Ablations: Conditioning the RL policy on both intermediate states and actions performs significantly better than conditioning on either alone.The comparison is conducted without RTC on mjc swimmer and AlohaTransferCube.
- Ablations: ARLI’s performance decays more slowly than DSRL as inference delay increases, indicating greater robustness to latency.This pattern appears across mjc swimmer and AlohaTransferCube with RTC.
- Ablations: More up-to-date intermediate information improves performance, and learning significantly improves when RL inference delay exceeds half the policy inference delay.These results are reported for ARLI with RTC on mjc swimmer.
6 Conclusion and Limitations
The paper concludes that observation augmentation enables effective asynchronous RL improvement under practical latency constraints. It also identifies assumptions about the pretrained policy, RTC’s reactivity trade-off, and DSRL’s task dependence as limitations.
- Conclusion: ARLI enables effective RL improvement with asynchronous policy inference by augmenting the observations supplied to the RL policy.The conclusion frames this as learning while mitigating latency effects.
- Limitations: ARLI assumes the pretrained policy can be split into a VLM backbone and an action expert requiring the noise input.Policies that do not satisfy this structure cannot use intermediate states, though ARLI can still use intermediate actions.
- Limitations: RTC can improve performance but may reduce reactivity by forcing old actions to be executed, making the RL policy less expressive.ARLI therefore faces a trade-off between temporal continuity and policy expressiveness.
- Limitations: Because ARLI relies on DSRL, it will likely struggle on tasks where DSRL struggles to learn.The limitation follows from previously observed task dependence of DSRL.
Contributions
The contributions span method development, simulated and real-world experimentation, theoretical analysis, implementation, advising, and writing. Multiple contributors led distinct parts of the evaluation and system development.
- Experiments: Simulation contributions covered Aloha and Kinetix experiments, broader simulated experiments, and learning-dynamics analysis.These roles included writing and experimental support.
- Methodology: Method contributions included adding intermediate actions and intermediate state to ARLI, alongside DSRL-plus-RTC reproduction.These contributions also included software architecture, implementation, simulation, and writing.
- Project Support: The project also included software architecture and implementation, theoretical results, method ideation, advising, resources, funding, and writing.These roles were distributed across the author team.
- Experiments: Real-world contributions included experiment leadership, execution, early testing, and experimental support.Several contributors supported evaluation on the robotic tasks.
A.1 Implementation
The implementation combines DSRL-SAC and residual RL with action-chunked latent-MDP training, task-specific observation architectures, ARLI intermediate-information concatenation, and RTC-integrated codebases.
- RL Training: ARLI and DSRL use DSRL-SAC with repeated latent actions, treating action-chunk sampling and execution as one latent-MDP step.Observations within the action chunk are ignored for RL training.
- RL Training: Residual RL outputs an action correction that edits the base action and can run independently with minimal inference cost.Residual edits are bounded to [0, 0.1] to avoid relearning the task from scratch.
- Observation Architecture: Kinetix uses symbolic vector observations with MLP actor and critic networks, while AlohaTransferCube and real-world tasks use images plus joint states.The observation and architecture vary by environment.
- ARLI Inputs: Intermediate states are concatenated with original observations, and intermediate actions are flattened and concatenated to the state vector.For image-based tasks, images are concatenated along the channel dimension and joint states separately.
- Software: The implementation modifies DSRL and OpenPI codebases to integrate RTC and adds Kinetix tasks to DSRL.AlohaTransferCube requires no additional modifications.
- Initialization: Experiments initialize from task-specific pretrained checkpoints, including π0.5 with LoRA for real-world tasks and π0 checkpoints for AlohaTransferCube.Real-world training data is recorded at 60 Hz from human remote control.
B Additional Ablations
Ablations show that conditioning πrl on intermediate information is consistently important, while RTC reduces how much additional information the final state representation needs. ARLI is also sensitive to RL inference delay, especially without RTC.
- Intermediate Information: Including all intermediate information consistently produces the best results across mjc swimmer, mjc walker, car launch, and AlohaTransferCube.The ablation compares conditioning on intermediate state and actions, with and without RTC.
- Intermediate Information: RTC reduces the amount of additional information required in the final state representation.This is the stated role of RTC in the intermediate-information ablation.
- RL Inference Delay: Removing RTC increases the requirement for the intermediate state to be up to date.The sensitivity experiment evaluates varying RL inference delays on mjc swimmer with and without RTC.
- RL Inference Delay: Significant policy improvements appear only when t_rl delay < 3 without RTC and t_rl delay < 2 with RTC.These thresholds are reported for the mjc swimmer delay ablation.
C Learning Dynamics of ARLI Noise Steering
The noise-steering diagnostic indicates that ARLI reshapes its learned noise distribution in trajectory- and phase-dependent ways, with stronger localized changes on failed held-out episodes. The analysis is descriptive rather than a population-level classifier.
- Checkpoint Evolution: Failed held-out episodes show a directional shift toward more displaced and somewhat narrower predicted steering-noise distributions at later checkpoints.Successful episodes exhibit this pattern less strongly.
- Checkpoint Evolution: Noise-distribution changes are concentrated around phases requiring the largest corrective steering rather than spread uniformly across trajectories.The first successful episode has the weakest localized signal because it is smooth and well aligned.
- Latent-Dimension Changes: Effective drift spans 22.7–26.7 of 32 latent dimensions between the first and final checkpoints.The counts are computed from symmetric KL contributions averaged over inference steps.
- Interpretation: ARLI reshapes noise in a trajectory- and phase-dependent manner rather than merely rescaling the pretrained policy’s noise globally.Failures combine larger mean displacement, stronger variance reduction, and concentrated distributional changes.
- Interpretation: Intermediate actions and, in full ARLI, a more recent state support steering corrections targeted to the effective state for the next action chunk.This connects the learned distributional changes to ARLI’s intermediate-information design.
- Scope: The diagnostic covers four held-out trajectories from one task and should not be interpreted as a population-level success/failure classifier.The authors characterize it as descriptive evidence about checkpoint-wise noise evolution.
D Theoretical Results
The theoretical results characterize when delayed action-chunking data supports learning and how inference delay limits achievable performance. Under transition-dynamics and support assumptions, delayed policies yield consistent data and near-optimal delayed control, while the learned non-delayed value can remain unattainable.
- Assumptions: The framework assumes trajectory data obeys the MDP transition dynamics, even when the behavior policy is non-Markovian.Each subsequent state is generated from the current state and action through the MDP transition kernel.
- Consistency: Strong open-loop consistency equates observed future-state distributions with environment distributions under the executed action sequence.The definition compares P(st+k′ | st, at:t+k′) with PD(st+k′ | st, at:t+k).
- Delayed policy: A d-delayed action-chunking policy uses delayed observation st−d while preserving an already committed action prefix.The policy approximates the best completion of the action chunk conditioned on the delayed observation and fixed prefix.
- Consistency: Data collected by d-delayed action-chunking policies is open-loop consistent, because committed prefixes and future chunks are independent of intermediate observations.This establishes the future-state relation required for open-loop consistency.
- Q-learning: Action-chunking Q-learning is optimal under delayed-policy data when the data distribution includes behavior from an optimal action-chunking policy.The result identifies the learned value with the optimal value achievable by a non-delayed action-chunking policy.
- Near-optimality: Even with an optimal action-chunking Q-function, the corresponding optimal policy is not achievable when decisions are delayed by d time steps.The theoretical guarantee therefore concerns near-optimal delayed control rather than exact realization of the non-delayed optimum.
- Near-optimality: The delayed optimality gap ω_d bounds the sub-optimality caused by making decisions from delayed observations and executing a predetermined action prefix.The theorem additionally requires optimal action chunks and their delayed-prefix completions to be in support; the delayed policy then satisfies a Bellman fixed-point characterization.