RLinf-USER: A Unified and Extensible System for Real-World Online Policy Learning in Embodied AI

Hongzhi Zang; Shu'ang Yu; Hao Lin; Tianxing Zhou; Zefang Huang; Zhen Guo; Xin Xu; Jiakai Zhou; Yuze Sheng; Shizhe Zhang; Feng Gao; Wenhao Tang; Yufeng Yue; Quanlu Zhang; Xinlei Chen; Chao Yu; Yu Wang

Source-linked AI summary

RLinf-USER: A Unified and Extensible System for Real-World Online Policy Learning in Embodied AI

Hongzhi Zang, Shu'ang Yu, Hao Lin, Tianxing Zhou, Zefang Huang, Zhen Guo, Xin Xu, Jiakai Zhou, Yuze Sheng, Shizhe Zhang, Feng Gao, Wenhao Tang, Yufeng Yue, Quanlu Zhang, Xinlei Chen, Chao Yu, Yu Wang

arXiv:2602.07837v3Feb 8, 2026cs.RO

TL;DR

Real-world online policy learning must handle heterogeneous hardware, unstable communication, and long-running interrupted experiments, while existing pipelines lack persistent recovery and cross-stage reuse. USER unifies robots and accelerators, adaptive communication, persistent buffering, and asynchronous learning, with simulation and real-world experiments supporting heterogeneous multi-robot policy learning across diverse paradigms.

Problem
Real-world online policy learning lacks systems that jointly handle heterogeneous platforms, unstable networks, long-running interruptions, and persistent recovery with cross-stage data reuse.
Method
USER unifies robots with accelerators, adaptive cloud–edge communication, persistent cache-aware buffers, and asynchronous data generation, training, and synchronization.
Results
Experiments in simulation and the real world support efficient policy learning across heterogeneous multi-robot fleets, manipulation tasks, and reinforcement, imitation, and human-in-the-loop learning.
Takeaways & Limitations
USER provides a unified and extensible systems foundation for deploying diverse online policy-learning paradigms on heterogeneous robots.

Abstract

from arXiv · show

Online policy learning directly in the physical world is a promising yet challenging direction for embodied intelligence. Unlike simulation, real-world systems cannot be arbitrarily accelerated, cheaply reset, or massively replicated, which makes scalable data collection, heterogeneous deployment, and long-horizon effective training difficult. These challenges suggest that real-world policy learning is not only an algorithmic issue but fundamentally a systems problem. We present USER, a Unified and extensible SystEm for Real-world online policy learning. USER treats physical robots as first-class hardware resources alongside GPUs through a unified hardware abstraction layer, enabling automatic discovery, management, and scheduling of heterogeneous robots. To address cloud-edge communication, USER introduces an adaptive communication plane with tunneling-based networking, distributed data channels for traffic localization, and streaming-multiprocessor-aware weight synchronization to regulate GPU-side overhead. On top of this infrastructure, USER organizes learning as a fully asynchronous framework with a persistent, cache-aware buffer, enabling efficient long-horizon experiments with robust crash recovery and reuse of historical data. In addition, USER provides extensible abstractions for rewards, algorithms, and policies, supporting online imitation or reinforcement learning of CNN/MLP, generative policies, and large vision-language-action (VLA) models within a unified pipeline. Results in both simulation and the real world show that USER enables multi-robot coordination, heterogeneous manipulators, edge-cloud collaboration with large models, and long-running asynchronous training, offering a unified and extensible systems foundation for real-world online policy learning.

I. INTRODUCTION · II. RELATED WORK

USER frames real-world online policy learning as a systems problem shaped by unaccelerated physical execution, heterogeneous platforms, unstable networks, and long-running experiments. It addresses these constraints through unified hardware abstractions, adaptive communication, asynchronous learning, persistent data management, and extensible policy-learning components.

I. INTRODUCTION: Real-world online learning is constrained because robots operate in real time and cannot be arbitrarily accelerated, reset, or replicated as in simulation.These constraints motivate learning policies directly in the physical world despite transfer degradation from dynamics, sensing, and interaction gaps.
I. INTRODUCTION: Heterogeneous platforms, unstable networks, and frequently interrupted, long-running experiments make real-world online policy learning a systems problem.The problem couples physical execution, communication, and optimization rather than involving algorithms alone.
I. INTRODUCTION: USER treats physical robots as first-class hardware resources alongside GPUs, enabling automatic discovery, uniform management, and flexible joint scheduling.This unified abstraction targets distributed deployments with heterogeneous robots and accelerators.
I. INTRODUCTION: USER makes data generation, training, transmission, and weight synchronization proceed independently through a fully asynchronous learning framework.This avoids tightly synchronized stages in real-world policy learning.
I. INTRODUCTION: USER provides an adaptive communication plane using tunneling-based networking, distributed data channels, and SM-aware synchronization across heterogeneous and isolated network domains.The design addresses cloud–edge learning with rollout at the edge and training in the cloud, where synchronous simulation pipelines can reduce efficiency.
I. INTRODUCTION: A persistent, cache-aware buffer supports streaming long-horizon data ingestion, recovery, and reuse beyond memory-centric, short-lived pipelines.The motivation is high-dimensional visual data and extended experiments requiring persistence, recovery, and cross-stage data reuse.
I. INTRODUCTION: Extensible abstractions for rewards, algorithms, and policies support online imitation and reinforcement learning for CNN/MLP, generative, and large VLA policies.These components operate within a unified execution and data pipeline for real-world settings.

A. Robot Learning Systems … B. Adaptive Communication Plane

USER frames real-world online policy learning as a systems problem, unifying heterogeneous robots and accelerators while supporting asynchronous, long-horizon training. Its adaptive cloud–edge communication plane enables distributed training across isolated network domains and heterogeneous compute resources.

A. Robot Learning Systems: USER replaces simulation-centric synchronous execution with a fully asynchronous pipeline so robots can continue operating while learning runs.
A. Robot Learning Systems: USER unifies heterogeneous robots and accelerators as first-class, schedulable resources and adds persistent, cache-aware buffering for long-horizon policy learning.
B. Data Management for Long-Horizon Learning: USER targets long-horizon real-world learning with massive visual data by extending beyond volatile-memory replay-buffer designs toward persistent data management.
III. SYSTEM ARCHITECTURE DESIGN: The system architecture virtualizes physical robots and accelerators as first-class hardware and connects them through a robust cloud–edge communication substrate.
A. Unified Hardware Abstraction Layer: USER’s hardware abstraction layer uniformly manages robots and accelerators, providing extensible interfaces, automatic discovery, and scheduling for heterogeneous deployments.
A. Unified Hardware Abstraction Layer: The hardware layer organizes deployments into heterogeneous nodes for policy inference, edge action execution, and centralized training, with node groups capturing hardware differences.
B. Adaptive Communication Plane: USER’s adaptive communication plane combines UDP tunneling for bidirectional cloud–edge connectivity, distributed channels for traffic localization, and SM-aware synchronization to limit GPU contention.

IV. LEARNING FRAMEWORK DESIGN … C. Extensible Policies, Algorithms, and Rewards

USER organizes real-world learning as a fully asynchronous pipeline with persistent, cache-aware storage and extensible interfaces for policies, algorithms, and rewards. This design keeps robots executing continuously, supports long-horizon data reuse and recovery, and accommodates diverse learning setups without re-engineering core pipelines.

A. Fully Asynchronous Pipeline: Physical data collection is the primary bottleneck, so USER avoids synchronous cascading stalls by decoupling robot execution from training.Synchronous pipelines can propagate training delays to execution, forcing robots to pause and reducing data efficiency.
A. Fully Asynchronous Pipeline: Multiple environment and rollout workers continuously stream robot observations and actions while teleoperation and reward workers provide corrections, demonstrations, and supervision.Data generation proceeds without being blocked by optimization.
A. Fully Asynchronous Pipeline: Learning workers asynchronously sample buffer mini-batches for reinforcement or imitation learning, then periodically synchronize updated weights to rollout workers.This closes the learning loop while maintaining uninterrupted robot execution.
B. Persistent-Cache-Aware Buffer: Unlike short-lived in-memory buffers, USER retains historical data across evolving policies and supports arbitrarily large datasets, crash recovery, and long-horizon learning.Persistence decouples storage from memory and improves robustness across failures and pipeline restarts.
B. Persistent-Cache-Aware Buffer: USER uses a persistent, index-based buffer with disk-backed trajectories, metadata-aware sampling, and a bounded FIFO in-memory cache.Indices retain policy versions, timestamps, and episode IDs, while new samples enter memory first and evicted entries remain indexed on disk.
C. Extensible Policies, Algorithms, and Rewards: USER exposes unified interfaces so heterogeneous policies, optimizers, and reward mechanisms share one execution and data pipeline.The framework is agnostic to model architectures and learning algorithms.
C. Extensible Policies, Algorithms, and Rewards: The policy abstraction spans CNN/MLP controllers, flow-matching policies, and multimodal VLA models, all deployed through a unified rollout abstraction.Examples include ResNet-style visual policies and π0/π0.5 architectures producing continuous actions.
C. Extensible Policies, Algorithms, and Rewards: USER supports interchangeable off-policy, flow-policy, human-in-the-loop, and imitation algorithms alongside rule-based, human-provided, and learned rewards.Rewards may be attached during rollout or computed offline, enabling RL, imitation, and human-in-the-loop learning without re-engineering deployment or data pipelines.

V. EXPERIMENT · A. Main Results

USER is evaluated across simulated and real-world embodied tasks to test its extensible learning framework and systems design. In five real-world manipulation tasks, diverse policies, algorithms, reward sources, and training modes achieve strong performance, including near-perfect or approaching-1.0 success and reward-model supervision comparable to human labels.

V. EXPERIMENT: USER’s unified pipeline accommodates diverse policies, algorithms, and reward sources while supporting high performance across multiple tasks.The broader experiment evaluates extensibility, unified hardware abstraction, adaptive communication, and asynchronous execution.
A. Main Results: USER evaluates extensibility through five Franka-arm manipulation tasks: Peg Insertion, Charger, Cap Tightening, Pick-and-Place, and Table Clean-up.The tasks span contact-rich precision, object transport, torque or pose control, and tabletop decluttering.
A. Main Results: The experiments combine CNN, flow-based, and large VLA policies with SAC, RLPD, SAC-Flow, and HG-DAgger across local and multi-GPU hardware.Small policies run on an RTX 4090 workstation, while π0 is trained and evaluated on four NVIDIA A100 GPUs.
A. Main Results: Within 2000s, RLPD, SAC, and SAC-Flow achieve near-perfect success on Peg-Insertion and Charger with similar overall performance.SAC performs worse on Charger, likely because higher precision and dense rewards induce suboptimal behaviors.
A. Main Results: RLPD reaches success rates approaching 1.0 on Pick and Place and Cap Tightening, with Cap Tightening converging quickly and Pick and Place requiring longer training.Pick and Place involves richer object dynamics than Cap Tightening.
A. Main Results: During HG-DAgger training of π0, human operators intervene to ensure episode success, while intervention steps per episode measure imitation-policy performance.Lower intervention steps directly indicate higher policy performance.
A. Main Results: A ResNet18-based binary reward model trained on approximately 1,600 frames provides peg-insertion supervision comparable to human rewards.The dataset has a success-to-failure ratio of roughly 1:3 and is built from 20 successful trajectories with post-completion stationary frames.

B. Advantages of a Unified Hardware Layer · C. Capability of the Communication Plane

USER’s unified hardware layer supports platform-independent online learning across multiple and heterogeneous robots, while its adaptive communication plane localizes traffic and reduces distributed-training overhead. Together, these capabilities enable parallel data collection, cross-embodiment learning, and efficient cross-domain deployment.

B. Advantages of a Unified Hardware Layer: USER enables online policy learning on real robots without platform-specific modification, supporting multi-robot training and robust learning across heterogeneous embodiments.The unified hardware abstraction treats diverse robots within one system framework.
B. Advantages of a Unified Hardware Layer: Two Franka robot arms concurrently trained on different manipulation tasks both converged within approximately 2500 seconds, matching single-robot baselines.Parallel data collection improves sample efficiency through multi-task training in one framework.
B. Advantages of a Unified Hardware Layer: A unified CNN-based policy successfully controlled a 7-DoF Franka arm and a 6-DoF ARX arm in a multi-colored buttonpressing task.The setup demonstrates shared visual-semantic representations across distinct embodiments.
C. Capability of the Communication Plane: Distributed channels reduce episode generation time by up to 3× in cross-domain deployments by lowering observation-and-action communication overhead.Rollout and environment interaction remain local, while communication between City B nodes becomes more efficient.
B. Advantages of a Unified Hardware Layer: Approximately two hours were required for heterogeneous training to reach full convergence, reflecting differences in arm DoF, end-effectors, cameras, and target colors.These variations make heterogeneous training more challenging than single-robot baselines.
C. Capability of the Communication Plane: Distributed channels consistently reduce communication overhead even when all three nodes use high-speed same-domain networks.The communication plane therefore improves efficiency under both cross-domain and favorable local-network conditions.
C. Capability of the Communication Plane: Cross-domain City B nodes achieve communication efficiency comparable to the all-same-domain setup by exploiting local high-bandwidth links and avoiding unnecessary cross-domain transfers.This comparison indicates that USER localizes traffic while preserving communication efficiency across deployment domains.

D. Validation for Persistent and Cache-aware Buffer · E. Asynchronous Design · VI. CONCLUSION

USER combines persistent cache-aware buffering with asynchronous training to improve throughput and convergence while supporting unified, extensible real-world online policy learning. Validation shows benefits from larger cache ratios, pipeline overlap, and carefully chosen weight synchronization intervals.

D. Validation for Persistent and Cache-aware Buffer: Larger cache-to-buffer ratios improve throughput, while persistent storage expands capacity beyond a pure in-memory buffer.The buffer uses in-memory caching alongside persistent storage to balance efficiency and capacity.
E. Asynchronous Design: USER’s asynchronous pipeline achieves higher throughput through pipeline overlapping despite similar per-stage latencies.The comparison profiles generation and training periods between consecutive executions.
E. Asynchronous Design: Training convergence on peg insertion with an RLPD CNN policy accelerates from 8000+ seconds to ∼1500 seconds under asynchronous training.The comparison is illustrated in Fig. 13a.
E. Asynchronous Design: Small weight synchronization intervals, such as 1 and 8, cause frequent in-episode updates that induce policy non-stationarity and can slow convergence or cause divergence.The ablation studies synchronization intervals on peg insertion with RLPD and a CNN policy.
VI. CONCLUSION: USER integrates robots as first-class hardware resources alongside GPUs with adaptive communication, persistent cache-aware buffering, and a fully asynchronous training pipeline.The conclusion presents these components as a unified infrastructure for real-world online policy learning.
VI. CONCLUSION: The system remains agnostic to policy architectures and optimization methods while supporting imitation learning, reinforcement learning, and human-in-the-loop learning.This summarizes USER’s extensibility across learning paradigms.

APPENDIX · A. Algorithm Implementation Details · 1) Problem Setting and Notation.:

This appendix section defines USER’s partially observable reinforcement-learning problem and notation, then outlines implementation details for integrated algorithms, including SAC’s entropy-regularized updates and replay-buffer training. It also identifies the SAC and SAC-Flow hyperparameter tables used to document the implementations.

A. Algorithm Implementation Details: The appendix provides implementation details and hyperparameters for four primary algorithms integrated into USER.The section first clarifies reinforcement-learning notation before presenting those details.
1) Problem Setting and Notation.:: USER’s problem is modeled as a POMDP with latent states, observations, actions, transition dynamics, rewards, and discount factor γ.The policy πθ(at | ot) conditions only on observations ot.
1) Problem Setting and Notation.:: Off-policy algorithms train from replay-buffer transitions represented as (ot, at, rt, ot+1), or generically as (o, a, r, o′).Time indices are omitted when clear from context.
1) Problem Setting and Notation.:: The notation distinguishes the critic Qψ(o, a), policy πθ(a | o), and target critic network Q ¯ ψ.These symbols establish the main function approximators used in the algorithm descriptions.
1) Problem Setting and Notation.:: SAC optimizes a stochastic policy by maximizing expected return together with policy entropy under maximum-entropy reinforcement learning.The temperature parameter α controls the strength of entropy H regularization.
1) Problem Setting and Notation.:: SAC trains its soft Q-function by minimizing the Bellman residual, using next-step actions sampled from the current policy and an EMA-updated target critic.The actor is updated separately through its stated policy objective.
1) Problem Setting and Notation.:: SAC is selected for real-world reinforcement learning because of off-policy sample efficiency, with network and entropy-regularization settings summarized in Table V.The appendix also identifies Table VI as the hyperparameter table for SAC-Flow.

3) SAC-Flow: · 4) RLPD:

SAC-Flow extends SAC with continuous-time flow-based policies, likelihood construction, and path-density actor optimization. RLPD combines online exploration with offline demonstrations through balanced sampling and ensemble critics to improve sample efficiency and convergence.

3) SAC-Flow:: SAC-Flow parameterizes the policy with a continuous-time flow model whose velocity network evolves a latent action variable.The velocity network uses either gated Flow-G or transformer-decoded Flow-T architectures to stabilize gradients.
3) SAC-Flow:: The latent action trajectory is generated deterministically through K Euler-integration steps from an initial Gaussian latent action.Each step updates A_ti using the velocity network vθ(t_i, A_ti, o), with A_t0 ∼ N(0, I).
3) SAC-Flow:: To satisfy SAC’s likelihood requirement, SAC-Flow uses noise-augmented rollout while preserving the final action’s marginal distribution.Its actor minimizes the joint path density over the K sampling steps, where A contains the intermediate action path.
3) SAC-Flow:: SAC-Flow is integrated into the framework, with flow-model hyperparameters including denoising steps and velocity backbone detailed in Tab. VI.The supplied passage also identifies Tab. VII as the hyperparameter table for RLPD.
4) RLPD:: RLPD combines offline demonstration data with online exploration to improve sample efficiency.Offline demonstrations guide policy learning and accelerate convergence.
4) RLPD:: RLPD constructs each mini-batch by sampling transitions from online and demonstration replay buffers at a fixed ratio.The batch combines transitions from B_online and B_demo.
4) RLPD:: To handle high UTD ratios, RLPD uses an ensemble of M Layer-Normalized Q-functions to stabilize target values.The target uses the minimum critic value across j = 1, ..., M together with the entropy term.
4) RLPD:: RLPD’s hyperparameters are provided in Tab. VII.This implementation detail accompanies its offline-guided learning procedure.

5) HG-DAgger: · B. Task Implementation Details · 1) Peg Insertion:

This section presents HG-DAgger as a human-intervention-based method for safely fine-tuning π0, then specifies the evaluation tasks and Peg Insertion implementation. Peg Insertion uses multimodal proprioceptive and visual observations with distance-based rewards in a resettable real-world setup.

5) HG-DAgger:: HG-DAgger is an interactive imitation learning algorithm for safe and efficient online fine-tuning, with a human expert monitoring and intervening when necessary.The method uses human oversight during policy execution.
5) HG-DAgger:: At each timestep, HG-DAgger selects the executed action through a gating mechanism.The supplied passage introduces the gating mechanism but omits its equation.
5) HG-DAgger:: State-action pairs collected during intervention are stored in Dintervene and used to update the policy with a behavior cloning loss.This intervention dataset provides the training examples for the BC objective.
5) HG-DAgger:: HG-DAgger fine-tunes π0 after supervised fine-tuning on a pre-collected dataset initializes the model with a non-zero initial success rate.The SFT and HG-DAgger parameters are listed in Table VIII.
B. Task Implementation Details: The task implementation section details task settings and task-specific training configurations for five evaluation tasks shown in Fig. 14.The section establishes the scope of the subsequent task descriptions.
1) Peg Insertion:: Peg Insertion requires high-precision insertion of a peg into its corresponding slot, with static-pose verification and automated resetting supporting real-world RL.These properties make the task suitable for repeated real-world experimentation.
1) Peg Insertion:: Table IX reports Peg Insertion training settings, using a single wrist camera, end-effector pose, velocity, force, torque, gripper pose, and distance-based rewards.The reward is derived directly from the distance between the end-effector and target pose.

2) Charger Plugging: · 3) Cap Tightening: · 4) Pick-and-Place:

The three tasks span precision-critical charger insertion, multi-turn cap tightening, and wide-exploration pick-and-place. Each uses task-specific camera views, workspace constraints, and human-in-the-loop procedures to support online learning.

2) Charger Plugging:: Charger plugging inserts a charger into a socket under narrow geometric tolerance, sparse visual features, and visual occlusion during approach.The task is defined as contact-rich manipulation requiring sub-millimeter precision.
2) Charger Plugging:: A desktop third-view camera provides visual feedback for charger plugging, while a constrained bounding box focuses the sub-millimeter operational workspace.The task-specific configurations and training settings are provided in Table X.
3) Cap Tightening:: Cap tightening drives a pre-positioned cap toward a target configuration across multiple full rotations, requiring cyclic regrasping for continuous multi-turn manipulation.The task emphasizes rotational displacement relative to translation.
3) Cap Tightening:: Cap tightening combines wrist and stationary third-view cameras, uses a task-specific bounding box, and receives sparse binary completion rewards from a foot-pedal-operated human.Environment resets are managed manually, with detailed settings in Table XI.
4) Pick-and-Place:: Pick-and-place transfers objects between two trays, but its vast exploration space creates a substantial sample-efficiency challenge.The illustrated instance involves transporting a randomly initialized rubber duck to a target container.
4) Pick-and-Place:: Pick-and-place uses wrist and 45-degree downward third-view cameras to combine fine-grained contact information with broad spatial context.This camera combination supports both local manipulation details and scene-level awareness.
4) Pick-and-Place:: In pick-and-place, a human operator mediates both reward signals and environment resets within the human-in-the-loop framework.The detailed task settings are provided in Table XII.

5) Table Clean-up: … 2) ARX arm:

The section describes a precision-demanding, long-horizon table-cleanup task and the robotic platforms and controllers used for experiments. It covers Franka’s force-aware impedance control and ARX-R5’s low-cost position-control setup.

5) Table Clean-up:: Table Clean-up requires placing a marker in an orange container, nesting it in a white receptacle, and closing a semi-transparent lid.The sequence combines long-horizon multi-stage manipulation with precision-critical grasping and lid closure.
C. Hardware Details: Experiments use two platforms: a 7-DoF Franka robot arm and a 6-DoF low-cost ARX robot arm.The section introduces these platforms as the two distinct robotic configurations used across experiments.
1) Franka Robot Arm:: The Franka Emika Panda is a 7-DOF collaborative research arm with high execution precision and integrated sensing.Most experiments use the Panda platform.
1) Franka Robot Arm:: Franka control uses FCI through libfranka, seven-joint torque sensing, and external RealSense D435i cameras for perception.These components provide force-and-torque inputs and visual observations for the policy.
1) Franka Robot Arm:: Franka reinforcement learning uses an impedance controller that regulates interaction forces and motion for compliant contact-rich behavior.The controller models the robot as a tunable mass-spring-damper system, following SERL [17].
2) ARX arm:: The cost-effective ARX-R5 is a 6-DOF arm without force/torque sensors, using PD position control and a wrist fisheye camera.Its RL policy sends position targets at 10 Hz, and the position controller tracks them at 200 Hz.