PaperlayerView exact revision on arXiv

Source-linked AI summary

AtomicRL: Atomic Reinforcement Learning for Long-Horizon Robotic Manipulation

Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang, Xiaodan Liang, Liang Lin

arXiv:2610.30097v1cs.ROcs.AIcs.CV

TL;DR

Long-horizon VLA reinforcement learning faces sparse, delayed feedback and uneven subtask mastery. AtomicRL combines atomic-task-aware credit assignment and adaptive exploration with a Real2Sim2Real pipeline. It reports gains over PPO in simulation and improved real-world task performance, including a 21.3% average real-world improvement.

  • Problem

    Existing VLA-RL methods face long-horizon credit-assignment challenges from terminal-only rewards and bottlenecks from uneven subtask mastery.

    Source: §1 · p. 1

  • Method

    AtomicRL combines SAPO's semantic-boundary gating of TD-residual propagation, BATE's mastery-based task sampling and exploration, and a Real2Sim2Real deployment pipeline.

    Source: §1 · p. 2

  • Results

    AtomicRL improves performance over PPO across LIBERO-LONG and CALVIN ABC-D and in real-world tasks, with a 21.3% average real-world improvement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments in simulation and on real-world robots demonstrate AtomicRL's effectiveness for long-horizon VLA reinforcement learning.

    Source: §1 · pp. 2–3

    Online RL still requires substantial environment interaction and training time, particularly in real-robot settings.

    Source: §A.1 · p. 13

Abstract

from arXiv · show

Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance. To address these challenges, we propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration. AtomicRL introduces Semantic-Aware Policy Optimization (SAPO), which uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit contamination while preserving intra-subtask credit assignment and value bootstrapping. In addition, Bottleneck-Adaptive Target Exploration (BATE) dynamically coordinates task sampling and local exploration incentives based on the mastery of individual atomic subtasks, directing learning toward the remaining bottlenecks. Finally, we develop a Real2Sim2Real pipeline that bridges online RL in simulation with real-world deployment, facilitating practical transfer to physical robots. On LIBERO-LONG, AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO across two VLA architectures. On CALVIN ABC-D, it improves full-task performance by up to 9.5% over PPO. Combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by an average of 21.3%. These results demonstrate that AtomicRL effectively improves the learning efficiency and manipulation performance of VLA policies on long-horizon tasks.

1 INTRODUCTION

AtomicRL addresses long-horizon VLA-RL challenges with atomic-task-based credit assignment and adaptive exploration. It reports gains over PPO in simulation and improved real-world task performance.

1 INTRODUCTION

  • Terminal-only rewards can make GAE propagate failure residuals backward and devalue correctly executed actions, while uneven subtask mastery leaves bottlenecks underserved by uniform sampling.The described credit contamination worsens as the horizon grows.Source: §1 · pp. 1–2
  • AtomicRL combines SAPO, which gates TD-residual propagation at semantic boundaries, with BATE, which directs task sampling and exploration toward less-mastered subtasks.A Progress–Completion Critic estimates subtask progress and completion boundaries, and supplies progress estimates for dense shaping rewards.Source: §1 · p. 2
  • The Real2Sim2Real pipeline reconstructs an interactive simulation from a real-world multimodal demonstration and connects simulation-based RL to robot deployment.The pipeline shifts costly physical trial-and-error to parallel simulation with reproducible resets.Source: §1 · p. 2
  • 3.6% and 4.0% gains over PPO on LIBERO-LONG, with π0.5 and OpenVLA-OFT, accompany a 9.5% full-task gain on CALVIN ABC-D and a 21.3% average real-world improvement.The LIBERO-LONG success rates are 99.0% with π0.5 and 98.8% with OpenVLA-OFT.Source: §1 · p. 2

2 RELATED WORK

Prior work explores RL for VLA policies and ways to address sparse feedback and costly real-robot interaction. AtomicRL distinguishes itself by gating cross-subtask advantage propagation and automating a single-demonstration Real2Sim2Real pipeline.

2.1 REINFORCEMENT LEARNING FOR VLA MODELS

  • AtomicRL gates advantage-signal propagation at atomic-task boundaries and allocates exploration to bottleneck tasks, addressing interference that dense reward shaping does not eliminate.The passage contrasts this with stage-level preference optimization and trajectory- or subtask-level reward guidance.Source: §2.1 · p. 3

2.2 ONLINE REINFORCEMENT LEARNING IN REAL WORLD

  • AtomicRL automates Real2Sim2Real reconstruction of an interactive simulation from a single demonstration, then transfers trained policies to physical robots.The passage describes environment construction in prior approaches as complex and costly to reproduce.Source: §2.2 · p. 3

3 METHOD

AtomicRL organizes long-horizon RL around atomic subtasks, combining outcome-sensitive credit propagation and mastery-adaptive exploration with PCC support and a simulation-to-robot pipeline.

3.1 OVERVIEW

  • AtomicRL integrates semantic-aware credit assignment and bottleneck-adaptive exploration, supported by PCC and a Real2Sim2Real workflow for simulation training and transfer evaluation.PCC estimates subtask progress and identifies completion boundaries; SAPO modulates credit propagation, while BATE adjusts task sampling and local exploration incentives.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.1.1 PROBLEM FORMULATION

  • The objective is to maximize full-task success under a fixed interaction budget by improving credit assignment and directing exploration toward bottleneck subtasks.Instructions are decomposed into ordered atomic subtasks, with semantic boundaries indicating transitions between them.Source: §3.1.1 · p. 4

3.2 SEMANTIC-AWARE POLICY OPTIMIZATION

  • SAPO uses PCC-identified semantic boundaries and subtask outcomes to attenuate TD-residual propagation across failed-subtask boundaries while retaining standard within-subtask traces.When a subtask fails, propagation into the completed prefix is attenuated; successful-subtask connections and value targets based on the resulting advantages are retained.Source: §3.2 · pp. 4–5

3.3 BOTTLENECK-ADAPTIVE TARGETED EXPLORATION

  • BATE adapts full-task sampling and local entropy-based exploration to subtask mastery, increasing exploration for poorly mastered subtasks and reducing it as mastery improves.Mastery is estimated from an exponential moving average of resolved subtask completion outcomes; sampling retains baseline weight for mastered-task coverage.Source: §3.3 · p. 5

3.4 PROGRESS–COMPLETION CRITIC

  • PCC estimates within-task progress and completion from recent observation–action pairs, identifying atomic-task boundaries and providing signals for SAPO and BATE.Completion requires progress and completion estimates to meet thresholds for N consecutive decision steps; PCC is trained offline on demonstrations, and progress estimates can optionally support shaping rewards.Source: §3.4 · pp. 5–6

3.5 REAL2SIM2REAL PIPELINE FOR REINFORCEMENT LEARNING

  • The Real2Sim2Real pipeline builds an aligned simulation environment, shared observation–action interface, and randomized layouts from a real-world demonstration, then trains and evaluates the policy on physical robots.The optimized policy is deployed through the same interface used for simulation training, following pre-deployment validation.Source: §3.5 · p. 6

4 EXPERIMENT

AtomicRL improves long-horizon manipulation results across simulated and real-robot evaluations, while ablations examine its reward design, components, and Progress–Completion Critic. The experiments also assess benchmark setups and simulation-to-reality correspondence.

4.1 EXPERIMENTS SETUP

  • The simulation evaluation uses LIBERO-LONG’s 10 long-horizon tasks and CALVIN’s ABC-D protocol, with 50 LIBERO trials per task and 1,000 CALVIN trials.Real-robot tasks use 25 trials per method, with mean success averaged across three tasks; the robot study collects 100 expert demonstrations per task for SFT.Source: §4.1 · pp. 6–7

4.2 RESULTS ON SIMULATION

  • 98.8% with OpenVLA-OFT and 99.0% with π0.5 on LIBERO-LONG exceed their PPO baselines by 4.0% and 3.6%, respectively.Both policies reach near-perfect success rates after 200 optimization iterations.Source: §4.2 · p. 7, Table 1 · p. 7
  • 80.1% full five-task success on CALVIN ABC-D is 9.5% above PPO, while average completed sequence length is 4.56 versus PPO’s 4.36.The average completed sequence length rises from 3.56 with SFT to 4.56 with AtomicRL.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 RESULTS ON REAL-WORLD ROBOT

  • 21.3% average improvement over SFT and 8.0% over PPO are reported across three real-world long-horizon tasks.The evaluation uses online RL in constructed real-to-sim environments and tests policy improvement and sim-to-real transfer.Source: §4.3 · p. 7, Table 3 · p. 7
  • Real-robot training curves show faster early improvement, especially on the longer-horizon Bread Placement task; simulation and robot success rates correlate at r = 0.909.The correlation aggregates nine combinations of three tasks and three methods.Source: §4.3 · p. 8, Figure 4 · p. 8, Figure 3 · p. 8

4.4 ABLATION STUDY

  • Under SAPO, subtask-level rewards outperform process-level dense rewards; the study anneals PBSR to reduce dense-reward weight over training.Dense rewards offer stronger exploration guidance at low success, but persistent auxiliary rewards may dilute the terminal success signal as the policy improves.Source: §4.4 · pp. 8–9, Table 4 · p. 9

SAPO PBSR BATE SR (%)

  • On LIBERO-LONG, success rises from 78.2% with PPO to 87.8% with SAPO, 89.6% with annealed PBSR, and 91.2% with BATE.The cumulative gain is 13.0 percentage points over PPO; SAPO contributes the largest individual increase.Source: SAPO PBSR BATE SR (%) · p. 9, Table 4 · p. 9
  • PCC achieves 95.92% precision, 89.20% recall, and 42.4 ms empirical inference latency.It estimates subtask progress and completion status from 150 training trajectories and is evaluated on 500 rollout trajectories.Source: SAPO PBSR BATE SR (%) · p. 9

5 CONCLUSION

AtomicRL organizes long-horizon VLA reinforcement-learning post-training around atomic subtasks, using progress estimates to guide credit assignment and bottleneck-focused exploration. Its Real2Sim2Real pipeline and evaluations support its potential for long-horizon robotic manipulation.

5 CONCLUSION

  • AtomicRL uses PCC to estimate progress and completion boundaries, SAPO to regulate cross-boundary TD-residual propagation, and BATE to target insufficiently mastered bottlenecks.The framework is centered on atomic task structure for long-horizon robotic manipulation.Source: §5 · p. 9
  • The Real2Sim2Real pipeline connects real-task reconstruction, simulation-based RL, and real-robot deployment, using parallel interactions and reproducible resets.Source: §5 · p. 9
  • Simulation and real-world experiments show faster convergence to a higher performance level, supporting AtomicRL’s potential for long-horizon robotic manipulation.Source: §5 · p. 9

A.1 FUTURE WORK AND LIMITATIONS

AtomicRL improves interaction-data use through credit assignment and exploration allocation, but leaves online RL efficiency unresolved. Its evaluation is focused on tabletop manipulation from SFT policies, with broader generalization still to be assessed.

A.1 FUTURE WORK AND LIMITATIONS

  • AtomicRL improves temporal credit assignment through SAPO and concentrates exploration on insufficiently mastered bottleneck subtasks through BATE.Source: §A.1 · p. 13
  • AtomicRL improves interaction-data utilization but does not fundamentally address online RL training efficiency, which remains costly, especially for real-robot training.Reducing interaction requirements, improving training efficiency, and lowering real-robot training costs remain future directions.Source: §A.1 · p. 13
  • Experiments focus primarily on long-horizon tabletop tasks initialized from SFT policies; effectiveness on open, dynamic tasks requiring stronger embodiment generalization remains unevaluated.The authors suggest transferring atomic-task structure to Agentic RL frameworks as a possible direction for open environments.Source: §A.1 · p. 13

A.2 PROGRESS-BASED SHAPING REWARDS

Progress-based shaping rewards (PBSR) use PCC progress estimates and atomic-task state to supplement sparse task rewards. The shaping term is gated by valid updates and terminal status, and its coefficient is annealed during training.

A.2 PROGRESS-BASED SHAPING REWARDS

  • PBSR optionally supplements sparse task rewards using the atomic-task index and discrete PCC progress estimates.Source: §A.2 · p. 13
  • The progress potential combines the active atomic-task index with within-task progress scaled by 1/B.After completion handling, progress is reset to zero when completion is confirmed, and the task index advances only when another task remains.Source: §A.2 · p. 13
  • The shaping contribution is zero on terminal transitions, including timeouts, and whenever no valid PCC update is available.The stored potential is initialized to zero on tracker reset and replaced after each valid update.Source: §A.2 · p. 13
  • The shaping coefficient is linearly annealed, and setting its final retained fraction ρ to zero removes PBSR after annealing.The shaping reward can be positive or negative and is not clipped.Source: §A.2 · pp. 13–14
  • PBSR changes the reward used to compute TD residuals, whereas SAPO regulates their propagation across subtask boundaries during advantage estimation.Source: §A.2 · p. 14

A.3 ADDITIONAL ABLATIONS

The SAPO gating-coefficient ablation finds its highest observed LIBERO-LONG success rate at κ = 0.3, while both stronger attenuation and higher propagation settings are compared.

A.3 ADDITIONAL ABLATIONS

  • In this setting, the highest observed success rate is 87.8% at κ = 0.3, 9.6 percentage points above the unattenuated baseline of 78.2%.κ controls trace propagation across boundaries preceding failed subtasks; gates preceding successful subtasks remain fixed at one.Source: §A.3 · p. 14
  • κ controls trace propagation across boundaries preceding failed subtasks; κ = 1 recovers standard GAE, while κ = 0 blocks cross-boundary propagation without removing one-step value bootstrap.Source: §A.3 · p. 14
  • Hard blocking at κ = 0 achieves 87.2%, while success falls to 85.0% at κ = 0.5 and 81.2% at κ = 0.8.The authors report these results as supporting soft attenuation in this setting.Source: §A.3 · p. 14

A.4 TRAINING DETAILS

PCC is trained offline on annotated demonstrations and kept fixed during RL post-training. Experiments use RLinf, with training on eight H200 GPUs and inference on one RTX 5090 GPU.

A.4 TRAINING DETAILS

  • PCC is trained offline on annotated demonstrations and remains fixed during RL post-training.The setup uses 150 annotated LIBERO trajectories and 50 trajectories per real-world task.Source: §A.4 · p. 14
  • PCC uses a frozen SigLIP visual encoder of approximately 400M parameters, initialized from the π0.5 visual backbone, with lightweight trainable temporal and prediction modules.Source: §A.4 · p. 14
  • RL post-training runs in RLinf on eight NVIDIA H200 GPUs, while inference uses one NVIDIA GeForce RTX 5090 GPU.Source: §A.4 · p. 14

A.5 ADDITIONAL RESULTS

AtomicRL improves training dynamics and value-function fitting on LIBERO-LONG and CALVIN ABC-D, while its Real2Sim2Real workflow supports online RL and deployment. Additional visualizations show AtomicRL completing a sequence that SFT fails to finish.

A.5 ADDITIONAL RESULTS

  • AtomicRL improves more rapidly early and outperforms PPO later on LIBERO-LONG and CALVIN ABC-D, with critic explained variance higher by about 12% and 5%, respectively.The value-function comparison is shown in Fig. 7; training dynamics are shown in Fig. 6.Source: §A.5 · p. 14, Figure 7 · p. 15
  • On Bread Placement, AtomicRL reaches 60% success around step 30, while PPO stays near 50% for the first 40 steps; AtomicRL also has higher late-stage success on Waste Disposal and Tabletop Clean.Bread Placement comprises six atomic subtasks; training curves are shown in Fig. 8.Source: §A.5 · p. 15, Figure 8 · p. 15

A.6 DETAILS OF AUTOMATED REAL2SIM2REAL PIPELINE

  • The automated Real2Sim2Real workflow connects real-task reconstruction, simulation-based online RL in RLinf, and real-robot deployment.Fig. 9 summarizes key steps and validation requirements.Source: §A.6 · p. 15, Figure 9 · p. 16

A.7 ADDITIONAL VISUALIZATIONS

  • In the illustrated sequence, SFT completes initial actions but fails later, whereas AtomicRL completes the sequence in simulation and real-world deployment.The executions are shown in Fig. 10.Source: §A.7 · p. 15, Figure 10 · p. 16
Loading 2610.30097v1…