Source-linked AI summary
AgenticRoboRL: Bringing Coding-Agent Post-Training to Agentic Robotics
Xinxin Liu, Bingxin Xu, Wenrui Bao, Ming Li, Wentao Bao, Chen Chen, Yuzhang Shang
TL;DR
Prior work often strengthens robot harness support rather than planners’ own capabilities, and differing harness execution mechanisms complicate post-training across systems. AgenticRoboRL combines SFT and multi-turn RL through harness-specific adapters while keeping execution components fixed. On LIBERO-PRO, it raises a 9B planner’s success rate from 18.1% to 71.0%.
Problem
Can coding-agent post-training improve robot planners, given prior work largely strengthens support around existing planners and harnesses differ in execution, feedback, and success criteria?
Source: §1 · p. 1Method
AgenticRoboRL combines SFT on successful demonstrations with multi-turn RL, using harness-specific adapters to connect recorded planner interactions to outcomes while execution components remain fixed.
Source: §1 · pp. 1–2Results
On LIBERO-PRO, the 9B planner improves from 18.1% to 71.0%; the same pipeline also improves success on CaP-X from 2.7% to 28.5%.
Source: §1 · p. 2Takeaways & Limitations
The results support post-training the planner as an effective way to improve robot-harness agents without retraining their underlying controllers.
Source: §5 · p. 8Comparisons with planners using memory provide system-level references and do not isolate the contribution of supervision from a teacher with memory.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Agentic robotics systems combine planners with robot harnesses to perform complex manipulation tasks. Recent work improves these systems through better harnesses and external memory, but relies on powerful closed-source models as planners and does not improve the planners' own capabilities. In contrast, coding agents have improved substantially by post-training the model itself. We therefore investigate whether post-training can also improve robot planners. However, applying it across robot harnesses requires accommodating different execution mechanisms, including VLA skill invocation and control-code generation. To this end, we introduce AgenticRoboRL, a framework for post-training robot planners across different harnesses. The training recipe combines supervised fine-tuning with multi-turn reinforcement learning. To accommodate different execution mechanisms, harness-specific adapters connect recorded planner interactions with task outcomes, enabling a common RL training pipeline while keeping execution components fixed. On LIBERO-PRO, AgenticRoboRL raises the success rate of a 9B-parameter planner from 18.1% to 71.0% without external memory, approaching the 72.1% achieved by Codex (GPT-5.5) with external memory. These results demonstrate post-training as an effective approach for improving robot planners. We will release the framework and trained planner checkpoints.
1 Introduction
Prior work strengthens robot harnesses and memory but largely leaves planners’ own capabilities unchanged, while applying post-training across harnesses is complicated by their different execution mechanisms. AgenticRoboRL combines SFT and multi-turn RL through harness-specific adapters, improving a 9B planner across two execution mechanisms.
1 Introduction
- Earlier work strengthens harness support or external memory, whereas this paper explores improving planners’ own capabilities through post-training, motivated by coding-agent advances.The authors frame whether coding-agent post-training can improve agentic robotics as their research question.Source: §1 · p. 1
- Harnesses vary in how they execute planner responses, what feedback they return, and how they determine task success, complicating a shared training pipeline.Some invoke learned VLA skills; others execute control code through perception and control APIs.Source: §1 · p. 1
- AgenticRoboRL addresses the gap with SFT on successful demonstrations followed by multi-turn RL, using adapters to connect planner interactions to episode outcomes across harnesses.The pipeline accommodates VLA skill invocation and control-code generation while keeping execution components fixed.Source: §1 · pp. 1–2
- On LIBERO-PRO, the 9B planner rises from 18.1% to 71.0%; on LIBERO-Plus it reaches 72.2% without additional training, and on CaP-X it improves from 2.7% to 28.5%.On LIBERO-PRO, the result approaches the 72.1% achieved by Codex (GPT-5.5) with external memory.Source: §1 · p. 2, Figure 1 · p. 2
2 Related Work
Related work spans robot harness systems, training planners for embodied agents, and post-training tool-using agents. AgenticRoboRL combines multi-turn planner training with an interface that accommodates different robot harness execution mechanisms.
Robot harness agents
- Robot harness agents pair language-level planners with executable skills or programs; AgenticRoboRL instead post-trains the planner while reusing harness execution components.Harness VLA combines a frozen VLA with analytic primitives and task memory [Zhang et al., 2026c], while CaP-X exposes control-code interfaces [Fu et al., 2026].Source: §2 · pp. 2–3
Training planners for embodied agents
- Embodied-agent training includes single-turn control-code RL [Fu et al., 2026], teacher-trajectory supervision [Li et al., 2026; Zhang et al., 2026b], and SFT plus online RL for mobile manipulation [Mi et al., 2026].AgenticRoboRL uses teacher-agent trajectories followed by multi-turn RL with task success as the sole reward, adapting supervision when prompts, tools, or memory access differ.Source: §2 · p. 3
Post-training of tool-using agents
- Coding-agent training combines successful teacher trajectories [Pan et al., 2024; Yang et al., 2026; Jain et al., 2025] with multi-turn RL on verifiable outcomes [Cao et al., 2025].AgenticRoboRL adapts recorded-interaction training interfaces, using a proxy and harness-specific adapters to connect planner calls to outcomes without requiring a shared low-level action representation.Source: §2 · p. 3
3 AgenticRoboRL Framework
AgenticRoboRL post-trains a robot planner while reusing the harness’s execution components. Its framework connects planner interactions to task outcomes across harnesses, combining supervised fine-tuning with multi-turn reinforcement learning.
3.1 Overview and Problem Setup
- The planner produces harness-native responses and learns from multi-turn interaction while perception, motion-planning, and execution components remain fixed.The objective uses terminal-success reward for task and initial-state draws, without intermediate rewards or a length penalty in the main RL configuration.Source: §3.1 · pp. 3–4
3.2 Training Infrastructure
- A recording proxy captures each actual planner request, and harness-specific adapters match calls to episode outcomes without imposing a shared execution interface.Incomplete or policy-version-mismatched records are checked before sample construction; infrastructure failures are excluded from optimization.Source: §3.2 · p. 4
3.3.1 Supervised Fine-Tuning
- SFT pairs teacher responses with the context the student would receive, supervising accepted response tokens while masking context tokens.For RPent, memory-dependent teacher responses are processed or filtered because the teacher has memory the student lacks; CaP-X uses successful episodes under a shared harness configuration.Source: §3.3.1 · pp. 4–5
3.3.2 Reinforcement Learning through Interaction
- RL assigns a leave-one-out advantage from terminal rewards to selected responses in valid groups sharing a task and initial-state configuration.Groups with identical rewards contribute no policy gradient; at most 16 eligible responses are selected per episode, and the update uses dual-clipped PPO with a correction for rollout/training-backend probability mismatch.Source: §3.3.2 · p. 5
4 Experiments
Across RPent and CaP-X, post-training improves the 9B planner on LIBERO-PRO; evaluation also tests robustness to LIBERO-Plus perturbations and compares two SFT recipes.
4.1 Experimental Setup
- The Qwen3.5-9B student is evaluated without external memory, while execution components including the VLA remain frozen; teacher demonstrations and RL rollouts use states 11–44, which do not overlap with evaluation initial states.LIBERO-Plus evaluation covered 10,030 configurations after training settings were fixed, measuring robustness without benchmark-specific training.Source: §4.1 · pp. 5–6
4.2 Results on LIBERO-PRO
- 71.0% success after RL (from 18.1%; SFT: 65.1%) on LIBERO-PRO, versus 57.9% for the original 27B planner without memory and 74.0% for the 27B teacher with memory.SFT accounts for most of the improvement, with RL adding 5.9 percentage points; on paired episodes, RL solves 121 that SFT fails and fails 74 that SFT solves (p = 0.001).Source: §4.2 · p. 6, Table 1 · p. 7
4.3 Robustness on LIBERO-Plus
- 72.2% success after RL on LIBERO-Plus, versus 19.2% for the original planner; post-training gains persist across perturbation categories.The RL planner has the highest overall success among methods not trained on LIBERO-Plus, with its largest margin on robot initial-state perturbations: 75.2% versus 46.2%.Source: §4.3 · pp. 6–7, Table 2 · p. 7
4.4 Generality across Harnesses
- 28.5% success after RL with CaP-X, up from 2.7% and above the 27B teacher’s 20.3%, showing gains with control-code generation.Improvements concentrate in the Object suites; Goal and Spatial performance remains low where the teacher rarely succeeds and provides little SFT data.Source: §4.4 · pp. 7–8, Table 3 · p. 8
4.5 Ablation Study
- 64.6% success after one epoch with rewrite-and-filter, versus 55.2% with mask-only; at similar update counts, the former reaches 60.2% versus 55.2%.The recipes differ in response content and supervision coverage, so the comparison does not isolate rewriting alone.Source: §4.5 · p. 8, Figure 3 · p. 8
5 Conclusion
AgenticRoboRL post-trains planners to improve robot-harness agents while reusing the harnesses’ execution components. Experiments with a 9B planner show gains across VLA skill invocation and control-code generation.
5 Conclusion
- Success on LIBERO-PRO rises from 18.1% to 71.0% with the 9B planner.The same trained planner achieves 72.2% on LIBERO-Plus without additional training.Source: §5 · p. 8
- AgenticRoboRL combines SFT on successful demonstration trajectories with multi-turn RL, linking recorded model calls to episode outcomes while reusing harness execution components.The framework is intended to improve how robot planners use existing harnesses.Source: §5 · p. 8
- The authors will release the framework, teacher trajectories, and trained planner checkpoints to support further research.Source: §5 · p. 8
A Policy Optimization Details
The policy optimization details define how recorded response probabilities inform the RL update and how responses contribute to its loss. Harness adapters report episode fields and connect recorded planner calls to episodes so sample construction and optimization can be shared.
A Policy Optimization Details
- For each recorded response, current, round-start, and rollout-server token probabilities are conditioned on the same recorded context and response prefix.The current trainer probability is πθ(ys,j | cs, ys,<j); the other probabilities are qsj and bsj.Source: A Policy Optimization Details · p. 12
- The PPO ratio compares the current planner with the trainer’s old-policy evaluation, while truncated importance sampling mitigates rollout-to-training backend probability mismatch.Source: A Policy Optimization Details · p. 12
- The loss applies only to response tokens, fixes old-policy probabilities and importance weights during each round, and omits KL and entropy terms.The main configuration uses ϵ− = 0.2, ϵ+ = 0.28, d = 3, and C = 2; response losses are token-averaged and weighted by inverse selected-turn count.Source: A Policy Optimization Details · p. 12
- Because batches need not contain complete episodes, the normalized RL loss is not an exact episode-average objective.Source: A Policy Optimization Details · p. 12
B Harness Adapters
- An adapter reports five episode fields, routes planner calls through a recording proxy, and matches each call to its episode; sample construction and optimization are shared.Integrating another harness requires producing those fields and call-to-episode matches.Source: B Harness Adapters · p. 12
C.1 Supervised Fine-Tuning with RPent
The supervised fine-tuning setup trains a 9B planner on target turns and selects a checkpoint using held-out validation states. The language model and visual merger are updated, while the vision encoder remains frozen.
C.1 Supervised Fine-Tuning with RPent
- SFT uses AdamW with a learning rate of 10^-5, batch size 64 target turns, 10 warmup steps, cosine decay, and gradient clipping at 1.0.The language model and visual merger are updated, while the vision encoder stays frozen; weight decay is zero.Source: §C.1 · p. 12
- The training set contains 40,047 target turns, making one epoch 626 optimizer steps.Source: §C.1 · p. 12
- The main results use the checkpoint at step 563, selected from ten checkpoints on validation states 45–49; it also initializes RL.Source: §C.1 · p. 12
C.2 Reinforcement Learning with RPent
The RPent RL setup collects rollouts over repeated rounds, validates outcomes and recorded calls, then updates the planner. The reported checkpoint follows round 6 and is selected using validation states disjoint from training and evaluation.
C.2 Reinforcement Learning with RPent
- The reported checkpoint is after round 6 of nine RL rounds, with planner weights fixed during rollout collection in each round.Source: §C.2 · p. 12
- Each round covers 64 task-and-initial-state groups, with six attempts per group for 384 episodes; rollouts use temperature 1.0 without token-distribution truncation.Source: §C.2 · p. 12
- Across the first six rounds, 2,304 episodes yield 26,052 selected responses and 206 optimizer steps; the checkpoint is selected on validation states 45–49.At most 16 responses are selected per episode, and groups with identical rewards across all six attempts contribute no training sample.Source: §C.2 · p. 12
C.3 Training with CaP-X
CaP-X training uses a 27B planner as teacher and applies SFT followed by RL to a 9B planner, with training groups drawn from tasks where the SFT planner has mixed validation success.
C.3 Training with CaP-X
- 979 successful teacher episodes with 2,370 target turns were collected on 26 tasks the 27B teacher solved at least once in five validation episodes.The teacher used the same CaP-X harness configuration as the student.Source: §C.3 · p. 12
- Table 5 identifies the fields reported by harness adapters and their sources in RPent and CaP-X.Source: Table 5 · p. 13
- RL starts from the SFT checkpoint and samples 64 groups with six attempts per round from 35 tasks where the SFT planner solves one to four validation episodes.The 35 tasks include all 26 tasks with teacher data and nine additional tasks; the reported checkpoint follows 15 rounds.Source: §C.3 · p. 13
- The remaining 25 evaluation tasks are excluded from training; the 35 RL tasks comprise 20 Object, nine Goal, and six Spatial tasks.Rollouts use initial states 11–44.Source: §C.3 · p. 13
C.4 Episode Limits and Sampling Settings
The section specifies episode and response limits, states that the same limits apply across collection, training, and evaluation, and gives RPent evaluation sampling settings.
C.4 Episode Limits and Sampling Settings
- RPent episodes end after at most 100 planner turns or 7,200 seconds, while CaP-X responses are limited to 32,768 tokens.Source: §C.4 · p. 13
- The same limits apply to teacher collection, RL rollouts, and evaluation.Source: §C.4 · p. 13
- RPent evaluation uses temperature 1.0, top-p 0.95, top-k 20, and a presence penalty of 1.5.Source: §C.4 · p. 13