PaperlayerView exact revision on arXiv

Source-linked AI summary

RSIPO: Recursive Self-Improving Policy Optimization for Agentic Reinforcement Learning via Skill-Policy Co-Evolution

Fan Zhang, Hao Wu, Jinyang Wu, Hongyu Qu, Zheng Lian, Xiao Luo, Yefeng Zheng, Bo Zhang, Haoxuan Li, Pheng-Ann Heng

arXiv:2610.15772v1cs.CL

TL;DR

Hybrid agentic RL combines trajectory-level rewards with token-level teacher supervision, but prior approaches overlook on-policy skill evolution and the reliability of teacher guidance. RSIPO co-evolves skills and policy through trajectory replay and empirical Bayes calibration. It outperforms compared methods across embodied and search-based tasks, reaching 98.6% average ALFWorld success with Qwen2.5-7B.

  • Problem

    Hybrid agentic RL methods often overlook on-policy skill evolution and assume skill-enhanced teachers provide reliable guidance, although they can underperform the student.

    Source: §1 · pp. 1–2

  • Method

    RSIPO uses trajectory replay to reward useful on-policy skill generation and empirical Bayes shrinkage to calibrate the student’s trust in teacher supervision.

    Source: §1 · p. 3

  • Results

    98.6% average task success with Qwen2.5-7B on ALFWorld; RSIPO also outperforms baselines across embodied and seven search-based task datasets.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    RSIPO evaluates self-generated skills through replay gains, using them both to reward useful skills and calibrate trust in teacher supervision.

    Source: §5 · p. 8

    Trajectory replay increases training costs, and RSIPO relies on resettable environments; extending it to non-resettable settings remains a challenge.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent focus in agentic reinforcement learning (RL) seeks to improve credit assignment in long-horizon tasks. One promising direction combines reinforcement learning with verifiable rewards (RLVR) and on-policy self-distillation (OPSD), integrating trajectory-level advantages with dense token-level supervision. This line of work often conditions a teacher model on privileged context, such as agentic skills, to score tokens generated by the student policy. Despite their progress, they largely overlook the on-policy evolution of agentic skills and the reliability of teacher supervision. Towards this end, we introduce Recursive Self-Improving Policy Optimization (RSIPO), a self-evolving agentic RL framework that enables agentic skills and the policy model to co-evolve throughout the training process. Specifically, RSIPO enables the policy itself to extract skills from completed trajectories and explicitly optimizes its skill-generation capability through trajectory replay. It further employs empirical Bayes shrinkage to dynamically calibrate the student's trust in teacher scores, accounting for the fact that skill-enhanced teacher guidance can sometimes be less reliable. Experiments with Qwen-family models on embodied interaction and search-based agentic tasks demonstrate that RSIPO outperforms all the compared baseline approaches. With Qwen2.5-7B as the base model, RSIPO achieves an average success rate of 98.6\% on ALFWorld, yielding absolute improvements of 2.5\% and 17.4\% over the second-best baseline and standard GRPO, respectively.

1 Introduction

Agentic RL trains policies to make sequential decisions in dynamic environments, while hybrid methods combine trajectory-level RLVR advantages with token-level OPSD supervision. RSIPO couples on-policy skill improvement with adaptive trust in teacher guidance and outperforms prior methods across embodied and search-based tasks.

1 Introduction

  • Hybrid agentic RL combines outcome-based trajectory advantages with dense token-level supervision from a teacher conditioned on privileged context.Agentic skills commonly provide that privileged context.Source: §1 · p. 1, Figure 1 · p. 2
  • Existing methods leave skills off-policy as the policy improves, while skill-enhanced teachers underperform students on 5.5%–22.4% of tasks.SFT skill-generation trajectories shift from predominantly failures to predominantly successes during RL, and teacher underperformance continues on 4%–11% of tasks thereafter.Source: §1 · pp. 1–2
  • RSIPO uses trajectory replay to reward on-policy skill generation and empirical Bayes shrinkage to calibrate student trust in teacher supervision.The resulting task-dependent coefficients can downweight unreliable guidance or disable distillation when the teacher is particularly unreliable.Source: §1 · pp. 2–3
  • With Qwen2.5-7B, RSIPO reaches 98.6% average ALFWorld success, 2.5% above SEED and 18.2% above GRPO+OPSD.Across multiple models, RSIPO also outperforms baselines on ALFWorld and seven search-based datasets.Source: §1 · pp. 2–3

2 Related Work

Agentic RL addresses sequential decision-making and credit assignment in multi-turn environments, while skill-based approaches add reusable natural-language experience. On-policy self-distillation supplies token-level guidance from a teacher conditioned on privileged context; RSIPO’s framework uses replay gains to evolve skills and calibrate teacher trust.

2.1 Agentic Reinforcement Learning

  • Agentic RL faces credit-assignment challenges because a single terminal reward must cover a long sequence of decisions; skill methods add reusable natural-language experience.Existing approaches often use fixed libraries, frozen generators, or skill objectives that do not explicitly account for execution utility.Source: §2.1 · p. 3

2.2 On-Policy Self-Distillation

  • On-policy self-distillation conditions the same model as teacher on privileged context, while hybrid methods combine that token-level guidance with RLVR.Prior skill sources include retrieved libraries, external analyzers, and supervised skill generation.Source: §2.2 · p. 3
  • RSIPO’s overview shows replay gains rewarding on-policy skill generation and calibrating the student’s trust in the skill-enhanced teacher.The framework connects skill improvement with teacher-supervision calibration.Source: Figure 3 · p. 4

3 Methodology

RSIPO builds on a hybrid RLVR–OPSD objective, then uses trajectory replay to optimize on-policy skill generation and calibrate trust in skill-enhanced teacher supervision. Type-level empirical Bayes shrinkage makes the replay-based trust estimates more reliable.

3.1 Preliminary and Notation

  • The preliminary hybrid objective combines trajectory-level RLVR advantages with gated token-level OPSD, where the teacher is the policy conditioned on a generated skill.The policy is initialized with supervised fine-tuning on externally written skills, and no skill is provided at test time.Source: §3.1 · p. 4

3.2 The Proposed RSIPO

  • Replay gains serve two roles: rewarding useful skill generation and calibrating the student’s trust in the skill-enhanced teacher.RSIPO uses the shrunk gain rather than the raw task-level gain for distillation weighting because noisy estimates could distort the loss weight.Source: §3.2 · pp. 5–6
  • RSIPO splits rollouts into evidence and control sets, rewrites a skill from an evidence trajectory, and replays the skill-enhanced teacher from the same initial state.Replay gains compare the replays with control rollouts; the control set is an unbiased sample of current student performance.Source: §3.2 · p. 5
  • Type-level empirical Bayes shrinkage pulls noisy or insufficiently distinctive task gains toward their type mean before they calibrate teacher trust.The shrunk gains estimate how much the skill-enhanced teacher outperforms the student; RSIPO uses them to weight OPSD, with weights bounded by ω_max.Source: §3.2 · pp. 5–6
  • RSIPO rewards skill-writing tokens according to replay gains centered on the mean for tasks of the same type.The reward is positive when the skill’s replay gain exceeds the type average and negative otherwise; invalid skills receive a penalty, while tasks with non-reproducible initial states receive zero.Source: §3.2 · p. 5

4 Experiment

Across three Qwen backbones, RSIPO leads on ALFWorld and is competitive on search-based tasks. Training analyses show how replay gains, dynamic trust, and evolving skills contribute to its performance.

4.1 Experimental Setup

  • Experiments use ALFWorld and seven search datasets, compare 13 baselines, and train Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B models.RSIPO runs for 150 steps, with eight rollouts per task and eight NVIDIA H20 GPUs per run.Source: §4.1 · p. 6

4.2 Main Results

  • 95.7%, 98.6%, and 95.0% ALFWorld success rates exceed SEED by 3.9%, 2.5%, and 3.0%, and standard GRPO by 20.7%, 17.4%, and 48.9%, respectively.These results use Qwen2.5-3B, Qwen2.5-7B, and Qwen3-1.7B, respectively.Source: §4.2 · p. 6, Table 1 · p. 7
  • Search-task accuracy is highest with Qwen2.5-7B and Qwen3-1.7B at 50.8% and 42.6%, while Qwen2.5-3B ranks second, 0.4% below SEED.The reported improvements over GRPO are 8.9%, 8.8%, and 1.8% across the three backbones.Source: §4.2 · p. 7
  • Without test-time skills, RSIPO outperforms skill-provided baselines across both benchmarks and all backbones, surpassing Skill-GRPO* by 10.3% on ALFWorld and 3.3% on search tasks with Qwen2.5-7B.The authors suggest this reflects internalization of knowledge from training-time generated skills.Source: §4.2 · p. 7

4.3 Further Analysis

  • With Qwen2.5-3B, training success rises from 63% to over 90% on ALFWorld and from 32% to 53% on search tasks; OPSD loss roughly halves on search tasks and stays stable on ALFWorld.Dynamic trust exceeds 1.1 within 20 steps on ALFWorld, but stays close to one on search tasks, where the teacher offers little extra benefit.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • On Qwen2.5-3B ALFWorld ablations, full RSIPO reaches 95.7%; adding dynamic trust and skill evolution contributes gains of 3.6% and 5.6%, and removing either costs 3.7% to 5.7%.The cumulative variants progress from 21.9% for the base model and 75.0% for GRPO to 86.4% for the SFT-initialized hybrid.Source: §4.3 · p. 8, Figure 4 · p. 8
  • On ALFWorld, Procedure skills exceed 94% in the final 45 steps as replay gain turns positive; early SFT-inherited Format or Recovery skills have negative average replay gain.The step-0 skill checks syntax, whereas the step-150 skill gives a systematic search-and-place procedure.Source: §4.3 · p. 8, Figure 5 · p. 8

5 Conclusion

RSIPO co-evolves agentic skills and the policy during training by measuring skill utility through replay and using the gains to reward skill generation and calibrate teacher trust. Experiments on embodied and search-based tasks show advantages over existing agentic RL methods.

5 Conclusion

  • RSIPO enables agentic skills and the policy to co-evolve throughout training.The framework compares skill-enhanced teacher replays with control rollouts to evaluate self-generated skill utility.Source: §5 · p. 8
  • Replay gains reward the policy for generating useful skills and calibrate its trust in teacher supervision.The paper describes this as a way to evaluate self-generated skills against control rollouts.Source: §5 · p. 8
  • Experiments on embodied and search-based tasks demonstrate advantages over existing agentic RL methods.Future work will extend RSIPO to larger models and coding and co-working tasks.Source: §5 · p. 8

Limitations

RSIPO's extra replay rollouts increase training costs, and its current approach relies on resettable environments. The paper also describes baseline families and notes a search-task-specific skill-output treatment.

Limitations

  • Replay rollouts increase training costs, creating a trade-off between training effectiveness and computational efficiency; RSIPO also relies on resettable environments, leaving live websites as a challenge.The authors cite simulators and search environments backed by offline retrieval corpora as examples of resettable settings.Source: Limitations · p. 9

A Introduction to Baselines

  • The baselines span training-free, RLVR, OPSD, and hybrid approaches; methods marked with * use skills at test time, unlike RSIPO and other unstarred methods.The hybrid group includes GRPO+OPSD, Skill-SD, RLSD, SDAR, StepOPSD, OPID, and SEED.Source: A Introduction to Baselines · p. 11

B Hyperparameters

  • On search-based tasks, invalid skills are not penalized, while valid outputs receive a small format reward.RSIPO-specific hyperparameters are otherwise shared across benchmarks and base models, except for analysis-output length and invalid-skill treatment.Source: B Hyperparameters · p. 11

C Training Algorithm of RSIPO

Each RSIPO training step samples task rollouts and generates skills, then uses replay gains and type-level statistics to update the policy. This combines skill-generation and task-execution optimization in one step.

C Training Algorithm of RSIPO

  • The policy samples G rollouts per task and writes a skill for each, producing RLVR and OPSD loss terms.The algorithm then performs an execution check for every task.Source: C Training Algorithm of RSIPO · p. 12
  • RSIPO splits rollouts into evidence and control sets, rewrites a skill from an evidence trajectory, and replays the skill-enhanced teacher K times from the same initial state to measure replay gain.The gain is defined in Eq. (5).Source: C Training Algorithm of RSIPO · p. 12
  • Type-level statistics produce shrunk gains, skill rewards, and trust coefficients that feed the overall policy objective.The resulting update optimizes skill generation and task execution in the same step.Source: C Training Algorithm of RSIPO · p. 12

D.1 Main Results on Qwen2.5-3B-Instruct

With Qwen2.5-3B-Instruct, RSIPO leads on ALFWorld and places second on search-based tasks, narrowly behind SEED.

D.1 Main Results on Qwen2.5-3B-Instruct

  • 95.7% average success rate on ALFWorld puts RSIPO 3.9% above SEED and 20.7% above GRPO.This is the best average success rate among the compared methods.Source: §D.1 · p. 12
  • 45.3% average accuracy on search-based tasks ranks RSIPO second, 0.4% below SEED.Source: §D.1 · p. 12
  • Table 3 compares RSIPO with all baselines using Qwen2.5-3B-Instruct.Source: §D.1 · p. 12

D.2 Training Dynamics of Other Base Models

Training dynamics across other base models show rising success rates and model- and benchmark-dependent patterns in OPSD loss and teacher trust. The appendixed prompts specify the contexts used for execution and skill generation.

D.2 Training Dynamics of Other Base Models

  • Training success rises steadily on both benchmarks; OPSD loss decreases on search-based tasks, while trust remains above one on ALFWorld and near one on search tasks.For Qwen2.5-7B, ALFWorld trust starts below one and exceeds one after about 20 steps; Qwen3-1.7B's ALFWorld OPSD loss falls to around 0.01.Source: §D.2 · p. 12

E Prompts

  • Task-execution prompts give the student the task, recent interaction turns, and current observation; the skill-enhanced teacher also receives the episode skill after the task.History is omitted at the first turn; the teacher uses this context for OPSD token scoring and execution-check replays.Source: E Prompts · pp. 12–13
  • Skill-generation prompts condition episode-skill instructions on trajectory outcome and request a general, answer-free strategy for search tasks.The search-task prompt also prohibits copying retrieved documents.Source: E Prompts · p. 13

F Case Study

The case studies show how the policy derives task-specific skills from selected successful trajectories and tests them through replays against controls.

F Case Study

  • Figures 9 and 10 mark successful and failed episodes, turn counts, and the selected evidence rollout alongside trajectory excerpts.The excerpts show actions at selected turns followed by the next observation or retrieved-document titles.Source: F Case Study · p. 13
  • All four ALFWorld replays succeed within 18 turns with the skill, while all control rollouts fail, yielding ∆x = 0.878 and r_x = 0.811.The selected 45-turn success informed a skill to search garbage cans and verify each pickup.Source: F Case Study · pp. 13–14, Figure 9 · p. 15
  • All four search-task replays answer correctly with the skill, compared with one of four control rollouts.The selected rollout produced a general strategy: identify the relative first, then search for the requested attribute, without naming the answer.Source: F Case Study · pp. 14–15, Figure 10 · p. 15

Related papers

Loading 2610.15772v1…