PaperlayerView exact revision on arXiv

Source-linked AI summary

R$^2$Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

Mingda Zhang, Yun Huang, Qiang Huang, Yanjin Li, Rui Qian, Qichao Zhao, Tiesunlong Shen, Na Zhao, Jian Wang, Xiaoying Tang

arXiv:2610.22740v2cs.AI

TL;DR

Orchestration learning faces fixed task pools, rollout advantage collapse, and unreliable skill evaluation. R2Orch nests self-play with task selection that preserves outcome diversity under skill withdrawal and controlled evaluation for retiring skills. It outperforms all baselines on twelve benchmarks and reports accuracy at zero skills.

  • Problem

    Existing orchestration methods rely on human-collected tasks and in-prompt skills, making them costly to scale and difficult to improve independently.

    Source: §1 · p. 1

  • Method

    R2Orch nests Proposer–Director self-play with Recursive Learnability for selecting tasks under candidate-skill withdrawal and Skill Withdrawal based on controlled evaluation.

    Source: §1 · p. 2, §4.2 · p. 4, §4.3 · p. 5

  • Results

    R2Orch outperforms all baselines on every benchmark across twelve benchmarks, improves six other Worker backbones without retraining, and keeps accuracy when tested with zero skills.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The two loops form a cycle in which failures become skills, skills become parameters, and withdrawals move the Director toward zero-skill deployment.

    Source: §1 · p. 2

    Initial skills are customized by hand for each dataset, so automatic derivation of initial skills for a new domain remains open.

    Source: Limitations · p. 9

Abstract

from arXiv · show

In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R$^2$Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R$^2$Orch consistently outperforms baselines across question answering, math, code, and interactive tasks. Our code is available at https://github.com/beita6969/R2Orch.

1 Introduction

R2Orch frames orchestration as nested self-play and skill-withdrawal loops, addressing fixed curricula, collapsing rollout advantages, and noisy skill evaluation. Across twelve benchmarks, it outperforms all baselines and supports zero-skill accuracy.

1 Introduction

  • Existing methods rely on pre-collected task pools, can lose group-based learning advantages when rollouts converge, and use unreliable skill-evaluation signals.When a skill is withdrawn from a frontier task, rollouts may become all-wrong; single noisy scores can also lead skill libraries to drift.Source: §1 · pp. 1–2
  • R2Orch nests an inner Proposer–Director self-play loop within an outer loop that improves how the inner loop learns.Recursive Learnability rewards task choices that preserve different rollout outcomes with a candidate skill withheld; Skill Withdrawal uses randomized hidden-skill rollouts as controlled evaluation.Source: §1 · p. 2
  • Failures become skills, skills become parameters, and withdrawals direct the Proposer toward new failures as the Director moves toward zero-skill deployment.The two loops form a cycle of skill internalization and withdrawal.Source: §1 · p. 2
  • Across twelve benchmarks, R2Orch outperforms all baselines, improves six other Worker backbones without retraining, and keeps accuracy when tested with zero skills.The evaluation spans question answering, math, code, and interactive tasks.Source: §1 · p. 2

2 Related Work

Prior work learns orchestration over fixed task pools, uses self-play to generate tasks and distill failures, and supplies or evolves skills in prompts or training. R2Orch combines self-play with skill withdrawal and bases task selection and retirement on within-rollout evidence.

Learned Orchestration and Self-Play

  • Learned orchestration searches workflows, graph edits, or planners over tasks collected in advance, while self-play methods reward regret, uncertainty, or learning progress and may distill failures into skills.Group-based RL offers no advantage when a task’s rollouts all reach the same outcome; R2Orch instead scores tasks with the candidate skill withheld.Source: §2 · p. 2

Skill Evolution and Self-Improvement

  • Prior work supplies skills at inference or during training, and withdraws them using success-rate thresholds, preset budgets, or leave-one-skill-out validation.Other approaches score rollouts with and without a skill or use paired tests for self-modifications.Source: §2 · pp. 2–3
  • Self-improvement systems revise memory, code, or answers, while R2Orch nests self-play in a skill-withdrawal loop and decides retirement through controlled evaluation within each randomized rollout group, without extra rollouts.The comparison distinguishes R2Orch’s group-level withdrawal evidence from keeping or retiring skills based on one noisy score.Source: §2 · p. 3

3 Preliminaries

The preliminaries define agent graphs and Director trajectories, then state the target: a Director trained with a Proposer should build successful graphs for deployment without skills.

3 Preliminaries

  • An agent graph is a directed graph of role-prompted agents with model routes, hand-off edges, layer assignments, and a designated output agent.A Director trajectory is a complete sequence of validated atomic edits from an empty graph to FINISH, with feedback returned after each edit.Source: §3 · p. 3
  • The Director’s edits include setting models and layers, considering relations, deleting agents, setting the output, and finishing; frozen Workers execute the resulting graph.The target is a Director that builds a successful agent graph without any skill in its context, trained on tasks chosen by a Proposer.Source: §3 · p. 3
  • The deployment target uses the verifier’s correctness after Workers execute the graph, with an empty skill set marking zero-skill deployment.Skills scaffold training, while the Proposer chooses tasks whose rollouts reach different outcomes.Source: §3 · p. 3

4 Method

R2Orch nests Proposer–Director self-play inside an outer skill-withdrawal loop, with both loops using shared randomized rollout groups. Recursive Learnability guides task choice, while Skill Withdrawal uses controlled evaluation to decide when skills can be removed.

4.1 Recursive Self-Play Orchestration

  • Self-play gains on a fixed scaffold minus withdrawal costs account for zero-skill success; each skill’s cost depends on its benefit and retrieval frequency.The method assigns Recursive Learnability to driving inner gains and Skill Withdrawal to limiting each withdrawal cost.Source: §4.1 · p. 4

4.2 Recursive Learnability

  • Recursive Learnability rewards tasks whose outcomes remain varied when the candidate skill is hidden, steering the Proposer toward learnable tasks as the scaffold shrinks.Its target vanishes when hidden-skill success is 0 or 1 and peaks at one half.Source: §4.2 · pp. 4–5

4.3 Skill Withdrawal

  • Skill Withdrawal removes a skill when controlled evaluation on the same randomized rollout groups shows the Director performs as well without it.Random hiding makes the within-group contrast an unbiased estimate of the skill’s effect; posterior decisions govern withdrawal and admission.Source: §4.3 · p. 5

5 Experiments

Across twelve benchmarks, R2Orch leads six baseline families, including on held-out benchmarks at zero skills. Its experiments also report backbone transfer, contributions from both loops, and findings on task selection and skill withdrawal.

Main results (RQ1)

  • R2Orch is best on all twelve benchmarks against six baseline families, and its lead holds on held-out benchmarks at zero skills.The comparisons span six baseline families.Source: §5 · p. 6

Backbone transfer (RQ2)

  • Without retraining, the Director improves all six other Worker backbones on every benchmark; weaker backbones gain most, narrowing the IID range from 25.7 to 6.4 points.OOD mean gains are largest on interactive tasks, whose many steps compound hand-offs, and smallest on math, which is near its ceiling.Source: §5 · pp. 6, 8, Figure 4 · p. 7

Recursive Self-Play Orchestration (RQ3)

  • Both loops are needed: skills provide early successes on hard tasks, while the Proposer avoids groups with uniform outcomes; zero-skill accuracy overtakes accuracy with skills midway.Most rollouts hide the candidate skill, training the Director to succeed without it.Source: Figure 4 · p. 7, §5 · p. 8

Recursive Learnability (RQ4)

  • Mixed outcomes arise about four times as often at intermediate success rates as at p=0 or p=1, and the trained Proposer learns to skip solved candidates.Learnability measured with skills in context trails on every IID benchmark and the mean OOD score.Source: §5 · p. 8, Figure 5 · p. 8
  • Because the hidden part already measures learnability, task selection requires no extra rollouts and uses the fewest rollouts per effective group; these groups yield the most distinct graph types.The caption describes Figure 5's selection-cost panels and Figure 4's graph-type panel.Source: Figure 4 · p. 7, §5 · p. 8, Figure 5 · p. 8

Skill Withdrawal (RQ5)

  • All 12 skill withdrawals pass the card-specific contrast criterion, with posterior E[Δ+] at most 0.019; around each withdrawal, seed cards no longer raise held-out accuracy.Success-rate thresholds, preset budgets, and validation gaps trail the paired contrast.Source: §5 · p. 8, Figure 5 · p. 8

6 Conclusion

R2Orch combines recursive task selection with skill withdrawal, and leads on all twelve benchmarks at zero skills.

Conclusion

  • R2Orch nests two loops: Recursive Learnability picks tasks learnable without the candidate skill, and Skill Withdrawal retires skills that stop helping.The conclusion presents these loops as the framework's core.Source: §6 · p. 8
  • The framework leads on all twelve benchmarks at zero skills.This is the conclusion's stated benchmark scope and deployment condition.Source: §6 · p. 8
  • The reported endpoint is zero-skill performance, after the Director has learned to work without withdrawn skills.The conclusion frames progress around skills no longer needed by the Director.Source: §6 · p. 8

Limitations

R2Orch begins with skills customized by hand for each dataset; automatically deriving initial skills for a new domain remains open.

Limitations

  • Initial skills are customized by hand for each dataset.This is the starting condition for the reported process.Source: Limitations · p. 9
  • Subsequent withdrawals, revisions, and distillations build on those initial skills.The limitation concerns the starting skills, not whether later skill updates occur.Source: Limitations · p. 9
  • Automatically deriving initial skills for a new domain remains an open problem.The passage identifies this as unresolved.Source: Limitations · p. 9

Ethical considerations

The study evaluates on public benchmarks as their creators intend and uses no human participants, annotators, or personal data.

Ethical considerations

  • R2Orch is evaluated on public benchmarks used as their creators intend.The passage describes the study's evaluation setting.Source: Ethical considerations · p. 9
  • The study involves no human participants, annotators, or personal data.These exclusions define the human-data scope reported.Source: Ethical considerations · p. 9
  • Agents that orchestrate tools can be misused, and a self-improving orchestrator shares this risk; the authors recommend sandboxed execution.They release the code for inspection.Source: Ethical considerations · p. 9

A Theory and Proofs

The appendix provides theoretical analyses of R2Orch’s outcome-diversity curriculum learning and skill-withdrawal mechanisms, including randomized skill contrasts, adaptive evidence, and Bayesian decisions. It connects these analyses through round accounting and clarifies assumptions and the relationship between the anchored curriculum’s practical update and ideal objective.

A.1 Conditional setup and policy likelihoods

  • Under fixed task and context sampling, the complete rollout-record score gives an unbiased reward gradient; a terminal-graph score suffices only when reward means are parameter-independent given the graph.Different editing histories that yield the same graph may have different costs, in which case the complete record is required.Source: §A.1 · p. 12

A.2 KL tilting and controllable progress

  • A policy-space KL tilt improves expected score according to the score’s accessible variance, which can be smaller than raw outcome variance under stochastic execution or restricted policies.The score projection onto the policy’s score span determines accessible outcome diversity; when every graph has the same expected correctness, redistributing graphs cannot improve success.Source: §A.2 · p. 13

A.3 Advantage energy, collapse, and shrinkage

  • For n≥2 independent Bernoulli(p) outcomes with centered advantages A_i=Y_i−Ȳ, expected advantage energy is (n−1)p(1−p); a realized group collapses exactly when all outcomes agree.A finite group can collapse at any p, including p=1/2. Posterior-mean plug-in learnability exceeds posterior expected learnability by 4Var(p | y).Source: §A.3 · pp. 13–14

A.4 Mixed-context gradients and the last skill

  • With separate centering, for G > 2 and n_h > n_v ≥ 1, hidden weight exceeds n_h/G by (n_h−n_v)/[G(G−2)]; a 3/2 split yields weights 2:1 and relative scale 3/4.A singleton part contributes no centered gradient.Source: §A.4 · p. 15

A.5 Round accounting and interacting withdrawals

  • Sequential withdrawals can have different costs from single-skill removals, so bounds accumulate over successive contexts and depend on matching evaluation and testing scopes.Redundant skills can each have zero individual removal cost while removing both incurs a cost; successive posterior bounds sum without requiring independent skill effects.Source: §A.5 · pp. 15–16

A.6 Tracking under separated scaffold changes

  • When scaffold modes change increasingly far apart, the stochastic inner process spends a vanishing fraction of its clock time in switch transients, or converges to the final equilibrium if switching stops.The tracking result assumes bounded iterates and noise, stable attracting equilibria for finitely many modes, and step sizes tending to zero with divergent total sum.Source: §A.6 · p. 16

A.7 Randomized contrasts, bootstrap means, and decisions

  • Randomized within-task contrasts estimate a skill’s effect, whereas group success alone cannot identify that effect even with a fixed group ratio.The contrast is unbiased for the task-specific effect; adaptive task selection preserves within-group randomization, though the ledger estimates average predictable effects and can drift from a current target.Source: §A.7 · p. 17
  • The Dirichlet Bayesian bootstrap estimates the recorded mean contrast, and posterior-risk rules govern withdrawal and admission decisions.Withdrawal is optimal when posterior expected positive effect is at most epsilon; admission is optimal when the posterior probability of a nonpositive effect is at most alpha.Source: §A.7 · pp. 17–18

A.8 The anchored curriculum and distinct pairs

  • The Gibbs distribution uniquely maximizes the anchored curriculum objective, but the practical distinct-pair update can have different stationary points because its pair weights affect the gradient.A two-candidate example demonstrates the difference; practical stationary points also depend on reward scaling and the trust region.Source: §A.8 · p. 18

A.9 The three propositions

  • The proposition proofs connect finite-run accounting and tracking to zero-skill evaluation, and show that randomized contrasts support scope-specific effects and withdrawal decisions.The tracking result is conditional on its assumptions; the contrast is unbiased for the actual test-task distribution’s mean effect, with bootstrap aggregation and Bayes decisions carried into round costs.Source: §A.9 · pp. 18–19
  • The policy-improvement proof links first-order gain to expected energy and quadratic allocation, while distinguishing parametric-Director and stochastic-Worker limits.For a parametric Director, the accessible part is bounded; with stochastic Workers, graph-controllable progress governs instead.Source: §A.9 · p. 19

B Estimators and Implementation Details

The implementation scores proposer tasks for learnability per compute, uses randomized evidence to assess skills, and updates the Director and Proposer with separate objectives.

Shrinkage and compute factor

  • The Proposer’s task score shrinks success estimates toward a dataset-level running rate and scales learning progress by rollout duration.Tasks lacking both hidden and visible parts use the whole-group success rate with the same collapse penalty.Source: B Estimators and Implementation Details · p. 19

Prior record and cost term

Director update

  • The Director uses a clipped PPO surrogate with whitened advantages, a base-model KL anchor, and threshold-based dropping of near-zero-advantage samples.A small share of dropped samples is retained for the anchor.Source: B Estimators and Implementation Details · p. 19

Proposer update

  • The Proposer compares two distinct task picks using opposite reward-difference advantages and a KL-regularized update constrained by a trust region.The trust region preserves stochastic choice within each candidate list.Source: B Estimators and Implementation Details · p. 19

Graph canvas

C Training Loop and Skill Ledger

Each round updates skills and the Director while tracking skill versions and evidence; admission and withdrawal rules are checked in a fixed order.

Training loop

  • The editor revises the least helpful skill first and distills new skills from groups where both hidden and visible parts fail.A changed or newly added skill starts a fresh ledger key; records for unchanged skill identities remain pooled across Director updates.Source: C Training Loop and Skill Ledger · p. 19

Skill ledger

  • Admission uses N_t, while an unadmitted trial skill is discarded after 3n_min records; W_t is checked first, preventing both rules from acting on one skill in a round.The passage specifies the rule ordering but not the threshold definitions.Source: C Training Loop and Skill Ledger · p. 19

Evidence and retesting

  • Each posterior uses records from the last 30 cycles, and 30% of tasks retest withdrawn skills so N_t can restore them.Tasks with no skills run all G rollouts without skills and add no record.Source: C Training Loop and Skill Ledger · p. 19

D Experimental Details

The experiments compare orchestration methods and ablations under shared workers, training tasks, and compute, with zero-skill testing and specified benchmark splits and evaluation procedures.

Benchmarks and splits

  • Six IID benchmarks supply training tasks and IID tests, while six OOD benchmarks are used only for testing; each test has 128 items, except AIME 2026 with 30.A separate held-out set contains 32 tasks per IID dataset, or 30 earlier-year AIME problems; the benchmark pools and sizes are released with the code.Source: D Experimental Details · p. 20

Metrics

  • Evaluation uses exact match and F1 for four QA datasets, accuracy for four math or science datasets, Pass@1 for MBPP+, success rate for ALFWorld and WebShop, and resolved-issue share for SWE-Bench.The passage also specifies Ans EM in Table 2.Source: D Experimental Details · p. 20

Baselines and fairness

  • All methods share frozen Workers, training tasks, and compute budgets, and R2Orch is tested with zero skills.The baseline set includes prompted models, SFT and GRPO, workflow search, RL-trained orchestration, skill internalization, and self-play with skill evolution.Source: D Experimental Details · p. 21

Ablation variants

  • Each ablation changes one component while holding Workers, training tasks, rollouts, and zero-skill testing fixed; whole-component removals target the orchestration loop, Recursive Learnability, or Skill Withdrawal.Other variants remove an orchestration part, replace a proposer reward, or replace the withdrawal rule with validation-tuned thresholds.Source: D Experimental Details · p. 21

Proposer objectives

  • The proposer-objective comparison replaces only its objective while keeping the Director update, scaffold, withdrawal rule, and training rollouts fixed.Each objective uses 24 groups of five rollouts per cycle; scores are five-run means at zero skills.Source: Figure 5a

Training dynamics

  • Training dynamics track each cycle’s checkpoint on held-out tasks with and without current skills, alongside the number of distinct graph types built.Candidate-task classes are based on prior success rate, and Figure 4d tracks the proposer’s probability on solved candidates as a 10-cycle moving average against an untrained random-choice model.Source: D Experimental Details · p. 21

Selection evidence and withdrawals

  • Withdrawal evidence counts seen-task picks by prior success rate and reports held-out gains for three seed cards together, while each withdrawal itself follows the posterior for that card’s contrast.Figure 5f shows the gain at checkpoint 0 and as a mean over 10 checkpoints around each withdrawal with a 95% interval; it is dataset-level evidence, and the MedQA topology card was restored after retesting.Source: D Experimental Details · p. 21

Compute and reporting

  • Training runs for 200 cycles at about 106 A100 hours, and the reported means use five runs for Table 2 and Figures 4e and 5a.Figure 5c,d report proposer-selection rollout and token costs; Table 4 lists the main-text constants, and PPO clip and learning rate set step size β.Source: Table 4 · p. 20, D Experimental Details · p. 21

Related papers

Loading 2610.22740v2…