PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evolve With a Reference:Anchored Training of Tool-Integrated Agents

Wenjie Liao, Liangjie Zhao, Zehong Cao

arXiv:2610.09856v1cs.AI

TL;DR

Self-evolving agents use their own generated tasks and feedback, but current-Executor signals can vanish at consensus or reward disagreement without indicating learning value. AnchorLoop adds a frozen previous Executor as a shared reference for Curriculum and Executor training. Across two backbones and thirteen benchmarks, it improves both mathematical and general reasoning averages over Agent0.

  • Problem

    Current-Executor feedback can yield vanishing group-relative advantages under full consensus, while Curriculum uncertainty rewards disagreement without indicating whether tasks support further learning.

    Source: §1 · p. 1

  • Method

    AnchorLoop reuses a frozen copy of the previous iteration’s Executor as a historical reference for both Executor cross-reference advantages and Curriculum agreement-based rewards.

    Source: §1 · p. 2

  • Results

    Across two backbones and thirteen benchmarks, AnchorLoop improves Math AVG over Agent0 by 2.5 points and General AVG by 2.7 to 2.8 points.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The historical reference supports self-evolving training without externally supplied tasks or answer labels, and AnchorLoop continues to show stronger gains in later iterations.

    Source: §1 · p. 2, §5 · p. 14

    During Curriculum training, the current Executor and refreshed anchor have identical parameters, so sampled agreement differences are a task-shaping proxy rather than evidence of inter-version improvement or correctness.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Self-evolving tool-integrated agents learn from tasks and feedback generated within their own training loop. A Curriculum Agent generates tasks, while an Executor Agent learns from self-consistency signals through reinforcement learning. However, relying solely on the current Executor for feedback has two limitations: group-relative advantages vanish under full consensus, while uncertainty-based curriculum rewards favor disagreement without showing whether the generated tasks support further learning. These limitations motivate an additional reference beyond the current Executor. We propose \textit{AnchorLoop}, which introduces a frozen copy of the previous iteration's Executor as a historical reference and reuses it on both sides of the training loop. For the Executor, the anchor provides a cross-reference advantage that evaluates current outputs against both current and historical majority answers. For the Curriculum, it provides an agreement-based reference based on differences in sampled majority agreement. Since the Executor and anchor have identical parameters during Curriculum training, this comparison serves as a proxy for task selection rather than evidence of inter-version improvement or correctness. Across 13 reasoning benchmarks, AnchorLoop improves over Agent0 by 2.5\% on mathematical reasoning and 2.8\% on general reasoning tasks. It also maintains higher effective-advantage variance and continues improving in later iterations as the unanchored baseline shows diminishing gains. These results demonstrate the benefit of introducing a lightweight historical reference into self-evolving tool-integrated agents without external task or answer supervision.

1. Introduction

Self-evolving agents generate their own tasks and feedback, but current-Executor signals can fail under full consensus or reward disagreement without showing learning value. AnchorLoop uses a frozen previous Executor as a shared reference, with reported gains across reasoning benchmarks.

1. Introduction

  • Current-Executor feedback has two limits: group-relative advantages vanish under full consensus, and disagreement alone does not show whether Curriculum tasks support further learning.Both learning signals are determined entirely by the current Executor’s behavior.Source: §1 · p. 1
  • AnchorLoop uses the previous iteration’s Executor as a frozen historical reference for both agents, adding temporal comparison without external tasks or answer labels.The paper distinguishes this shared-reference use from SPIN, which retains human reference responses [2].Source: §1 · p. 2
  • For Executor training, the anchor supplies a cross-reference advantage against current and anchor majority answers; for Curriculum training, it supplies an agreement-based reference from sampled majority-agreement differences.The anchor is refreshed from the previous Executor at each outer iteration and remains frozen throughout that iteration.Source: §1 · p. 2
  • During Curriculum training, the current Executor and refreshed anchor have identical parameters, so their agreement difference is a task-shaping proxy, not evidence of inter-version improvement or correctness.During subsequent Executor training, the current Executor evolves while the anchor stays fixed; the historical advantage can add relative information when majorities differ but still vanishes under full current consensus.Source: §1 · p. 2
  • AnchorLoop improves over the unanchored baseline by 2.5–2.8% points on average across two backbones and thirteen reasoning benchmarks, while continuing to improve in later iterations.The approach combines historical anchoring, cross-reference advantages for Executor training, and agreement-based references for Curriculum training.Source: §1 · p. 2

2. Preliminaries

The self-evolving loop has a Curriculum Agent generate tasks and an Executor learn through tool interaction, using the Executor’s sampled agreement as feedback. Group-relative normalization gives no relative preference when all sampled trajectories agree.

2. Preliminaries

  • The Curriculum Agent generates training tasks, while the Executor learns to solve them through multi-turn interaction with external tools; both tasks and learning signals arise within the loop.The Executor’s behavior evaluates generated tasks, and its sampled outputs supply agreement-based learning signals.Source: §2 · pp. 2–3
  • For each task, the Executor samples trajectories and uses the unique modal valid answer as a pseudo-label; invalid or missing outputs receive zero reward and do not vote.Groups without a unique modal answer are unresolved and excluded from updates requiring a pseudo-label.Source: §2 · p. 3
  • The agreement reward measures consistency with the sampled pseudo-label, not independently verified correctness, so a consistently incorrect group can receive the same reward as a correct one.Source: §2 · p. 3
  • GRPO normalizes rewards within the sampled group, but when every trajectory agrees on the same valid answer, all advantages are zero and provide no relative preference.Source: §2 · p. 3

3. Method

AnchorLoop refreshes the previous Executor as a frozen reference and reuses it for Executor advantages and Curriculum rewards across alternating training stages.

3.1. Historical Anchor for Self-Evolution

  • At each outer iteration, the previous Executor becomes a frozen anchor reused for both Curriculum and Executor training, without external tasks or answer labels.Source: §3.1 · p. 3

3.2. Executor: Cross-Reference Advantage

  • Executor training evaluates current outputs against current and anchor majorities, then centers advantages over current groups; the historical term cannot restore preference under full consensus.Source: §3.2 · pp. 3, 5

3.3. Curriculum: Anchor-Referenced Reward

  • Curriculum adds a positive sampled-agreement difference to inherited uncertainty, tool-use, repetition, and format rewards; it is a shaping proxy, not correctness or inter-version improvement.The repetition term groups candidate tasks by pairwise BLEU distance below drep.Source: §3.3 · p. 6

3.4. Alternating Training with an Iterative Anchor

  • Curriculum is updated with GRPO before Executor ADPO on same-iteration format-valid tasks; current groups are resampled each inner step, while anchor groups are cached by task and batch.Source: §3.4 · pp. 7–8

4. Experiments and Results

Experiments evaluate AnchorLoop on two base models and thirteen reasoning benchmarks, then examine its benchmark results, training dynamics, anchor choices, and Curriculum behavior.

4.1. Experimental Setup

  • The evaluation uses Qwen3-4B-Base and MiMo-7B-Base across seven mathematical and six general-reasoning benchmarks, reporting models after three outer iterations.AnchorLoop generates its complete Executor task pool without externally supplied training tasks or answer labels.Source: §4.1 · p. 9

4.2. Main Results

  • Math AVG gains were 2.5 points on both backbones, while General AVG gains were 2.8 and 2.7 points, respectively.AnchorLoop also records the highest reported score in every mathematical and general-reasoning benchmark column on both backbones.Source: Table 1 · p. 10, §4.2 · p. 10

4.3. Training Dynamics

  • By iteration 3, Math AVG reaches a 2.5-point gap over Agent0 as AnchorLoop retains later-iteration gains of 2.8 and 2.0 points.Agent0’s incremental gains decrease to 2.0 and 0.9 points in iterations 2 and 3.Source: §4.3 · p. 10
  • By iteration 3, within-group effective-advantage variance is 0.28 for AnchorLoop and 0.07 for Agent0, versus 0.43 for both initially.The higher variance does not establish that variance causes the performance difference; both advantage components vanish when all current answers are identical.Source: §4.3 · p. 11
  • The positive sampled Curriculum reference-gap fraction falls from 54.7% initially to 16.1% for Agent0 and 32.9% for AnchorLoop by iteration 3.During Curriculum training, the current Executor and refreshed anchor have identical parameters, so the gap characterizes sampled agreement rather than correctness or inter-version progress.Source: §4.3 · pp. 11–12

4.4. Anchor Analysis

  • In matched ablations, the Curriculum reference alone changes Math/General averages from 52.7/45.1 to 53.6/45.9; the Executor historical advantage alone reaches 54.4/46.7.Combining both yields 55.2/47.9; endpoint comparisons do not isolate either mechanism or establish a synergistic interaction.Source: §4.4 · p. 12
  • The iterative anchor reaches Math AVG 55.2 at iteration 3 and 56.3 at iteration 4, versus 53.0 and 53.3 for an anchor fixed to the base model.The comparison associates refreshing the anchor with stronger later-iteration performance in this setting, without isolating checkpoint age as the sole source.Source: §4.4 · p. 12, Figure 3 · p. 13

4.5. Curriculum Analysis

  • Across the evaluated 4 × 4 weight grid, the default (α, λ) = (0.6, 0.4) reaches Math AVG 55.21, and all configurations remain within 1.5 points.The result does not characterize settings outside the grid or remove the need to select the weights.Source: §4.5 · p. 13, Figure 3 · p. 13

5. Conclusion

AnchorLoop uses the previous iteration’s Executor as a frozen reference to address weakening feedback in self-evolving training. Across two backbones and thirteen benchmarks, it improves over Agent0 and continues gaining in later iterations.

5. Conclusion

  • AnchorLoop addresses weakening feedback in both self-evolving Curriculum and Executor training.Source: §5 · p. 14
  • Across two backbones and thirteen benchmarks, AnchorLoop improves Math AVG over Agent0 by 2.5 points and General AVG by 2.7 to 2.8 points.It also retains higher effective-advantage variance and shows stronger gains in later iterations.Source: §5 · p. 14
  • A frozen copy of the previous Executor supplies an independently sampled agreement reference for Curriculum training and helps construct the Executor’s cross-reference advantage.The approach uses no external tasks or answer labels.Source: §5 · p. 14

A. Experimental Details

The experimental details specify the training setup, hyperparameters, sandbox, prompts, and answer-resolution rules used for AnchorLoop. Tasks are generated and solved through Python tool interaction, with fixed formats and group-resolution rules governing training signals.

A.1. Training Setup

  • AnchorLoop uses VeRL with VeRL-Tool, runs on eight A100 80GB GPUs, and retains all Curriculum tasks that pass the fixed output format.The Curriculum-generated Answer field is used only for format checking, not shown to the Executor or used as a training label; anchor rollouts are reused across Executor inner steps.Source: §A.1 · p. 18

A.2. Hyperparameters

  • The defaults use α = 0.6 for the historical advantage and λ = 0.4 for the reference reward, with shared Curriculum reward weights across both backbones.The Executor uses G = 8 trajectories and Curriculum batches of B = 64; its KL term uses the frozen base model as πref.Source: §A.2 · pp. 18–19, Table 3 · p. 19

A.3. Sandbox Configuration

  • Python calls run in stateless subprocesses with a 30-second timeout and 1 GB memory limit, and complete tag pairs trigger execution.The evaluation does not provide a matched same-checkpoint comparison with the Python tool disabled.Source: §A.3 · pp. 19–20

B. Prompt Templates

  • The Curriculum and Executor prompts remain short and unchanged through three outer iterations; the anchor acts through rewards and advantages, not prompt edits.Source: B. Prompt Templates · p. 20

B.1. Curriculum Agent Prompt

  • The Curriculum prompt requests one self-contained, multi-step math problem with a single numerical answer that can be verified by computation.It permits high-school or early undergraduate mathematics and access to Python for intermediate checks.Source: §B.1 · p. 20

B.2. Executor Agent Prompt

  • The Executor prompt has it interleave reasoning with Python calls and produce a final answer in a fixed format for consensus parsing and binary reward.Source: §B.2 · p. 20

Executor Agent Prompt

  • The Executor is instructed to solve step by step, wrap code in complete Python tags, and place its numerical answer on a separate final-answer line.Source: Executor Agent Prompt · p. 21

B.3. Answer Parsing and Group Resolution

  • Invalid or missing outputs receive zero reward and remain in the fixed group size, while unresolved groups are omitted from updates requiring a pseudo-label.When either Curriculum comparison group is unresolved, its frontier term is zero but other reward terms remain active.Source: §B.3 · p. 21

C.1. Baseline Methods

The experiments compare AnchorLoop with untrained references and self-evolving training methods, and evaluate it on mathematical and general reasoning benchmarks. The analysis also describes how historical-advantage patterns are classified in matched training records.

C.1. Baseline Methods

  • The baseline set spans untrained models, self-evolving and self-play methods, and Agent0, which AnchorLoop augments with an iterative anchor while retaining ADPO.Trained baselines use the same backbones and evaluation protocol; reported table results were evaluated in this work.Source: §C.1 · pp. 21–22

C.2. Evaluation Benchmarks

  • Evaluation covers thirteen benchmarks, with domain averages computed as unweighted means and Agent0 and AnchorLoop scores averaged over five training runs.AMC23, AIME24, and AIME25 use mean@32; the other benchmarks use greedy pass@1.Source: §C.2 · p. 22

C.2.1. Mathematical Reasoning

  • Mathematical reasoning is evaluated on AMC23, MATH-500, GSM8K, Minerva, OlympiadBench, AIME24, and AIME25.Source: §C.2.1 · pp. 22–23

C.2.2. General Reasoning

  • General reasoning evaluation covers SuperGPQA, MMLU-Pro, BBEH, GPQA-Diamond, HumanEval, and AGIEval.Source: §C.2.2 · p. 23

D. Anchor Mechanism Analysis

  • The Anchor Mechanism Analysis examines compute and memory overhead, refresh schedules, rollout caching, and conditions where the anchor stops contributing useful signal.Source: D. Anchor Mechanism Analysis · p. 23

D.1. Historical-Advantage Structure

  • Historical-advantage statistics use matched task-and-batch records with complete, resolved groups, then classify centered vectors as zero, positively collinear, or non-collinear.The descriptive counts do not establish that the historical term causes the main performance gains.Source: §D.1 · p. 24

D.2. Cost Analysis

The cost analysis reports the runtime overhead of anchoring, compares refresh schedules, and measures the effect of caching anchor rollouts. Results apply to the tested implementations and settings rather than matched compute budgets or a universal refresh rule.

D.2. Cost Analysis

  • Cached AnchorLoop takes 24.1 hours per iteration versus 20.5 for Agent0, a 17.6% overhead on Qwen3-4B-Base.The methods are compared after the same three outer iterations, not at matched wall time or GPU hours.Source: §D.2 · p. 24

D.3. Anchor Refresh Schedule

  • The every-iteration refresh schedule has the highest reported Math AVG among the three evaluated schedules and checkpoints within the first five outer iterations.The comparison does not establish an optimal rule beyond the evaluated settings.Source: §D.3 · pp. 24–25, Table 8 · p. 25

D.4. Anchor Caching Efficiency

  • Caching reduces mean single-iteration runtime from 32.7 to 24.1 hours, a 26.3% reduction, compared with uncached AnchorLoop.The comparison covers complete implementations and does not isolate a pure systems effect or establish a theoretical lower bound on anchor overhead.Source: §D.4 · p. 25

D.5. Failure Case Analysis

The failure cases show that the anchor can provide no comparison when sampled groups lack unique modal answers, or fail to reveal a shared wrong answer. The appendix also outlines additional benchmark, iteration-dynamics, and hyperparameter analyses.

D.5. Failure Case Analysis

  • When both sampled groups tie at the highest count, the task receives Rfront = 0 and is omitted from the pseudo-label-dependent Executor update.Other Curriculum reward terms remain active.Source: §D.5 · pp. 25–26
  • When both groups share the same wrong modal answer, Ahist is zero and agreement rates cannot identify the mistake.The example concerns rollouts that count linear arrangements instead of circular arrangements and overlook rotational equivalence.Source: §D.5 · p. 26

E. Additional Experimental Results

  • The appendix’s additional experiments track individual mathematical benchmarks, MiMo-7B-Base iteration dynamics, and hyperparameter sensitivity under the Section 4.1 protocol.The section says its iteration endpoints match Table 1 and its dynamics extend Figure 2 to MiMo-7B-Base.Source: E. Additional Experimental Results · p. 26

E.1. Per-Benchmark Trajectories on MiMo-7B-Base

On MiMo-7B-Base, AnchorLoop’s advantage over Agent0 grows across outer iterations, and its larger gains are concentrated on harder mathematical benchmarks.

E.1. Per-Benchmark Trajectories on MiMo-7B-Base

  • The average gap between AnchorLoop and Agent0 widens from 0.4 at iteration 1 to 1.4 at iteration 2 and 2.5 at iteration 3.Every benchmark improves monotonically under both methods; iteration 0 is the shared tool-augmented base.Source: §E.1 · pp. 26–27, Table 10 · p. 27
  • AnchorLoop’s largest gains over the tool-augmented base are 9.2 points on Minerva, 8.5 on AIME25, and 7.9 on AIME24.GSM8K gains 4.6 points, with its starting score already above 91%.Source: §E.1 · p. 27

E.2. Iteration Dynamics on MiMo-7B-Base

On MiMo-7B-Base, AnchorLoop continues gaining on Agent0 through iteration 3, while also retaining greater effective-advantage variance. The appendix reports that the default hyperparameter setting is the highest-performing point on both tested grids.

E.2. Iteration Dynamics on MiMo-7B-Base

  • AnchorLoop’s math AVG gains are +3.3, +2.1, and +1.6 points over the three iterations, versus Agent0’s +2.9, +1.1, and +0.5.The iteration-3 gap is 2.5 points, matching the gap reported for Qwen3-4B-Base.Source: §E.2 · p. 27, Figure 4 · p. 28
  • At iteration 3, effective-advantage variance is 0.30 for AnchorLoop and 0.08 for Agent0.These are stage averages; both advantage terms vanish when all current answers in a group are identical, and the panel does not isolate a causal effect on performance.Source: §E.2 · p. 27, Figure 4 · p. 28
  • At iteration 3, the positive agreement-gap fraction is 33.7% for AnchorLoop and 17.2% for Agent0.The fraction measures how often independently sampled groups produce a positive majority-agreement gap, not correctness or newly solved tasks.Source: §E.2 · p. 28, Figure 4 · p. 28

F. Case Studies and Qualitative Examples

  • The qualitative appendix presents selected tasks and reasoning traces from Qwen3-4B-Base, including comparisons between anchor and post-update Executor outputs.The twelve displayed tasks span three outer iterations; individual outputs do not establish how often differences occur or that Rref caused them.Source: F. Case Studies and Qualitative Examples · p. 28, §F.1 · p. 29

E.3. Hyperparameter Sensitivity on MiMo-7B-Base

  • On MiMo-7B-Base, the default (α, λ) = (0.6, 0.4) reaches the highest reported Math AVG, 63.50, among the tested grid settings.Across the tested grid, Math AVG ranges from 60.47 to 63.50; results do not establish robustness beyond the tested ranges or transfer without selection.Source: §E.3 · p. 28, Figure 5 · p. 29

F.1. Extended Post-update Executor Comparisons

  • Across selected examples, anchor outputs show omitted constraints, missing reasoning steps, or limited manual search, while some updated outputs add structural steps or verification.The examples were selected for illustration and do not estimate prevalence or establish that tool use first appears at iteration 3.Source: §F.1 · pp. 29, 32, 34

F.2. Selected Curriculum Tasks Across Outer Iterations

The selected Curriculum tasks are organized into eight topic sequences, each pairing tasks from iterations 0 through 3. The examples show differences among prompts but do not establish a general progression or causal effect.

F.2. Selected Curriculum Tasks Across Outer Iterations

  • The eight selected topic sequences group tasks across iterations by shared mathematical topics, enabling comparison of the displayed prompts.The grouping does not establish that later tasks were edited from earlier tasks.Source: §F.2 · p. 34
  • Some later prompts combine more operations or request numerical verification, but the selected sequences do not establish a general increase in difficulty.They also do not show when tool use first appears during training or establish a causal effect of Rref.Source: §F.2 · p. 37

F.3. Full Executor Reasoning Traces

The four selected traces contrast anchor answers with post-update Executor answers: the anchor makes errors in unit conversion, counting, positional value, and an incomplete search, while the current majorities correct the examples. Python checks appear in the displayed post-update traces, but these cases do not measure how often such differences occur.

F.3. Full Executor Reasoning Traces

  • 204 minutes is the Executor's total; half the anchor rollouts mishandled 2 hours 24 minutes as 2.24, splitting the batch.The post-update Executor converts the time to 2.4 hours and reaches a unanimous answer.Source: §F.3 · p. 37
  • 73 residents own all four; the current majority counts exactly-k groups separately, unlike the anchor's malformed intersection expression.The displayed anchor incorrectly reports 213 residents as owning all four.Source: §F.3 · p. 38
  • 45 placements satisfy the grid constraints; the anchor's column reading uses a + d instead of the positional value 10a + d.The current majority derives the count from a + b + c = 8 and reports a unanimous answer.Source: §F.3 · pp. 38–39
  • The solutions in the 1–100 range are n ∈ {3, 20}; the anchor stops after finding n = 3 and misses n = 20.The current majority connects the problem to the negative Pell equation and verifies both solutions.Source: §F.3 · pp. 39–40
  • Python checks appear in the displayed post-update traces, but four selected cases do not establish robustness or a pattern across tasks.The traces illustrate Python use in these selected solutions only.Source: §F.3 · p. 40
Loading 2610.09856v1…