PaperlayerView exact revision on arXiv

Source-linked AI summary

RSI-Opt: Recursive Self-Improvement Optimizer Agents For Model Training

Binghang Lu, Yige Sun, Guang Lin

arXiv:2610.32026v1cs.AI

TL;DR

Fixed optimizer-agent policies do not retain lessons across training runs. RSI-Opt evolves an editable policy using evidence from earlier runs, improving training-suite performance while transfer and adaptation outcomes vary across models and tasks.

  • Problem

    Optimizer recipes are often chosen in advance, and a fixed optimizer-agent policy does not retain useful lessons for subsequent runs.

    Source: §1 · p. 1

  • Method

    RSI-Opt evolves an optimizer-agent policy comprising a prompt, playbook, code library, and decision cadence, using a meta-agent and archive of evaluated policies.

    Source: §1 · pp. 1–2

  • Results

    Evolved policies improve the training suite for all three LLMs, but held-out performance varies: GPT-5.5 and GPT-6.1 improve mean summary scores across task groups, while Opus 5.5 has a lower aggregate held-out score.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    Most observed benefit on seen tasks comes from the initial optimizer change, while later decisions benefit both baseline and evolved agents on unseen tasks.

    Source: §6 · p. 15

    Evolution uses only three tasks, and broader task coverage, more independent evolution runs, and a separate policy-validation suite would strengthen generalization claims.

    Source: §6 · p. 15

Abstract

from arXiv · show

Conventional neural-network training typically follows a predefined optimization recipe, motivating a LLM agent that monitors training dynamics and adjusts optimizer settings as training progresses, but a fixed agent policy does not accumulate experience across runs. We introduce RSI-Opt, a plug-and-play Recursive Self-Improvement Optimizer Agents which evolves the knowledge that guides this process: a prompt, a playbook of procedures, a library of tested optimizer code, and a decision cadence. A meta-agent rewrites this policy using evidence from earlier runs, and an archive retains evaluated policies for further evolution. Across 5 generations on three training tasks, the meta-agents develop reusable procedures and code and conduct their own CPU proxy experiments. RSI-Opt substantially improves optimizer-agent performance, with representative raw-space summary $z$-scores increasing from $+10.48$ to $+14.33$ on the evolution suite with Opus 5.5 and from $-0.36$ to $+3.11$ on held-out tasks with GPT-5.5 through cross-model policy transfer. On the Burgers PDE, it reduces mean relative $L_2$ error by approximately 2.7-fold with Opus 5.5 and 8.8-fold with GPT-6.1 over the fixed-prompt agent baseline, although PDE transfer varies across LLMs and tasks. Controlled comparisons show that the evolved policy combines reusable optimization knowledge with adaptation to each task: initial decisions capture most observed gains on seen tasks, while continued adaptation improves outcomes on unseen tasks. These findings demonstrate how procedures and executable tools accumulated across training runs can improve optimizer-agent performance and support transfer to scientific machine learning.

1 Introduction

RSI-Opt addresses the optimizer agent’s failure to carry lessons between runs by evolving an editable policy. The study examines the resulting artifacts, transfer, and adaptation.

1 Introduction

  • RSI-Opt evolves the optimizer agent’s prompt, playbook, code library, and decision cadence using evidence from earlier runs.An archive retains evaluated policies for continued exploration; the underlying LLM weights, meta-agent instructions, and evaluator remain fixed.Source: §1 · pp. 1–2
  • Evolved policies improve the training suite for all three LLMs, while held-out outcomes vary by model and task group.GPT-5.5 and GPT-6.1 improve mean summary scores in every task group; Opus 5.5 has a lower aggregate held-out score despite improving four of six held-out tasks.Source: §1 · p. 2
  • The study analyzes reusable update modules, parameter-role checks, scheduling procedures, and CPU proxy experiments alongside measured outcomes.The harness, policies, transcripts, and results are released.Source: §1 · p. 2

2 Related work

Prior work studies learned or discovered update rules, agent and policy improvement, and within-run optimizer adaptation. RSI-Opt combines cross-run policy evolution with within-run optimizer-agent adaptation and tests transfer to PDE tasks.

Learned and discovered optimizers

  • Learned optimizers meta-train update rules across tasks, while symbolic search discovers algorithms such as Lion [4]; their update programs are typically fixed before a target run.The passage also names established optimizers including Adam, AdamW, Shampoo, SOAP [30], and Muon as building blocks.Source: §2 · p. 2

Agents that improve programs and policies

  • Prior work evolves prompts, code-generation scaffolds, agent systems, and algorithms; the Darwin Gödel Machine [35] maintains an archive of modified agents for further exploration.Persistent lessons and skill libraries are also mechanisms for carrying experience across episodes [31].Source: §2 · pp. 2–3

Optimizer agents within a training run

  • OPAL studies within-run optimizer and diagnostic revisions, whereas RSI-Opt targets policy improvement across runs by accumulating procedures and executable tools.The comparison also distinguishes RSI-Opt from population-based training, which adapts hyperparameters through selection and replacement across runs [11].Source: §2 · p. 3

Optimization for physics-informed training

  • PDE tasks test transfer beyond the ML evolution tasks, but do not establish a new PDE discretization or a universally better PINN optimizer.The passage situates this work amid concerns about imbalanced PINN losses and ill-conditioned landscapes.Source: §2 · p. 3

3 Method

RSI-Opt adapts optimizer programs within runs and evolves the policy guiding that adaptation across runs. The policy combines a prompt, playbook, code library, and decision cadence; an archive supports policy exploration and selection.

3.1 Within-run adaptation

  • Within a run, the agent inspects losses and diagnostics at scheduled decisions and edits optimizer and diagnostic programs while model, data, loss, and update budget remain fixed.The final evaluation metric is withheld from the in-run agent, and training pauses during decisions, making latency affect wall-clock cost rather than update count.Source: §3.1 · p. 3, Figure 1 · p. 4

3.2 The evolving policy

  • A policy specifies the prompt template, natural-language playbook, Python code library, and decision cadence; its references are read-only during a run.The policy fixes the number of decision opportunities, and generation 0 adds pointers to initially empty playbook and library files.Source: §3.2 · pp. 3–4

3.3 Across-run policy evolution

  • The outer loop evolves optimizer-agent policies by using run evidence to revise parent policies, evaluating valid children, and retaining them in an archive.Archive scores guide parent sampling and final selection; lower-scoring valid children remain available for exploration.Source: §3.3 · pp. 4–5
  • Policy scores aggregate task-and-seed z-scores against static references, with positive values indicating improvement over the static reference.Policies use the same task and seed set during evolution; this normalization is not a significance test, and low static variability can produce large z-scores.Source: §3.3 · p. 5
  • Final aggregate comparisons use raw-space summary scores recomputed from printed mean metrics and static standard deviations, without changing policy selection.Rounded table entries limit precision, and summary statistics do not support reconstructing individual-seed confidence intervals or seed-win counts.Source: §3.3 · p. 5
  • The recursive improvement is limited to the in-run agent’s policy; LLM weights, meta-agent instructions, scoring rule, and task splits remain fixed.A matched-budget control would be needed to isolate inheritance’s benefit over independent proposals.Source: §3.3 · p. 6

3.4 Implementation

  • Both agents use a tool-calling loop with bash and file-editing tools in isolated network namespaces, without GPU or API-credential access.The inner-loop training code remains untouched, and long conversations are summarized after exceeding 160k tokens.Source: §3.4 · p. 6

4 Experimental setup

The experiments evaluate optimizer agents on fixed training tasks and transfer tasks, using multiple LLMs and controlled baselines. Policies are selected using training-suite results and then assessed on fresh seeds and held-out tasks.

Tasks

  • Table 1 identifies the tasks and defines static results as each task’s own recipe, reported as mean ± standard deviation over five seeds.Each task’s model, loss, data protocol, update budget, and starting optimizer are fixed.Source: §4 · p. 6, Table 1 · p. 7
  • Nine fixed training tasks span a three-task evolution suite, three held-out ML tasks, and three held-out PDE solvers evaluated by relative L2 solution error.Held-out tasks are for transfer evaluation rather than policy selection; the PDE tasks differ from the evolution suite in architecture, objectives, sampling, and metric.Source: §4 · pp. 6–7

Models

  • The selected policy is evaluated unchanged with Claude Opus 5.5, GPT-5.5, and GPT-6.1, while Opus 5.5 serves as both meta-agent and in-run agent during evolution.The policy is evolved using the training suite; held-out outcomes do not replace the preselected final policy.Source: §4 · pp. 7–8

Baselines and budget controls

  • Full baseline and RSI-Opt runs share 20 decision opportunities, available schedule families, a backend, and a per-run cost cap, though cadence and realized resource use can differ.Baselines include each task’s static recipe, an optimistically selected tuned-static reference, and a fixed-prompt single-agent baseline.Source: §4 · p. 7

Policy selection and final evaluation

  • Figure 2 tracks parent-child links and the best archive score during evolution, then compares per-task z-scores for generation 0 and the selected final policy.Its plotted evolution uses training-suite seeds 100 and 101; fresh-seed comparisons are reported separately.Source: Figure 2 · p. 8
  • The three highest-scoring eligible policies are re-evaluated on seeds 102 and 103, and the highest four-seed mean determines the final policy.Final comparisons use fresh seeds 200, 201, and 202; these seeds do not guide evolution or final-policy selection.Source: §4 · pp. 7–8

5 Results

Across five generations, the meta-agents accumulated reusable procedures and executable tools, including a change tested in a CPU proxy before later in-run use. The evidence also has boundaries: the proxy-to-task contribution was not isolated, and indirect held-out exposure cannot be ruled out.

5.1 What knowledge evolves?

  • The archive’s best policy, p016, reached +14.63 and had a four-seed mean of +14.55, narrowly exceeding two other eligible candidates.The generation-0 score was +10.30; fresh-seed comparisons were reserved for later comparisons.Source: §5.1 · p. 8
  • Most accumulated knowledge was in procedures and library modules, including parameter-role checks, schedule helpers, diagnostics, and a hybrid Muon updater.A recurring procedure was to make an early evidence-based change and retain it unless later observations justified another edit.Source: §5.1 · pp. 8–9
  • A CPU proxy test found that adding Muon on hidden UNet convolutions reduced validation loss in every tested configuration, including 0.188 to 0.174 at 3000 steps.Subsequent in-run agents applied the proposal, and the UNet evolution score rose from about +9 to +19 on both generation-3 seeds; isolating its causal contribution would require a controlled replay.Source: §5.1 · p. 9
  • Two policies descended from a meta-agent that read a held-out task description were excluded, but archive-mediated indirect exposure could not be ruled out.The audit identified no additional direct reads of held-out material.Source: §5.1 · p. 9

5.2 How do agents compare with static recipes?

Optimizer agents can outperform static recipes, but the evolved policy’s gains vary across tasks and LLMs. It improves the seen-task score and transfers positively to GPT-5.5 and GPT-6.1, while Opus 5.5’s held-out aggregate score declines.

5.2 How do agents compare with static recipes?

  • With Opus 5.5, the single-agent baseline improves GPT validation loss from 4.480 to 4.332 and ViT accuracy from 55.8% to 62.6% versus static recipes.Its mean heat-equation error is approximately 6,000 times lower than the static recipe’s.Source: §5.2 · pp. 9–10

5.3 What does the evolved policy transfer?

  • On fresh seeds from the seen training suite, RSI-Opt raises Opus 5.5’s score from +10.48 to +14.33, a difference of +3.85.These are new seeds on tasks used during evolution, not unseen tasks.Source: Table 3 · p. 10, §5.3 · p. 10

5.2 How do agents compare with static recipes?

  • The single-agent baseline is not uniformly better: GPT-5.5 worsens both Burgers error and LSTM loss relative to static recipes, and outcomes depend on task and LLM.The oracle learning-rate sweep’s mean held-out score is +0.80, compared with +5.22 for the Opus baseline.Source: §5.2 · p. 10

5.3 What does the evolved policy transfer?

  • With the same policy transferred unchanged to GPT-5.5, the mean held-out score rises from -0.36 to +3.11.The held-out ML and PDE summary-score differences are +5.36 and +1.59, respectively.Source: §5.3 · p. 10
  • For Opus 5.5, the held-out aggregate score falls from +5.22 to +4.87, although RSI-Opt improves the mean metric on four of six tasks.It reduces mean Burgers error by about 2.7× and heat error by about 2.2×, but loses 1.1 ViT accuracy points and increases fractional-PDE error.Source: §5.3 · p. 10, Table 4 · p. 11
  • With GPT-6.1, Burgers mean relative error falls from 4.82 × 10^-3 to 5.46 × 10^-4, about 8.8 times lower.Its overall held-out summary score rises from +0.80 to +4.30, although its LSTM loss worsens and fractional-PDE error changes little.Source: §5.3 · pp. 10–11, Table 4 · p. 11
  • Portable knowledge may help when it compensates for weaker task-specific choices but interfere with an already effective strategy; improvements are not uniform across models or domains.GPT-agent library adoption shows that the artifact is used, but its performance determines whether adoption is beneficial.Source: §5.3 · p. 11

5.4 When does evolution specialize?

Training-suite scores rose at each evaluated checkpoint, while held-out ML performance peaked after generation 1. Later policies increasingly recorded recipes for the small evolution suite, motivating broader validation despite its added evaluation cost.

5.4 When does evolution specialize?

  • The training-suite score rose at every checkpoint, from +10.48 to +14.33, while held-out ML performance peaked after generation 1 at +9.75.The corrected trajectory comparison does not establish the ordering of PDE checkpoints.Source: §5.4 · p. 11
  • Later policies increasingly documented measured recipes for the three evolution tasks, consistent with specialization to the small training suite.The playbook table grew from 6 to 18 runs; a broader evolution or separate policy-validation suite could help, at additional evaluation cost.Source: §5.4 · p. 11

5.5 Reusable code and online adaptation

The controls distinguish reusable optimizer code from online adaptation: the library template is applied without an agent, while other conditions replay a first change, allow one decision, or permit 20 decisions. Initial changes capture most of the observed training-suite benefit, but continued decisions improve held-out ML outcomes.

5.5 Reusable code and online adaptation

  • The unedited library template approaches the full agent’s UNet score and performs similarly on heat, but reaches 35% ViT accuracy versus 56% for the static recipe and worsens LSTM and Burgers performance.It recovers roughly half the small GPT’s improvement and has an originally reported held-out ML group score of -21.80; the tested defaults do not establish that every carefully selected fixed optimizer would fail.Source: §5.5 · p. 12
  • Training-suite scores are +14.56 with the first change frozen, +14.33 for the full agent, and +13.82 with one decision.These observed means place most of the seen-task benefit in the initial adaptation, but do not establish equivalence or show that later decisions are never useful.Source: §5.5 · p. 12, Table 5 · p. 13
  • On held-out ML tasks, first-change frozen and one-decision RSI-Opt scores are +6.82 and +5.05, versus +8.23 for the full agent.The corresponding baseline scores are +6.35, +6.12, and +8.71; continued adaptation improves held-out outcomes for both policies, with no extra advantage for RSI-Opt in the size of that later-decision benefit.Source: §5.5 · p. 12, Table 5 · p. 13
  • On Burgers, final error is 5.5 × 10^-2 with the first change frozen and 1.5 × 10^-3 with the full agent.On the LSTM, the frozen first-change condition ends at z −1.9, compared with +3.3 for the full evolved agent; these are final-outcome comparisons, not before-and-after measurements within one trajectory.Source: §5.5 · pp. 12–13

5.6 Transfer to PDE training

PDE transfer is selective across models and tasks, although both GPT models improve mean metrics on all three PDE tasks. The component ablations remove the library, playbook, or evolved prompt changes from the final Opus policy, while leaving cadence unisolated.

5.6 Transfer to PDE training

  • GPT-5.5 improves its PDE summary score from -1.90 to -0.32, and GPT-6.1 improves from +1.61 to +1.74; both improve mean metrics on all three PDE tasks.GPT-5.5 reduces mean Burgers error by about 1.7× and heat error by about 15×, while GPT-6.1 achieves approximately 8.8-fold Burgers and 2.0-fold heat improvements.Source: §5.6 · p. 13
  • Opus’s PDE summary score falls from +1.72 to +1.51: Burgers and heat improve, but fractional-PDE error worsens.Heat benefits particularly from schedules that reduce the learning rate near the end of training; the static recipe’s final errors span 0.008–0.73 across seeds, and static tuning was limited.Source: §5.6 · p. 13

5.7 Component ablations

  • The Opus component ablations remove the library, playbook, or evolved prompt changes while retaining the other components; cadence is not isolated by these removals.Generation 0’s prompt replaces the evolved template in the prompt ablation.Source: §5.7 · p. 13

5.8 Agent behavior and resource use

Evolved policies change how agents work: they often make a main change early and reuse evolved modules, while fixed-prompt agents more often build and revise programs from scratch. The reported component and usage measurements are descriptive, and the stated decision limits do not equalize realized compute.

5.8 Agent behavior and resource use

  • Evolved-policy agents often make their main change early and reuse evolved modules, while fixed-prompt agents more frequently construct and revise programs from scratch.Table 7 reports component use and program-changing decisions; these descriptive measurements do not show that individual components caused performance changes.Source: §5.8 · pp. 13–14, Table 7 · p. 14
  • Table 6 ablates the evolved Opus policy; full-policy and baseline scores use recomputed raw-space summaries, while ablation ML scores retain their original precision.PDE and combined held-out aggregate columns are omitted where consistent raw-space ablation scores cannot be reconstructed.Source: Table 6 · p. 14
  • Equal decision opportunities and per-run caps constrain agent access but do not equalize realized compute.Token accounting omits optimizer overhead, CPU proxy experiments, GPU time, and pauses for LLM calls, so it does not establish total-cost savings or future runs needed to amortize evolution.Source: §5.8 · p. 14

6 Discussion and limitations

RSI-Opt keeps accumulated knowledge in inspectable text and code rather than changing LLM weights, but transfer varies with the model and task. The authors qualify conclusions about generalization, cost, controls, uncertainty, and safety given the experiments’ scope.

6 Discussion and limitations

  • GPT-5.5 and GPT-6.1 improve mean raw-space summary scores in every task group, while Opus gains on the evolution suite but has a lower aggregate held-out summary score.Opus improves four of six held-out task means; GPT-6.1 improves mean Burgers error by approximately 8.8× but worsens LSTM loss, so transfer depends on policy, model, and task.Source: §6 · p. 14
  • Initial optimizer changes account for most seen-task benefit, while both baseline and evolved agents benefit from later decisions on unseen tasks.The tested unedited library template fails on some architectures, and the comparisons do not isolate reasoning from additional observations, code modifications, or compute; a stronger development-selected frozen recipe could improve on that template.Source: §6 · p. 15
  • Evolution uses three tasks and two seeds per policy, and the early held-out ML peak is consistent with specialization to this small suite.Re-evaluating top candidates and using fresh final seeds reduce reliance on single estimates but do not make evolution tasks unseen; broader coverage, independent runs, and a separate validation suite would strengthen generalization claims.Source: §6 · p. 15
  • With three final seeds per cell, uncertainty estimates have limited precision, and z-score magnitude alone does not establish statistical significance.A z-score measures improvement relative to static-recipe seed spread; small reference variability can make a task dominate the mean, so raw metrics and per-task outcomes are needed alongside aggregates.Source: §6 · p. 15
  • The controls do not isolate recursive inheritance against matched-budget generation-0 search or the archive against restricted-parent search.Static comparisons cover starting recipes, an oracle learning-rate sweep, and one unedited library template; schedule-matched Adam, optimizer hybrids, and suitable PINN recipes would better separate known optimizer choices from agent adaptation.Source: §6 · pp. 15–16
  • Runs use a single GPU and relatively short update budgets, so transfer to longer, multi-GPU training is untested.Equal decision limits do not equalize actual charges or wall-clock time; full cost accounting would include outer-search evaluations, CPU proxies, GPU use, cache pricing, and LLM pauses.Source: Table 8 · p. 15, §6 · p. 16
  • Sandbox restrictions reduce specific risks but do not prove evaluation manipulation is impossible or certify the absence of indirect exposure.Restrictions were introduced during the project rather than applied uniformly from its start, and generated optimizer code runs inside the training process; safety and robustness of reasoning-augmented LLM agents remain open concerns.Source: §6 · p. 16

A Appendix

The audit reviewed in-run and meta-agent tool calls, finding two early reads of OPAL example files and excluding the exposed policy lineage from later selection. It does not resolve whether information could have propagated indirectly through archive summaries.

Appendix

  • The audit covered 528 in-run transcripts with 18,895 tool calls and 20 meta-agent transcripts with 949 tool calls.These counts describe the audited corpus, not a guarantee about unlogged actions or information flow.Source: A Appendix · p. 19
  • The in-run scan flagged 1 call, and review found no direct reads of other runs, task scripts, data, logs, or API credentials.Source: A Appendix · p. 19
  • Two meta-agents read OPAL example files before filesystem restrictions; the exposed policy and its descendant were excluded from later parent and final selection.One read an MNIST example not used as a benchmark task; the other read the description of a noisy-label example designated as held out.Source: A Appendix · p. 19
  • Archive summaries could have carried information to other lineages, a provenance question the documented audit does not resolve.The authors state this is not evidence that indirect propagation occurred.Source: A Appendix · p. 19
  • Final isolation hid other runs, task scripts, data, logs, the paper, and example tasks, but shared-filesystem paths remained visible.Logged shell and file-tool calls alone do not establish all accesses available to imported code or rule out every form of evaluator manipulation.Source: A Appendix · p. 19

Related papers

Loading 2610.32026v1…