PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Agent Iteration: One Formal Framework for Iterative Policy Improvement and Recursive Self-Improvement

Hongyao Tang, Yi Ma, Pengyi Li, Yifu Yuan

arXiv:2609.13406v1cs.AIcs.LG

TL;DR

The paper addresses the lack of a unified formal account for diverse recursive self-improvement systems and their relationship to generalized policy iteration. It proposes GAI, which models learning as cyclic agent evaluation and improvement organized by two dials. The framework places existing systems on shared axes and makes RSI defects statable condition by condition, while identifying open limitations and questions.

  • Problem

    Existing recursive self-improvement instances lack a unified formal account that includes GPI as a special case and locates where its guarantees fail.

  • Method

    GAI defines an agent as modifiable system components and models learning as alternating agent evaluation and agent improvement, classified by two dials.

  • Results

    GAI describes GPI and RSI as two settings of one alternating cycle, places existing systems on the same axes, and makes RSI defects statable one condition at a time.

  • Takeaways & Limitations

    The framework is presented as a first step toward a formal characterization of RSI that makes systems comparable and supports principled analysis and design.

  • Takeaways & Limitations

    GAI describes how systems change but does not establish reachability, affordability, complexity, or compute-time claims, and several theoretical questions remain open.

Abstract

from arXiv · show

When we speak of recursive self-improvement (RSI), are we speaking of a phenomenon, a mechanism, or a prospect? Towards autonomous and evolving intelligence, RSI is being claimed at many scales, while no single framework that formally describes these emerging instances exists. Its counterpart in the classical realm, iterative policy improvement, is characterized by generalized policy iteration (GPI), a framework of broad applicability with well-understood theoretical properties, but only where the update principle and the evaluation base lie outside the agent. In this paper, we propose Generalized Agent Iteration (GAI), a formal framework that describes iterative policy improvement and RSI as two cases of a single learning paradigm. GAI defines the agent as a configuration of modifiable components within a system and models the learning process as a cycle of agent evaluation and agent improvement. Two pivotal dials then distinguish the instances: whether the improving mechanism is part of the agent and whether the standard it is measured against is grounded outside it. The former dial delineates the boundary between GPI and RSI, and the latter determines a system's polarity as anchored, goal drift, or fully self-referential. Moreover, we use these coordinates to place existing systems on the same two axes and make the defects of recursive self-improvement statable one condition at a time. We see this paper as a first step toward exploring a formal characterization of RSI that rests on the classical account, makes existing systems comparable, and provides a principled basis for analyzing and designing new ones.

1 Introduction

The paper introduces GAI to unify iterative policy improvement and recursive self-improvement, motivated by the lack of a formal account for systems that modify their own improvement mechanisms.

  • GAI describes iterative policy improvement and recursive self-improvement as two cases of one learning paradigm.
  • The framework defines learning as cyclic agent evaluation by multifaceted critics followed by agent improvement through adoption of a proposed new instance.
  • Two dials distinguish systems: whether the improving mechanism belongs to the agent and whether the evaluation standard remains grounded outside it.
  • The second dial yields anchored, goal-drift, and fully self-referential forms depending on whether the standard remains fixed, can be rewritten, or disappears externally.
  • GAI positions existing self-improvement systems on these axes and states four recursive-self-improvement defects as violations of classical GPI conditions.

2 Background

Generalized policy iteration provides a broad cycle of evaluation and improvement, while recursive self-improvement extends the idea to systems that modify the mechanisms by which they improve. The paper identifies a missing unified account and formalizes two axes for comparing these cases.

  • Reinforcement Learning and Generalized Policy Iteration (GPI): GPI alternates evaluation, which moves values toward consistency with the current policy, and improvement, which makes the policy better according to those values.The steps may be partial and interleaved, with greedy improvement as the canonical case.
  • Reinforcement Learning and Generalized Policy Iteration (GPI): GPI is implementation-agnostic, covering evaluation operators such as Bellman, temporal-difference, and Monte-Carlo updates alongside greedy, softmax, or truncated improvement operators.Its generality comes from focusing on the interaction between evaluation and improvement rather than their concrete implementations.
  • Recursive Self-Improvement Agents: Recursive self-improvement concerns systems that improve the mechanisms by which they improve, with examples spanning self-modifying agents, research loops, and evolving agent components.The literature includes empirical validation of rewrites against external benchmarks and systems that evolve parameters, prompts, memory, tools, or scaffolds.
  • The Absence of a Unified Formal Framework: The paper addresses the absence of a unified formal account that treats GPI and recursive self-improvement as neighboring cases on one continuum.Existing terms such as Gödel agents, meta-learning, and agentic loops do not make this shared structure explicit.
  • The Absence of a Unified Formal Framework: The framework distinguishes systems by whether the improving mechanism lies inside the agent and whether the evaluation standard remains externally grounded.These two axes are introduced as the basis for placing representative methods in a common comparison space.

3 The Generalized Agent Iteration (GAI) Framework

GAI formalizes iterative policy improvement and recursive self-improvement as one evaluation-and-improvement cycle over modifiable agent components. Two dials distinguish these cases by locating the modifier and determining whether the evaluation standard remains grounded outside the system.

  • 3 The Generalized Agent Iteration (GAI) Framework: GAI extends generalized policy iteration to agent-based systems whose improving components can themselves be edited.The framework retains an alternating evaluation-and-improvement structure while broadening what the agent may modify.
  • 3.1 The Self-Improving System and Agent: A system is a configuration of components including a policy, critics, modifier, and evaluation base, while the agent is a selected subset of those components.Agent improvement replaces the current agent instance with one proposed by the modifier.
  • 3.3 Polarity of Self-Improvement: Anchored, Goal Drift, and Fully Self-Referential (Dial 2): The second dial asks whether the evaluation base is grounded in external world and goal content or can move under the agent’s edits.The framework distinguishes grounded standards from bases that drift or provide no external constraint.
  • 3.2 GPI and Recursive Self-Improvement as Two GAI Instances (Dial 1): The first dial asks whether the modifier belongs to the agent: external modifiers yield GPI, whereas an internal modifier enables recursive self-improvement.When the modifier is internal, it can rewrite itself, closing recursion without an external meta-layer.
  • 3.2 GPI and Recursive Self-Improvement as Two GAI Instances (Dial 1): With an external modifier and world-reward base, GAI reduces to GPI, and standard policy or value iteration converges to an optimal policy in finite MDPs.With an internal modifier and grounded base, the corresponding case is anchored recursive self-improvement; allowing the base into the agent produces unanchored settings.

4 Connections to Existing Self-Improvement Systems

The paper applies the two GAI dials to read existing self-improvement systems uniformly, including fixed-loop practice, formal recursive self-improvers, and empirical families.

  • The two GAI dials provide a uniform reading of the systems discussed in Section 2.
  • The framework covers fixed-loop practice, formal recursive self-improvement, and the empirical families that follow it.
  • Table 3 summarizes the resulting classification.

1 Fixed Outer Loops

Fixed outer loops improve a model or harness inside a designer-scripted pipeline, while leaving the improvement mechanism itself unchanged.

  • Fixed Outer Loops: Fixed-loop pipelines script proposal, training, and selection, preventing the system from changing the control loop itself.
  • Fixed Outer Loops: Under GAI, this is the Anchored setting with the modifier frozen outside the agent.
  • Fixed Outer Loops: The agent is typically a single optimized model or harness rather than the policy-and-critic pair of the first GAI row.

2 The Gödel Machine and Its Descendants

The Gödel machine exemplifies anchored recursive self-improvement, while descendants replace proof with empirical validation or broaden what can be rewritten.

  • The Gödel Machine and Its Descendants: The Gödel machine rewrites itself when it proves that a rewrite raises a fixed external utility, making it an anchored self-improver.
  • The Gödel Machine and Its Descendants: Gödel-agent descendants replace proof with empirical validation while retaining an external benchmark, placing them in the Anchored row.
  • The Gödel Machine and Its Descendants: Hyperagents puts the task agent and meta agent in one editable program, allowing the modification procedure itself to be rewritten.Because only the scaffold is modifiable, these systems remain bounded in what they can improve.
  • The Gödel Machine and Its Descendants: In the GAI reading, a modifiable evaluator is itself an agent component because it corresponds to the modification critic U.

3 Co-Evolving Evaluators

Co-evolving evaluators move self-improvement away from the anchored setting by changing the standard used to measure progress, while fully self-referential systems remain proposals.

  • Co-Evolving Evaluators: The Red Queen Gödel Machine co-evolves the agent and evaluator, updating the standard against which self-improvement is measured.
  • Co-Evolving Evaluators: This places it in the Goal Drift row because the progress-measuring mechanism changes even if the external objective remains fixed in name.
  • Co-Evolving Evaluators: Fully self-referential self-improvement removes external signals from the loop but has been proposed rather than demonstrated.
  • Co-Evolving Evaluators: The two dials classify these departures from the anchored setting and record what each departure costs.

5 Defects of Recursive Self-Improvement

The paper identifies defects that arise when recursive self-improvement violates GPI’s external evaluation and cheap, monotone improvement conditions. These defects include unrestricted search, self-evaluation, and loss of goal faithfulness when the evaluation base is ungrounded.

  • 5 Defects of Recursive Self-Improvement: Recursive self-improvement loses GPI’s guarantees when its improvement mechanism and evaluation instrument become agent components.GPI relies on cheap, monotone improvement and externally grounded objectives and evaluators; RSI may give up both conditions.
  • 5.1 Search over Candidate Selves and Self-Evaluation: An unbounded search over candidate selves lacks a general procedure for deciding which proposed modification is best.Gödel-machine proof gates and statistical risk budgets are described as restrictions on this search.
  • 5.1 Search over Candidate Selves and Self-Evaluation: When the modification critic is itself modifiable, the evaluated object and evaluation instrument coincide, making evaluation structurally non-external.Restrictions such as proof gates remain part of the agent’s content rather than an outside constraint.
  • 5.2 The Consequences of an Ungrounded Base: Ungrounded self-consistency conditions can admit systems that serve the goal poorly because they constrain only the modifier and critic, not the goal.They also provide no faithfulness guarantee when the reported standard can move; the delusion-box example supplies an early formal analogue.
  • 5.3 Goal Drift: Table 4 organizes the four defects by triggering dial setting, violated GPI condition, and supporting evidence.Whether an internal faithfulness monitor could detect its own critic’s drift remains open.
  • 5.3 Goal Drift: When the evaluation base is mutable while the goal remains fixed, progress measurement can drift even though the objective retains its original name.The Red Queen Gödel Machine is presented as an instance whose evaluator co-evolves with the agent.

6 Discussion, Limitations, and Conclusion

The discussion presents GAI as a formal description of how autonomous systems change, while delimiting what the framework does not model. The conclusion positions GAI as a first step toward comparing and analyzing GPI and recursive self-improvement.

  • 6 Discussion, Limitations, and Conclusion: GAI describes system change but does not specify how an agent’s modification is computed, reached, or afforded.The configuration space is a reference idealization without reachability or complexity claims.
  • 6 Discussion, Limitations, and Conclusion: The framework counts alternating world actions and agent updates in steps rather than modeling their computational time scales.Practical guards are treated as external restrictions, while deciding whether guards are agent components remains open.
  • From Catalogue to Results: Several formal questions remain open, including characterizing self-consistent solutions, formalizing the reduction to GPI, and testing internal faithfulness monitoring.The paper states that only some defects are candidates for theorems, while structural defects would persist regardless of those results.
  • Conclusion: GAI unifies GPI and recursive self-improvement as two settings of one alternating cycle distinguished by two dials.The dials concern whether the improving mechanism is inside the agent and whether the evaluation standard remains grounded.
  • Conclusion: The paper uses these coordinates to place existing systems on shared axes and state recursive-self-improvement defects one violated condition at a time.The authors present this as a first step toward a formal characterization of RSI that supports comparison, analysis, and design.

Appendix: Notation

The appendix defines notation for the world, agent, system, critics, modifier, evaluation bases, and agent instances used by GAI. It formalizes the agent as a modifiable subspace within a larger system space.

  • Appendix: Notation: The world is represented as W world = (S, A, p, r), with states, actions, transitions, and rewards, while G denotes an external goal or preference.The notation also includes γ for discounting and E for expectation.
  • Appendix: Notation: O is the system’s component set, Co each component’s content space, and π and V denote the policy and action critic.The modifier m proposes system changes, U evaluates the modifier, and ρ supplies the critics’ evaluation base.
  • Appendix: Notation: χ denotes a full system in X, where X is the full system space formed from component content spaces.The displayed notation defines χ ∈ X.
  • Appendix: Notation: Ag ⊆ O identifies the agent’s modifiable components, and XAg is the corresponding agent subspace.The agent subspace is formed from the content spaces of components in Ag.
  • Appendix: Notation: ξ denotes an agent instance, while χ[ξ] denotes the system obtained by replacing its agent part with ξ.∆(Y) denotes the set of probability distributions over Y.
Loading 2609.13406v1…