PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Improvements in Modern Agentic Systems: A Survey

Zhe Ren, Yimeng Chen, Dandan Guo, Guowei Rong, Tonghui Li, R. B. Xiong, Qingfeng Lan, Wenyi Wang, Li Nanbo, Yibo Yang, Mingchen Zhuge, Jürgen Schmidhuber

arXiv:2607.13104v1cs.AIcs.CLcs.LG

TL;DR

Modern agents combine a foundation model with a scaffold, making their evolution mechanisms a central object of study. This survey formalizes and organizes self-improvement into slower foundation-model and faster scaffold pathways.

  • Problem

    A central question is how agents evolve through mechanisms that induce new policies rather than being treated as static systems, since those mechanisms drive their evolution.

    Source: §9 · p. 55

  • Method

    The survey formalizes agents as closed-loop systems and classifies self-improvement into foundation-model and scaffold pathways, integrating applications and evaluation.

    Source: §1 · p. 6, §9 · p. 55

  • Results

    The synthesis distinguishes slower parametric foundation-model improvement, driven by demonstrations, evaluative feedback, or exploration, from faster non-parametric scaffold updates to prompts, memory, tools, or full scaffolds.

    Source: §10 · p. 57

  • Takeaways & Limitations

    Reliable continuous self-improvement requires attention to security factors and challenges, with mechanism-level benchmarking and end-to-end evaluation both considered.

    Source: §1 · p. 6

    Truly open-ended recursive self-improvement remains a grand challenge; current systems operationalize it as bounded, verifiable loops within human-designed objectives and safety protocols.

    Source: §6.4 · p. 41

Abstract

from arXiv · show

Self-improving autonomous agents are moving from research prototypes to deployed systems. The primary goal is controllable evolution, or adaptation, from experience with minimal or even no human input. This survey frames modern self-improving agents as adaptive systems that convert experience into accumulated capability gains. We offer a system-level framework that represents a modern agent as a configuration coupling a foundation model with an operational scaffold of prompts, memory, tools, and control logic. Within this framework, self-improvement is formalized as a self-induced update operator that obtains and commits updates to model parameters or scaffold components. We organize prior work by update target and by the signals that drive change, then review applications and discuss evaluation, before closing with open problems and future directions. For convenience, we track technical updates on https://github.com/selfimproving-agent/awesome-Self-Improving-Agents.

1 Introduction

The survey frames self-improvement in foundation-model agents as adaptation through changes to either the model or its surrounding scaffold, driven by signals from interaction. It traces the field’s evolution and organizes prior work while examining applications, evaluation, and open challenges.

Introduction

  • Foundation-model agents revive self-improvement by using natural language to reduce the search space for modifications, building on earlier self-referential work such as Schmidhuber’s framework (Schmidhuber, 1987).The historical lineage includes systems that modify their code or policies and meta-learning systems that discover learning algorithms.Source: §1 · pp. 5–6
  • Modern agents pair a foundation model with a scaffold of instructions, memory, tools, and control logic that constructs context, selects actions, and enforces constraints.The survey uses “scaffold” to emphasize the modifiable structures surrounding the model.Source: §1 · p. 5
  • The survey’s taxonomy separates slower, more stable foundation-model updates from faster, more reversible scaffold changes, and classifies both by update target and signal source.Scaffold changes can affect prompts, memory, tools, or end-to-end control logic.Source: §1 · p. 6
  • The survey connects its taxonomy to applications including software engineering, web navigation, and scientific discovery, then reviews evaluation protocols, security factors, and challenges to reliable continuous improvement.It contrasts mechanism-level benchmarking with end-to-end evaluation.Source: §1 · p. 6

2 Historical Context and Theoretical Foundations

The section traces self-improvement from early adaptation and self-reference to explicit modification of learning rules, architectures, and agent systems. Foundation models provide a scalable substrate for rapid contextual adaptation and persistent improvement, while renewing questions of evaluation, control, and safety.

2.1 Foundational Concepts (1790s-1960s)

  • Self-improvement’s roots include error-driven parameter adjustment, while mid-century adaptive systems changed behavior but retained fixed architectures and learning algorithms.Least squares and the Ceres prediction example illustrate error-based adjustment and generalization; Gödel’s self-referential formal statements offered a blueprint for code manipulation, while Good’s Intelligence Explosion remained conceptual.Source: §2.1 · pp. 7–9

2.2 Symbolism and Heuristic Self-Modification (1960s–1980s)

  • Symbolic systems treated knowledge and heuristics as editable objects: AM searched over mathematical concepts, while EURISKO modified strategies but relied on users to evaluate and prune them.This dependence on external evaluation highlights the need for robust internal evaluation signals and verification in autonomous recursive improvement.Source: §2.2 · pp. 9–10

2.3 Connectionism and the Emergence of Meta-Learning (1980s–2000s)

  • Connectionist approaches made learning rules and learning procedures themselves targets of optimization, extending self-modification into continuous weight spaces and recursive learning systems.Fast-weight programmers and self-referential weight matrices modify weights using feedback; the Gödel Machine adopts self-modifications only when it can prove they improve expected cumulative reward, with every component subject to modification.Source: §2.3 · pp. 10–11

2.4 Formal and Architecture-Level Self-Improvement (2000s-2020s)

  • Work on formal self-reference examined safe reasoning under uncertainty and resource limits, while engineering advanced constrained self-modification, architecture search, and self-play.These efforts addressed risks including self-deception and reward tampering, and helped establish the theoretical and engineering foundations for modern foundation-model agents.Source: §2.4 · p. 11

2.5 Scalable Foundation Models and Agentic Systems (2020s–Present)

  • Foundation models split modern self-improvement into rapid, training-free adaptation and persistent parameter updates, with broader systems combining parameter and scaffold loops.In-context learning links to transient key-value-cache changes and fast-weight programmers; slower loops include RLHF and self-critique, reflection, skill libraries, and systems that iterate data, reward models, and policies.Source: §2.5 · pp. 11–12

3 Definitions

The section defines an FM-based agent through its model parameters and operational scaffold, then formalizes self-improvement as durable updates driven by the agent’s own execution. It distinguishes parameter and scaffold changes, reusable skills, and connections to established learning paradigms.

3.1 Formulation of Agentic Systems

  • An FM-based agent couples foundation-model parameters θt with a scaffold of prompts, memory, tools, and control logic; transient execution state affects behavior but is not intrinsic architecture.The parameters and scaffold jointly determine the induced policy, while transient state Xt is typically discarded or reset at task boundaries.Source: §3.1 · pp. 12–13

3.2 Formal Definition of Self-Improvement

  • SI-FMA defines self-improvement as a self-induced operator that uses execution-generated signals to durably update intrinsic model parameters or scaffold components.Execution can generate trajectories, critiques, verification results, or proposed edits; the update commits changes beyond transient state evolution and may consolidate successful strategies.Source: §3.2 · p. 13
  • Self-improvement has complementary modes: self-generated data or supervision can update model parameters, while direct action can modify prompts, memory, tools, or control logic.Parameter learning uses experience induced by the agent’s policy; scaffold changes are typically faster, more reversible, and more context-dependent than parameter changes.Source: §3.2 · pp. 14–15
  • A skill is a reusable serialized update to the agent’s configuration, stored in a substrate such as a tool, instruction, memory, weights, or control logic.Skills can be repeatedly invoked or used once as portable installers; object-level skills perform tasks, whereas meta-level skills modify the agent itself and can participate in a self-referential improvement loop (Schmidhuber, 1987; 1993; 2003b).Source: §3.2 · p. 15

3.3 Connections to Related Learning Paradigms

  • Relative to classical RL, parameter updates resemble policy optimization, but scaffold updates alter the effective decision process through explicit structured changes and may use self-generated supervision.Classical RL typically assumes a fixed state representation and action space and external scalar rewards, whereas scaffold updates can reshape those structures and feedback can come from the agent’s own trajectories, judgments, or verifiers.Source: §3.3 · p. 16
  • SI-FMA overlaps with online learning when agents adapt repeatedly during deployment, but also includes batched or offline updates; its active-learning connections extend to self-generated feedback and curiosity-driven exploration.Parameter updates face distribution shift and forgetting, while scaffold updates improve transparency but risk memory poisoning, tool-semantic drift, and brittle templates; active acquisition choices depend on query policy, intrinsic objective, and feedback trustworthiness.Source: §3.3 · pp. 16–17

4 A Taxonomy of Existing Approaches

The taxonomy organizes self-improvement approaches by update target and self-generated learning signal, further grouping scaffold updates by the component modified.

4 A Taxonomy of Existing Approaches

  • The taxonomy provides a common frame for comparing approaches, distinguishing foundation-model updates from scaffold updates and using execution-generated signals to drive change.Source: §4 · p. 17

4.1 Foundation Model Improvement

  • Foundation-model improvement updates model parameters while leaving the scaffold unchanged, amortizing learned capabilities across future interactions but typically requiring more time and computation.Parameter history can support validation and rollback if an update degrades performance or violates constraints.Source: §4.1 · p. 17
  • Foundation-model updates are classified by signals: generated demonstrations, evaluative feedback such as rewards or critiques, and exploratory interaction trajectories with environment-grounded outcomes.Source: §4.1 · p. 17

4.2 Scaffolding Improvement

  • Scaffolding improvement keeps model parameters fixed and structurally updates prompts, memory, tools, or the full architecture, typically enabling fast, reversible, task-specific adaptation.These are updates to the agent’s intrinsic configuration, not transient shifts in working memory.Source: §4.2 · pp. 17–18

5 Foundation Model Improvement

This section surveys foundation-model improvement methods that update model parameters or agent policies using internally generated or interaction-derived learning signals. It covers demonstrations, evaluative feedback, and exploratory experience, including grounded task environments and simulated proxies, along with their design challenges and reliability risks.

5 Foundation Model Improvement

  • The framework treats foundation-model self-improvement as parameter updates driven by self-generated demonstrations, internal evaluations, or experience from external or simulated environments.These categories are not mutually exclusive; practical systems may combine them and are classified by their dominant signal source and objective.Source: §5 · pp. 18–19

5.1 Intrinsic Generative Demonstrations

  • Agents can use their compressed semantic priors to generate instruction-response pairs, reasoning trajectories, or execution logs, shifting the bottleneck from human annotation to generation and filtering.Strategies include evolving instruction difficulty, filtering for consistency or correctness, curriculum generation, and uncertainty-driven targeted LoRA adaptation (Acikgoz et al., 2025).Source: §5.1 · pp. 19–20
  • Demonstrations can supervise reasoning, tool use, or multimodal tasks, but defective or low-diversity generated data can reinforce errors, cause forgetting, or produce pattern collapse.Safeguards include retaining benchmark data, stronger verification and filtering, diversity-aware selection, targeted generation, and human auditing.Source: §5.1 · pp. 20–21

5.2 Intrinsic Evaluative Feedback

  • Intrinsic evaluative feedback uses endogenous judgments—such as scores, preferences, confidence, consistency, critiques, or revisions—as parameter-update signals without new environment interaction.Rubric-based judgments, agreement across samples, and corrective critiques provide different forms of supervision; their reliability depends on robust criteria and evaluators.Source: §5.2 · pp. 21–23
  • Evaluative loops can reduce reliance on human labels but risk reinforcing shared blind spots, superficial criteria, or correlated errors, so they benefit from external anchors and evaluator separation.Suggested safeguards include human or context-based validation, tracking evaluator disagreement, and reviewing scoring criteria.Source: §5.2 · pp. 23–24

5.3 Extrinsic Exploratory Experience

  • Extrinsic exploratory experience updates parameters from policy trajectories, whose readable artifacts can support reinforcement learning, fine-tuning, preference construction, and failure analysis.Interaction may be costly, rewards sparse or delayed, verifiers gameable, and learned world models counterfactual; methods use grounded task environments, simulated proxies, or both.Source: §5.3 · p. 24

5.3.1 Interaction with Grounded Task Environments.

  • Grounded interaction updates foundation-model policies from environmental trajectories, using executable verifiers such as unit tests (Agent-RLVR (Da et al., 2025)) or learned reward models for web and UI trajectories (WebRL (Qi et al., 2025); UI-Genie (Xiao et al., 2025)).Trajectory feedback can also support step-level supervision and rewards that combine task success with execution efficiency.Source: §5.3.1 · pp. 24–25
  • Self-generated tasks retain extrinsic supervision: agents choose tasks, but execution or environmental feedback determines success, as in Absolute Zero (Zhao et al., 2025a) and ETO.Absolute Zero generates tasks and solutions through self-play; ETO learns contrasts between successful and failed trajectories in fixed environments.Source: §5.3.1 · p. 25
  • AgentGym (Xi et al., 2024) provides unified interaction, evaluation, and training APIs across agent tasks, reducing the engineering cost of iterative experience collection and comparison.Source: §5.3.1 · p. 25

5.3.2 Interaction with Simulated Proxy Environments

  • A learned world model predicts next states and rewards from current states and actions, letting policies gain simulated experience instead of repeatedly querying the original environment.This can improve sample efficiency and reduce exploration cost or risk when direct interaction is slow, expensive, or unsafe.Source: §5.3.2 · p. 25
  • Foundation-model world models use pretrained generative priors to predict policy-compatible observations, reducing task-specific interaction and enabling planning, trajectory synthesis, and failure analysis.WMPO (Zhu et al., 2025b) optimizes over imagined rollouts to avoid costly physical trial and error; GLoW (Kim & seung-won hwang, 2026) reports 100–800× fewer real interactions than RL baselines using structured textual memory.Source: §5.3.2 · pp. 25–26
  • These methods face sparse rewards, costly real interaction, proxy overfitting, reward hacking, capability regression, hallucinated dynamics, and trajectory-length/context-window constraints.The section highlights more reliable verifiers and reward models, calibrated world-model uncertainty, and training that avoids sacrificing broad foundation-model capabilities.Source: §5.3.2 · p. 26

6 Scaffolding Improvement

Scaffolding improvement adapts an agent’s operational components while keeping foundation-model parameters frozen, using execution-derived signals to drive updates. The section moves from component-level changes to full-scaffold reconfiguration, with version history supporting validation and rollback.

6 Scaffolding Improvement

  • Scaffolding updates change prompts, memory, tools, or control logic while freezing foundation-model parameters, using execution-derived signals and version history for validation and rollback.The scaffold is formalized as Σt := (pt, mt, Tt, gt), and updates as Σt+1 = IMPROVEΣ(Σ1:t; St).Source: §6 · p. 26

6.1 Prompt-based improvement

  • Prompt-based improvement refines task communication, objectives, and constraints through the agent’s instruction layer without changing foundation-model parameters.It updates the prompt template using the learning signal: pt+1 = IMPROVEp(p1:t; St).Source: §6 · p. 27

6.2 Memory-based improvement

  • Memory-based improvement stores, prunes, and retrieves past experiences to support cumulative learning, long-horizon reasoning, and trustworthy adaptation.The memory update is mt+1 = IMPROVEm(m1:t; St).Source: §6 · p. 27

6.3 Tool-based improvement

  • Tool-based improvement autonomously refines and manages callable external modules, extending execution capabilities beyond the foundation model’s native limits.The tool set is updated as Tt+1 = IMPROVET(T1:t; St).Source: §6 · p. 27

6.4 Full scaffolding improvement

  • Full-scaffolding improvement reconfigures the codebase and operational logic, integrating perception, reasoning, and execution; scaffold edits can be composed, with archive-based exploration and stronger acceptance tests.Full-scaffolding methods subsume component-level edits.Source: §6 · p. 27

6.1 Prompt

Prompt self-improvement updates reusable instruction layers or dynamic interaction context through increasingly informative learning signals, from scalar scores to critiques, population selection, and directional textual guidance. The survey distinguishes these update targets and organizes prompt-refinement methods by their signal.

6.1.4 Textual Gradient Optimization

  • Across prompt-based approaches, richer signals—from scores to critiques, population selection, and directional guidance—enable less heuristic, more targeted, increasingly automated and sample-efficient refinement without changing foundation-model parameters.The survey characterizes this progression as turning prompt optimization from an ad hoc practice into a signal-driven improvement loop.Source: §6.1 · pp. 27–28, §6.1.4 · p. 31

6.1 Prompt

  • Reusable system prompts alter an agent’s behavioral prior, whereas context construction refines dynamic conditioning for particular interactions.Context construction includes exemplar selection, retrieval assembly, and long-term playbooks.Source: §6.1 · p. 27

6.1.1 Scalar-Feedback Optimization

  • Scalar-feedback methods search discrete prompt spaces using task scores, with APE proposing and scoring candidates and OPRO using prior prompt-score trajectories to guide later proposals.Other approaches use reinforcement learning, Bayesian optimization in a continuous latent space, or scalar human-preference scores; BPO optimizes user inputs without changing model weights.Source: §6.1.1 · p. 28

6.1.2 Qualitative-Feedback Refinement

  • Qualitative-refinement methods use critiques or suggestions to target prompt updates; Reflexion stores failure reflections in memory, while MAPS induces reusable rules from failures and injects them into prompts.Self-Refine and multi-agent debate use self-critique for transient output correction, while later scaffolding methods persist feedback to change prompting policy or instructional context.Source: §6.1.2 · pp. 28–29
  • ACE treats prompts and memory as evolving text playbooks, using a Generator–Reflector–Curator pipeline to curate qualitative feedback while mitigating brevity bias and context collapse.Scrable iteratively refines a structural system prompt using qualitative evaluation and stops when text quality meets a predefined standard.Source: §6.1.2 · p. 29

6.1.3 Population-Based Evolution

  • Population-based evolution selects prompts by fitness and uses LLM-driven semantic crossover and mutation; Promptbreeder also evolves the mutation instructions, while GEPA guides variation with textual reflections on successes and failures.DEEVO uses multi-agent debate win rates as fitness when scalar scores are unavailable; GEPA’s reflective guidance makes search more targeted and sample-efficient.Source: §6.1.3 · pp. 29–30

6.1.4 Textual Gradient Optimization

  • Textual-gradient optimization gives directional feedback an update rule, with TextGrad backpropagating qualitative gradients through multi-component language systems and MetaTextGrad optimizing the instructions that compute and apply those gradients.The survey notes that textual gradients could potentially be translated into low-rank parameter updates, linking prompt refinement with model fine-tuning.Source: §6.1.4 · p. 30

6.2 Memory

The section frames memory as an externalized, signal-driven scaffold that evolves its objects, structure, and processing while the foundation model remains frozen. Its self-improvement loop selectively creates, organizes, retrieves, updates, and deletes memory in response to experience.

6.2 Memory

  • Memory evolves as a state of stored objects and their organization, updated by execution signals such as retrieval failures, task feedback, or capacity limits; the focus is external memory with a frozen foundation model.The framework distinguishes Memory Objects, Memory Structure, and Memory Processing, with processing operations governing both what is stored and how it is organized.Source: §6.2 · p. 31

6.2.1 Memory Object

  • Processed trails compress raw trajectories into reusable strategies; when exact surface details resist summarization, agents selectively retain high-value artifacts validated through trial and error (Wang et al., 2025j; Ouyang et al., 2025b; Zhao et al., 2024a; Suzgun et al., 2025).Execution feedback filters or compresses experiences, mitigating context-window limitations and storage costs (Cao et al., 2025).Source: §6.2.1 · pp. 31–32
  • Explicit memory objects favor inspectability and controllability, while implicit latent objects offer compact, fast associative access but are harder to audit and may drift over time.Explicit objects include compressed interaction trails, selectively preserved artifacts, and dynamically maintained external knowledge; implicit approaches include latent sequences, reconstructed hidden states, and KV-cache augmentations.Source: §6.2.1 · pp. 32–33

6.2.2 Memory Structure

  • Flat structures preserve chronological traces but face recency bias and retrieval scaling limits, while hierarchies compress histories into reusable layers but risk fragmenting evidence when their taxonomy mismatches tasks.Hierarchical designs include task-oriented plans, semantic abstraction, tiered storage, and separate transient and persistent multimodal memory (Lee et al., 2024b; Sun & Zeng, 2025; Helmi, 2025; Koley, 2025; Cheng & Schwing, 2022; Song et al., 2024a).Source: §6.2.2 · pp. 33–35
  • Graph memories enable associative, multi-hop reasoning but add maintenance overhead, whereas vector retrieval scales semantic recall yet can retrieve similar but decision-irrelevant items (Chhikara et al., 2025; Wu et al., 2025e; Ravuru et al., 2024; Singh et al., 2025a).Systems mitigate these risks with relational or causal edges, hybrid relevance-recency-importance signals, re-ranking, adaptive controllers, and specialized stores (Rasmussen et al., 2025; Wang et al., 2025e; Park et al., 2023; Tan et al., 2025b; Wang & Chen, 2025).Source: §6.2.2 · p. 35

6.2.3 Memory Processing

  • Creation selectively distills raw interactions, while reading combines relevance signals, structure-aware traversal, retrieval gating, or fetched trajectories to guide behavior without parameter updates (Salama et al., 2025; Park et al., 2023; Zhang et al., 2025d; Zhou et al., 2025a).Creation patterns include semantic compression, context-aware add/update/delete/no-op decisions, and write policies optimized for downstream utility; over-writing increases noise, while under-writing sacrifices capability.Source: §6.2.3 · p. 36
  • Feedback-guided updates refresh, consolidate, or attenuate memory, and deletion prunes obsolete or redundant entries; over-pruning loses knowledge, while under-pruning preserves noise (Liang et al., 2026b; Xu et al., 2025f; Zhang et al., 2025g; Bach, 2025; Kang et al., 2025).The lifecycle observes saliency, creates and organizes objects, retrieves context for action, evaluates outcomes, then updates or deletes entries to retain high-value knowledge and prune noise.Source: §6.2.3 · pp. 36–37

6.3 Tool

Tool governance metacognition turns tool use from static execution into a self-directed cycle that expands and improves an agent’s operational capabilities. It acts through dynamic routing, iterative refinement, and autonomous tool creation.

6.3 Tool

  • Tool governance metacognition lets agents assess tool necessity, utility, and reliability through dynamic routing, iterative refinement, and autonomous creation.Together, these dimensions shift tool use from static lookup to a generative cycle that manages and expands the agent’s operational tools.Source: §6.3 · p. 37

6.3.1 Dynamic Tool Routing

  • Tool routing spans retrieval and graph methods, learned policies, and interactive clarification or repair, balancing coverage and adaptability against reliability, feedback needs, and compute.Retrieval methods leverage prior trajectories or tool dependencies; learned policies face data and distribution-shift risks, while interactive routing uses clarification and local repair to address ambiguity and execution failures.Source: §6.3.1 · pp. 37–38

6.3.2 Iterative Tool Refinement

  • Iterative refinement debugs fragile tools and gates what enters reusable skill repositories, preventing stored errors from compounding through future reuse.The loop generates, executes, and revises using error traces and environment feedback; systems improve it with specialized critique, reusable API abstractions, or refined tool documentation.Source: §6.3.2 · pp. 38–39

6.3.3 Autonomous Tool Creation

  • Autonomous tool creation expands capabilities by turning insufficient-tool cases or exploration into executable functions, while requiring validation and integration to avoid brittleness.Systems automate tool lifecycles from code synthesis through dependency installation and debugging, and use standardized protocols or registration pipelines to integrate new tools without destabilizing routing.Source: §6.3.3 · p. 39

6.4 Full Scaffolding

Full scaffolding self-improvement treats an agent’s entire operational logic and codebase as mutable, enabling structural reorganization through self-referential updates. Although open-ended recursive self-improvement remains a grand challenge, current systems implement bounded, verifiable loops under human-designed objectives and safety protocols.

6.4 Full Scaffolding

  • Full scaffolding improvement makes the agent’s operational logic and codebase mutable, while current systems remain bounded by objectives, benchmarks, verification, and safety protocols.The self-referential loop can evolve the improver alongside the agent, potentially discovering new update strategies and enhancing intrinsic evolvability.Source: §6.4 · pp. 40–41
  • The agent’s improver uses execution traces and evaluations to generate code changes, often as patches, which verifiers such as tests and safety checks gate before adoption.Scaffolding is represented as a serializable program executed in an environment such as an interpreter, compiler, or sandbox.Source: §6.4 · p. 40
  • AlphaEvolve iteratively improves algorithms using evaluator feedback, while ShinkaEvolve (Lange et al., 2026) searches for program discoveries and optimizations across tasks.ShinkaEvolve uses balanced parent sampling, novelty-based rejection sampling, and bandit-based selection of an LLM ensemble.Source: §6.4 · p. 41
  • ADAS (Hu et al., 2025), EvoFlow, Self-Taught Optimizer (Zelikman et al., 2024), and Agent Symbolic Learning explore designs, workflows, recursive improvement, and joint component updates.EvoFlow evolves diverse workflows for cost–performance trade-offs; Self-Taught Optimizer selects among LM-generated versions, while Agent Symbolic Learning updates prompts, tools, and pipelines using natural-language losses and gradients.Source: §6.4 · p. 41
  • Gödel Agent, Darwin Gödel Machine (Zhang et al., 2026d), Huxley-Gödel Machine (Wang et al., 2026b), and Live-SWE-Agent exemplify self-referential and evolutionary approaches.DGM grows an archive and tree of diverse coding agents; HGM uses clade-level metaproductivity, and Live-SWE-Agent evolves its scaffolding at runtime for software issue solving.Source: §6.4 · p. 41

7 Applications

This section surveys self-improvement across application domains, including general computer control, focusing on how environments and interaction-derived feedback drive iterative updates. It also covers domain-specific constraints such as environment fidelity, scalability, cost, safety, contextual verification, and transfer.

7.1 Software Engineering

  • Executable feedback makes software engineering a tractable testbed for persistent scaffold revision and model-parameter updates, but each route has distinct trade-offs.Agents can rewrite or evolve scaffolds during or between tasks, while test outcomes can train models; scaffold changes are faster but may overfit, whereas model training is costlier and vulnerable to reward hacking and evaluation artifacts.Source: §7.1 · pp. 42–43

7.2 Web Navigation and Automation

  • Web agents improve through cross-session memory and grounding updates or interaction-driven model training, but changing interfaces, sparse feedback, and safety risks complicate transfer.Mind2Web, WebArena, VisualWebArena, WorkArena, and BrowserGym provide data or evaluation substrates rather than self-improving methods; drift-aware evaluation and safe exploration are needed to limit site-specific overfitting and risky actions.Source: §7.2 · pp. 43–44

7.3 Games and Strategic Reasoning

  • Games support scalable self-play and reusable skill or memory scaffolds, yet narrow opponent distributions, strategic cycling, and persuasion or deception risks demand broader evaluation and constraints.Voyager and Odyssey reuse skills and curricula, while ExpeL and Richelieu accumulate lessons or reflections across interactions; fixed-opponent evaluation can mislead when strategies are non-transitive.Source: §7.3 · pp. 44–45

7.4 Scientific Discovery

  • Scientific agents accumulate reusable research workflows and tools, while parameter updates depend on valid feedback; weak verification, heterogeneous evidence, and hazardous actions remain central challenges.Systems such as AI Scientist and AI co-scientist revise plans, protocols, and tool policies; progress requires reproducibility-centered evaluation, evidence tracking, standardized protocols, and governance that bounds risk.Source: §7.4 · pp. 46–48

7.5 Embodied AI and Robotics

  • Embodied agents improve through iterative robot-generated data and policy updates or evolving curricula, memory, and safety logic, but sim-to-real transfer, hardware cost, and irreversible failures constrain practice.MEDAL++ enables reset-free practice by learning to perform and undo tasks, while RoboGen evolves generated tasks and scenes; transfer-focused curricula and cross-embodiment evaluation help distinguish skill acquisition from benchmark overfitting.Source: §7.5 · pp. 48–49

7.6 General Computer Control

  • General computer control targets reusable procedures for learning unfamiliar applications, with scaffold gains from persistent memory, hierarchical planning, and curricula that assess trajectories across episodes.OSWorld, WindowsAgentArena, and OSWorld-MCP provide execution-based evaluation environments; the domain’s breadth across files, windows, dialogs, and applications makes adaptation to unseen software conventions central.Source: §7.6 · p. 49
  • Methods update agent parameters from collected interaction traces and learned evaluative signals, including reward-guided exploration, reflection tuning, and sustained online reinforcement learning.Other approaches use verifiable trajectories with step-wise reinforcement learning or synthetic action decisions seeded by a small demonstration set.Source: §7.6 · p. 49
  • General computer control must safeguard irreversible actions, verify outcomes amid operating-system, account, or user-context dependence, and foster transfer beyond surface UI patterns.The discussion calls for reliable state-based verification where possible, cautious policies for irreversible actions, and exploration strategies and curricula that encourage procedural abstractions.Source: §7.6 · pp. 49–50
  • Early meta-reinforcement-learning work let agents adapt through long-term statistics on the effects of their own modifications during lifelong trials (Schmidhuber, 1994; Schmidhuber et al., 1997; 1998; Schmidhuber, 1999; 2006a).Source: §7.6 · p. 50

8 Evaluation

Self-improvement evaluation must track capability over time and distinguish genuine transfer from artifacts of the update process. The section sets out reporting protocols and benchmarks for both isolated update mechanisms and realistic task domains.

8.1 Measuring Improvement

  • Evaluation treats self-improvement as a performance trajectory under cumulative resource limits, not a single terminal score, and separates capability gains from pipeline artifacts.The evaluation framework considers metric-based checks and judge-based scores, while benchmarking distinguishes mechanism-focused from domain-focused suites.Source: §8 · p. 50, §8.1 · p. 50

8.1.1 Metric-Based Measurement

  • Reports should show full learning curves under fixed budgets, held-out transfer, costs and human oversight, plus regressions, safety violations, and variance across seeds.Recommended items also include initial and post-budget performance, checkpoints, acceptance criteria, and early-stopping rules; hidden or temporally shifted tasks help test transfer beyond the improvement signal.Source: §8.1.1 · p. 51

8.1.2 Judge-Based Measurement

  • Parameterized judges enable evaluation of open-ended tasks without executable oracles, but agents may exploit judge biases, requiring transparent judge specifications and independent, calibrated final evaluation.Protocols should report the judge model, prompt, rubric, evidence, and judging budget separately from agent execution costs, and assess reliability through repeated runs, multiple judges, or verifiable checks.Source: §8.1.2 · pp. 51–52

8.2 Benchmarking Improvement

  • Benchmark choice depends jointly on whether updates target model parameters or scaffolding and whether evaluation is single-shot or interactive, shaping transfer, regression, and cost tests.Mechanism-focused benchmarks isolate how improvement is produced, whereas domain-focused benchmarks evaluate it under realistic task and feedback constraints.Source: §8.2 · p. 52

8.2.1 Mechanism Benchmarks

  • Mechanism benchmarks isolate parameter or scaffold updates across static and interactive interfaces, testing transfer under budgets and attributing gains through component ablations, fixed-environment replays, and regression checks.Model-level evaluation emphasizes provenance, dataset separation, and retention; scaffold-level evaluation isolates changed components and checks their specific failure surfaces.Source: §8.2 · p. 52, §8.2.1 · pp. 52–53

8.2.2 Domain Benchmarks

  • Domain suites apply fixed-budget trajectories, held-out transfer, regression, and safety reporting to realistic environments, including executable software tasks, dynamic web interfaces, and safety-constrained embodied and desktop control.Protocols specify feedback signals, permitted update channels, budgets, and held-out sets; domain-specific constraints include hidden software regression tests, web-layout generalization, and physical safety and recovery.Source: §8.2.2 · pp. 53–55

9 Discussion

The discussion frames self-improving agents as closed-loop systems whose evolution is driven by signal generation and stable updates, then identifies system-design safeguards and six future research directions. It calls for continuous improvement supported by robust feedback, safe self-modification, and ongoing evaluation.

9.1 Implications for System Design

  • Scaffold edits enable fast, reversible adaptation, whereas slower parameter updates transfer capabilities but obscure credit assignment and require renewed safety testing.With noisy feedback, validate scaffold changes through execution tests and defer distillation or finetuning until behavior is stable; parameter updates can lose rare recovery strategies and invalidate prior safety bounds (Qiu et al., 2025a; Lyu et al., 2026; Dou et al., 2024; Wu et al., 2024a).Source: §9.1 · p. 55
  • Treat critics as attack surfaces: decouple proposal from acceptance, constrain critic evolution to monotone changes, and gate self-modification through audited verification.Agents can exploit critics, making exploit-resistance a capability bottleneck (Zhan et al., 2024b; Zhang et al., 2025f); evolving critics should remain outside unconstrained agent control (Fang et al., 2025b; Li et al., 2025d). For full-scaffolding changes, verifier-gated checks should cover correctness, tool permissions, and robustness to random state perturbations; the system should run as untrusted code in a protected environment.Source: §9.1 · pp. 55–56

Theme A: Algorithmic Paradigms for Lifelong Adaptation

  • Future test-time adaptation should dynamically update retrieval, routing, and memory policies while preventing localized patches from eroding long-term global performance.The training-deployment divide can compromise real-world robustness; scalable adaptation is proposed to address it (Yang et al., 2026a).Source: Theme A: Algorithmic Paradigms for… · p. 56
  • Agents should seek informative experiences, using prediction errors or verifier disagreement as intrinsic signals to improve sample efficiency in sparse-feedback settings.Future intrinsic rewards should drive exploration safely and avoid degenerate self-deception (Schmidhuber, 1991c; Zhao et al., 2026b).Source: Theme A: Algorithmic Paradigms for… · p. 56
  • Research should transfer scaffold-level multi-step recovery strategies into smaller models’ weights and jointly optimize weights and scaffolds despite difficult credit assignment.Scaffold improvements such as iterative debugging and self-reflection can incur high context and latency costs; joint optimization must determine whether failures call for prompt, tool-wrapper, or gradient changes (Qiu et al., 2025c; Hebbar et al., 2026).Source: Theme A: Algorithmic Paradigms for… · p. 56

Theme B: Complexity, Constraints, and Open-World Robustness

  • Improvement should optimize resource use, limiting wasted exploration through dynamic context allocation, lightweight invariant checks, and penalties for unproductive iterations.The survey argues evaluation should prioritize improvement efficiency over static peak performance, since open-ended methods can exhaust compute and token budgets without advancing policy (Feng et al., 2026a).Source: Theme B: Complexity, Constraints, and… · p. 56
  • Co-evolving specialized agents could share reusable tests, patches, and tool wrappers through secure, version-controlled protocols to avoid single points of failure and cascading reward attacks.This direction addresses the inefficiency of single-agent exploration in vast search spaces (Chen et al., 2025d; Wang et al., 2026c).Source: Theme B: Complexity, Constraints, and… · pp. 56–57
  • Open-world evaluation should use non-stationary simulators to expose agents to drifting interfaces and adversarial inputs, while testing resistance to forgetting and environmental change.The survey also points toward learned neural runtimes that unify computation, memory, and I/O rather than relying on fixed external execution interfaces (Sunil et al., 2026a; Froger et al., 2026; Zhuge et al., 2026b; Rivard et al., 2026).Source: Theme B: Complexity, Constraints, and… · p. 57

10 Conclusion

The survey frames modern self-improvement through two complementary pathways: slower parametric foundation-model updates and faster non-parametric scaffold updates. It identifies open technical challenges and anticipates agents that can modify their own reasoning structures.

10 Conclusion

  • The survey synthesizes self-improving systems as slower parametric foundation-model improvement and faster non-parametric scaffolding improvement.The foundation-model loop draws on intrinsic generative demonstrations, intrinsic evaluative feedback, or extrinsic exploratory experience; the scaffolding loop updates prompts, memory, tools, and full scaffolds.Source: §10 · p. 57
  • Scalable and secure evaluation, alignment under continuous adaptation, and integration of fast and slow improvement loops remain pressing open challenges.Source: §10 · p. 57
  • Self-referential agents may modify their own reasoning structures, while the survey calls for systems that are measurable, attributable, safely bounded, and progressively autonomous.The survey aims to provide a common language and roadmap for researchers building such systems.Source: §10 · p. 57

Notation

The notation guide defines recurring symbols for learning signals, environment interaction, prompt and memory updates, and scaffold improvement, alongside indices and optimization operators.

Notation

  • rt and ct denote reward-like signals and natural-language critique; τt, s, a, r, and s′ denote experience, environment state, action, reward, and next state.The same notation defines revised target output t, consistency or confidence aggregator C(·), critique-and-revision operator R(·), induced policy πθt,Σt, and learned world model W(sk+1, rk | sk, ak).Source: Notation · p. 97
  • P, p, p∗, f(p), and pt+1 denote the prompt search space, candidate, best-scoring prompt, prompt score, and updated prompt.Refine(·), Critique(·), and Output(pt) denote prompt refinement, critique, and generated output.Source: Notation · p. 97
  • Pt and N denote a prompt population and its size, while Fit(p), pchild, Crossover(·), and Mutate(·) describe prompt fitness and evolutionary operations.The notation also defines g(pt) as textual guidance, ⊕ as textual update, and IMPROVEp as the prompt improvement operator.Source: Notation · p. 97
  • objectt and structuret denote stored memory objects and their structure or index; Create, Read, Update, and Delete denote memory write, retrieval, revision, and pruning operations.IMPROVEm denotes memory improvement; IMPROVET denotes tool improvement.Source: Notation · p. 97
  • IΣt denotes a scaffold improver, ⟨Σt⟩ its serializable encoding, exec(·) its execution, ˜Σt+1 a proposed update, and V(·) scaffold validation.These symbols describe scaffold improvement notation.Source: Notation · p. 97
  • t and t + 1 denote self-improvement iterations, k an inner step or candidate index, while arg max, arg min, and ⟨·⟩ denote optimization and serialization conventions.These conventions are used throughout or in the cited sections.Source: Notation · p. 97

Related papers

Loading 2607.13104v1…