Source-linked AI summary
Self-Improvements in Modern Agentic Systems: A Survey
Zhe Ren, Yimeng Chen, Dandan Guo, Guowei Rong, Tonghui Li, R. B. Xiong, Qingfeng Lan, Wenyi Wang, Li Nanbo, Yibo Yang, Mingchen Zhuge, Jürgen Schmidhuber
TL;DR
Self-improving agents face costly supervisory-signal acquisition and require controllable adaptation with minimal human input. This survey formalizes and organizes the field around foundation-model and scaffold improvement, synthesizes evaluation and applications, and concludes that bounded, verifiable loops offer a measurable path under human-designed objectives and safety protocols.
Problem
Costly manual preference labels and human evaluation limit supervisory-signal acquisition for self-improving agents, motivating internal evaluative processes.
Method
The survey presents a unified systems formulation and taxonomy distinguishing foundation-model improvement from scaffold improvement across prompts, memory, tools, and full scaffolds.
Results
The survey integrates its classification with applications and critically analyzes mechanism-level and end-to-end evaluation, including security challenges for reliable continuous self-improvement.
Takeaways & Limitations
Bounded, verifiable self-improvement loops within human-designed objectives and safety protocols provide a principled and measurable path toward reliable adaptation.
Takeaways & Limitations
Truly open-ended recursive self-improvement remains a grand challenge, with current systems operating as bounded, verifiable loops under human-designed objectives and safety protocols.
Abstract
from arXiv · showhide
Self-improving autonomous agents are moving from research prototypes to deployed systems. The primary goal is controllable evolution, or adaptation, from experience with minimal or even no human input. This survey frames modern self-improving agents as adaptive systems that convert experience into accumulated capability gains. We offer a system-level framework that represents a modern agent as a configuration coupling a foundation model with an operational scaffold of prompts, memory, tools, and control logic. Within this framework, self-improvement is formalized as a self-induced update operator that obtains and commits updates to model parameters or scaffold components. We organize prior work by update target and by the signals that drive change, then review applications and discuss evaluation, before closing with open problems and future directions. For convenience, we track technical updates on https://github.com/selfimproving-agent/awesome-Self-Improving-Agents.
1 Introduction
Modern agentic systems have shifted from task-specific designs to foundation-model-based agents whose autonomous operation depends on an operational scaffold. This survey organizes self-improvement into foundation model improvement and scaffold improvement, then connects the taxonomy to applications, evaluation, security, and open challenges.
- Historical Context and Evolution: Modern agentic systems have shifted from task-specific models or hand-engineered modules to foundation models, including LLMs and VLMs.Natural language serves as a shared interface for representation, reasoning, and interaction.
- A Unified Formalization and Systematic Taxonomy: An operational scaffold surrounds the foundation model with instruction schemes, memory systems, tool interfaces, and control logic for autonomous operation.The survey treats the neural core and scaffold together as the basis of FM-based self-improving agents.
- A Unified Formalization and Systematic Taxonomy: The survey categorizes self-improvement into foundation model improvement and scaffold improvement, covering prompt optimization, memory evolution, tool governance, and full-scaffold redesign.Foundation model improvement targets slower but more stable long-term consolidation by updating the underlying model.
- Empirical Landscape, Evaluation Paradigms, and Frontiers: The survey connects its classification with applications in software engineering, web navigation, and scientific discovery while analyzing mechanism-level benchmarking and end-to-end evaluation.It also highlights security factors and challenges for reliable, continuous self-improvement.
2 Historical Context and Theoretical Foundations
Self-improvement predates modern foundation models and evolved from error-driven parameter adaptation toward systems that modify representations, learning procedures, and their own operational components. Its theoretical foundations span feedback and homeostasis, formal self-reference, symbolic strategy manipulation, meta-learning, and provably justified self-modification.
- Early optimization: Least squares established systematic parameter adjustment to minimize prediction errors, forming a mathematical foundation for neural-network learning.Historical accounts connect Gauss’s astronomical observations of Ceres with adjusting predictor parameters to generalize to its new location.
- Feedback and adaptation: Cybernetics framed adaptation as feedback-driven behavior, while homeostasis emphasized internal-state adjustment for maintaining viability under perturbations.Turing’s child machine further proposed shaping internal configuration through training and external feedback, including reward and punishment.
- Formal self-reference: Gödel’s formal systems enabled self-reference by representing data and programs through universal coding and constructing statements about the computation of other formal systems.The same work also identified fundamental limits of computation-based artificial intelligence.
- Symbolic systems: Symbolic AI recast self-improvement as manipulating explicit knowledge and strategy representations, progressing from self-reproducing automata to heuristics treated as modifiable Lisp objects.AM searched over symbolic programs encoding mathematical ideas, while EURISKO generated, modified, and tested its own problem-solving heuristics.
- Meta-learning and self-modification: Connectionism reframed self-improvement as optimizing learning dynamics, including optimizers, inductive biases, and adaptation rules, with fast-weight and self-referential systems modifying their own weights.The Gödel Machine extended this trajectory by adopting self-modifications only when it could prove they would improve expected cumulative future reward.
3 Definitions
The survey defines a foundation-model-based agent through its intrinsic configuration of model parameters and operational scaffolding, while separating this architecture from ephemeral execution state. Self-improvement is a self-induced operator that durably updates the intrinsic configuration, encompassing parameter-space learning, execution-space self-modification, and reusable skills.
- Agent configuration: A foundation-model-based agent couples foundation-model parameters θ_t with an operational scaffold Σ_t that conditions, grounds, and connects the model to the external world.The scaffold includes prompts, memory, tools, and control logic such as routing, scheduling, and safety constraints.
- Agent configuration: Ephemeral execution state X_t shapes immediate action selection but is typically reset at task boundaries, unlike the intrinsic configuration (θ_t, Σ_t).Examples of X_t include key-value caches, intermediate plans, and short-term working memory.
- Self-improvement operator: Self-improvement is formalized as a self-induced operator U that executes the current policy to generate signals and then durably updates the intrinsic components θ or Σ.Signals can include interaction outcomes, critiques, verification results, or proposed edits; durable updates may nevertheless be undone.
- Self-reference modes: The two principal self-reference modes are foundation-model improvement in parameter space and scaffolding improvement in execution space.The former uses self-generated experience or auxiliary artifacts to shape future parameters, whereas the latter directly edits prompts, memory, tools, or control logic.
- Skills: A skill is a named, reusable instance of U that serializes an update to the agent’s configuration for retention and reuse.Skills may be repeatedly invoked routines or one-time installers whose persistent changes remain portable across agents and sessions.
- Skills: An object-level skill acts on task or world state by invoking E to execute a typically multi-step sub-task routine, while U writes that routine into its substrate during acquisition.The paper compares this scope to a temporally extended option in hierarchical reinforcement learning.
4 A Taxonomy of Existing Approaches
Section 4 introduces a taxonomy that organizes self-improvement approaches by update target and, for foundation model improvement, by the form of the self-induced signal. It distinguishes foundation model updates to parameters from scaffolding updates to prompts, memory, tools, and control logic while noting their differing adaptation properties.
- Taxonomy framework: The taxonomy provides a common reference frame for comparing heterogeneous self-improvement approaches in FM-based agents.It uses IMPROVE_target(·; S_t) to denote an abstract self-improvement procedure and S_t to denote its update signal.
- Foundation model improvement: Foundation model improvement updates underlying model parameters while leaving the agent-level scaffold unchanged.The agent’s execution generates learning signals consumed by a parameter-update procedure, with learned capabilities committed directly into θ_t.
- Foundation model improvement: Parametric updates amortize adaptation costs across future interactions but typically require longer timescales and higher computational overhead.The update is committed into the parameter history θ_1:t, supporting validation across successive updates.
- Foundation model improvement: Foundation model improvement is further classified by signal form, including intrinsic generative demonstrations and intrinsic evaluative feedback.Generative demonstrations provide explicit training instances, while evaluative feedback supplies rewards, preference pairs, or critiques for optimization and alignment.
- Scaffolding improvement: Scaffolding improvement updates the operational scaffold while keeping foundation model parameters fixed, enabling fast, reversible, task-specific adaptation without catastrophic-forgetting risks.Structural changes to Σ_t = (p_t, m_t, T_t, g_t) alter how the frozen model is conditioned, grounded, and constrained in later executions.
- Scaffolding improvement: Scaffolding improvement is decomposed by targeted component, including prompt optimization, memory evolution, and tool governance.These updates modify structured prompts or exemplars, experience storage and retrieval, and the agent’s action space, respectively.
5 Foundation Model Improvement
Foundation-model improvement updates the underlying model parameters to internalize new behaviors and reasoning patterns. The survey organizes these methods by whether learning signals come from generated demonstrations, internal evaluations, or experience gathered through external interaction.
- Overview: Foundation-model improvement targets the underlying parameter set θt, updating it to internalize new behaviors and reasoning patterns.The framework writes the transition as At+1 = IMPROVE(A1:t; St) with θt+1 = IMPROVEθ(θ1:t; St).
- Overview: Methods are categorized by dominant learning-signal source: intrinsic generative demonstrations, intrinsic evaluative feedback, and extrinsic exploratory experience.These categories are not mutually exclusive in deployed systems, where practical pipelines may mix signals within one improvement cycle.
- Intrinsic Generative Demonstrations: Intrinsic generative methods treat the foundation model as both cognitive learner and data synthesizer, producing demonstrations that become the training signal.The generated dataset is represented as St ≈ Dt, and a quality-control operator may filter or weight examples before optimization.
- Intrinsic Generative Demonstrations: Generation strategies expand seed instruction-output pairs through techniques such as instruction rewriting and complexity evolution, but self-consistency can reinforce confident errors.Curriculum-based generation can improve complex-task learning while remaining vulnerable to weaknesses in the selected curriculum.
- Intrinsic Evaluative Feedback: Intrinsic evaluative methods address the cost of manual preferences and human evaluation by reframing supervisory-signal acquisition as an internal evaluative process.In this regime, foundation models participate in producing the supervision used for capability alignment and refinement.
- Extrinsic Exploratory Experience: Extrinsic exploratory methods ground the learning signal in outcomes after acting, using trajectories collected by executing the current policy πθt,Σt.The formalism represents this experience as St ≈ τt.
6 Scaffolding Improvement
Scaffolding improvement evolves an agent’s operational scaffold while keeping foundation-model parameters frozen, using execution-derived signals to update prompts, memory, actions, or broader structure. Version histories support validation and rollback, while component interventions can compose into full-scaffolding methods with archive exploration and stronger acceptance tests.
- Scaffolding improvement decomposes an agent’s operational scaffold into instruction interpretation, contextualization through knowledge and memory, world action, and overall structural evolution.
- With θt+1 = θt, scaffold updates follow Σt+1 = IMPROVEΣ(Σ1:t; St) using execution-derived signals such as outcomes, critique, or errors.Maintaining Σ1:t enables validation and rollback against harmful modifications.
- Prompt-based improvement: Prompt-based improvement updates the input and instruction layer through pt+1 = IMPROVEp(p1:t; St), refining task communication, objectives, and constraints without changing model parameters.
- Memory-based improvement: Memory-based improvement develops an evolving internal cognitive resource, while scaffold interventions may jointly edit multiple components rather than operating mutually exclusively.Full-scaffolding methods add archive-based exploration and stronger acceptance tests to component-level edits.
6.1 Prompt
Prompt-based self-improvement treats prompts and context construction as scaffold-level targets updated through increasingly structured learning signals. Across four paradigms, feedback progresses from scalar scores to critiques, population selection, and textual gradients, making refinement more targeted and automated.
- 6.1 Prompt: Prompts act as core behavioral priors, while context optimization refines dynamic exemplar selection, retrieval assembly, and long-term playbooks.Both are scaffold-level updates but differ in whether they alter the agent’s core behavioral prior or its dynamically constructed context.
- 6.1 Prompt: Prompt refinement comprises scalar-feedback optimization, qualitative-feedback refinement, population-based evolution, and textual-gradient optimization.These paradigms are organized by the form and richness of their learning signal St.
- 6.1.1 Scalar-Feedback Optimization: Scalar-feedback methods search for prompts that maximize task accuracy or reward, using nondirectional scores as learning signals.APE uses an LLM to propose instruction candidates and select the best one, while RLPrompt directly optimizes downstream task-accuracy rewards.
- 6.1.2 Qualitative-Feedback Refinement: Qualitative-feedback refinement updates prompts from textual critiques, error analyses, or natural-language suggestions rather than single scores.The feedback is typically generated by an evaluator or the LLM from historical execution and can persist into prompting policies or instructional context.
- 6.1.3 Population-Based Evolution: Population-based evolution treats prompts as genes and uses fitness-based selection, semantic crossover, and mutation to explore candidate pools.EvoPrompt uses LLM-guided semantic operators, while Promptbreeder extends evolutionary search through self-referential mechanisms.
- 6.1.4 Textual Gradient Optimization: Textual-gradient optimization supplies structured directional feedback that diagnoses errors and prescribes revisions applied by an LLM optimizer.Its update rule is pt+1 = pt ⊕g(pt), with the textual gradient g(pt) serving as the learning signal.
- 6.1.4 Textual Gradient Optimization: As learning signals become richer—from scalar scores to critiques, population selection, and directional guidance—prompt updates become less heuristic and more targeted.The survey links this progression to increasingly automated and sample-efficient refinement.
6.2 Memory
This section frames memory as dynamic cognitive scaffolding whose objects, structure, and processing mechanisms adapt through signal-driven updates. The resulting self-improvement memory loop unifies writing, organization, retrieval, and pruning to support scalable long-horizon behavior.
- Framework: Memory-based improvement decomposes into Memory Objects, Memory Structure, and Memory Processing, with the scaffold state defined as mt := (objectt, structuret).This decomposition distinguishes what is stored, how it is organized, and how it is created, retrieved, updated, or forgotten.
- Framework: Signal-driven Write, Read, Update, and Delete operations determine what to consolidate into objectt+1 and how to reorganize structuret+1.The section focuses on non-parametric memory rather than knowledge internalized in foundation-model weights.
- Memory Objects: Explicit memory objects improve interpretability and controllability, whereas implicit objects provide compact, fast associative access but complicate debugging, correction, and safety auditing.Explicit objects require curation to avoid context pressure, while implicit representations can drift when repeatedly updated.
- Memory Objects: Self-improving agents store processed interaction trails, curated raw artifacts, or externally grounded knowledge instead of relying solely on exhaustive interaction histories.Processed trails support reusable strategies, curated artifacts preserve exact details, and external references improve verifiability through utility-based updating and pruning.
- Memory Processing: The Self-Improvement Memory Loop unifies memory writing, structural organization, adaptive retrieval, and feedback-based pruning into a continuous lifecycle for scalable self-improvement.This framework elevates memory from a static cache to an engine supporting adaptive behavior.
6.3 Tool
Tool Governance Metacognition reframes self-improving agents’ tool use from static, predefined execution into dynamic management of routing, refinement, and autonomous creation. Together, these mechanisms form a dynamic skill repository with sustainably expandable capabilities.
- Tool Governance Metacognition: Tool Governance Metacognition replaces static, manually curated toolkits with dynamic, self-directed management of operational tools.It is instantiated through Dynamic Tool Routing, Iterative Tool Refinement, and autonomous tool creation.
- Dynamic Tool Routing: Dynamic tool routing selects, sequences, and coordinates tools while balancing coverage, reliability, and decision cost.The section distinguishes retrieval- and graph-based, policy-learning, and proactive and interactive routing paradigms.
- Iterative Tool Refinement: Iterative tool refinement turns fragile programs into dependable skills while gatekeeping entries into the structural skill repository.It also prevents unreliable tools from corrupting future behavior through repeated retrieval and compounding errors.
- Iterative Tool Refinement: A canonical refinement loop alternates generation, execution, and revision, using error traces and environment feedback until a verifier confirms success.VOYAGER establishes this baseline by feeding execution feedback into subsequent code revisions.
- Autonomous Tool Creation: Autonomous tool creation expands capability boundaries by synthesizing reusable executable functions when existing tools are insufficient.Created tools must be validated, documented, and integrated without destabilizing routing policies; synthesis may be triggered by immediate needs or open-ended curiosity.
- Tool Governance Metacognition: The resulting architecture transforms a static toolkit into a dynamic skill repository with sustainably expandable capabilities.Its growth-oriented design combines dynamic routing, iterative refinement, and autonomous creation.
6.4 Full Scaffolding
Full scaffolding self-improvement treats an agent’s entire operational logic and codebase as mutable, enabling structural reorganization and self-referential evolution. Current systems realize this recursive-self-improvement paradigm through bounded, verifiable loops governed by objectives and safety protocols.
- Architectural scope: Full scaffolding improvement makes the agent’s entire operational logic and codebase mutable rather than tuning isolated prompts, tools, or memory components.This represents the deepest level of architectural intervention.
- Self-referential improvement: The improvement procedure runs within the current scaffolding, allowing the improver and the agent it improves to evolve together.Learning signals can come from execution traces and evaluations such as task outcomes, unit tests, self-critique, and cost signals.
- Programmatic updates: A full-scaffolding update produces a candidate program from the agent’s own code, often as a patch that is accepted only after verification.Verifiers may include unit tests, regression suites, and safety checks.
- Representative systems: Representative systems include AlphaEvolve, ShinkaEvolve, ADAS, EvoFlow, Self-Taught Optimizer, Gödel Agent, and Darwin Gödel, spanning evolutionary search, workflow evolution, and recursive self-modification.These systems target algorithm discovery, agent-design optimization, cost–performance trade-offs, or open-ended program discovery.
- Bounded recursion: Full scaffolding is closest to recursive self-improvement, but current systems operationalize it as bounded, verifiable loops within human-designed objectives and safety protocols.Truly open-ended recursive self-improvement remains a grand challenge.
7 Applications
Self-improving agents appear across software engineering, web automation, games, scientific discovery, embodied AI, and general computer control, with each domain’s environment shaping its improvement targets and bottlenecks. The section emphasizes that feedback quality, environment fidelity, and safety determine how reliably agents can evolve.
- Cross-domain overview: Application domains differ in their substrates, from tests and continuous integration to browsers, simulators, executable workflows, robotic environments, and virtualized desktops.These environments’ fidelity, scalability, and cost shape each domain’s dominant bottlenecks, improvement targets, and iteration modes.
- Software engineering: Software engineering supports both scaffold and foundation-model improvement because compilers, tests, linters, and continuous integration provide dense, automatable feedback.Scaffolding can be modified directly, while executable test outcomes can provide rewards or supervision for updating model parameters θ.
- Web navigation and automation: Web automation requires persistent improvement because interfaces change, user intents map to varied layouts, and failures may surface only late in long-horizon trajectories.Methods use grounding, cross-session memory, reflections, finetuning, and exploration-feedback loops to improve behavior across tasks.
- Games: Games offer scalable, simulated feedback for self-improvement but can produce brittle competence through simulator exploitation, narrow opponent distributions, or self-play.Such competence may fail when the population shifts or the rules change.
- Scientific discovery: Scientific discovery must address evaluation, heterogeneous evidence management, and stable scaffold evolution because novelty, correctness, and reproducibility are difficult to verify autonomously.Apparent improvements may instead exploit weak proxies, while useful changes must persist across tools, data formats, and rapidly evolving literature.
- General computer control: General computer control makes safety central because agents may delete files, enter passwords, or initiate financial transactions.Improvement processes therefore require safeguards and conservative recovery strategies.
8 Evaluation
Evaluation of self-improving agents should track capability evolution over iterations under fixed resource budgets, rather than relying on static or final scores. Robust protocols must test transfer beyond the improvement signal while reporting efficiency, stability, regressions, and safety over time.
- Evaluation objective: Evaluation tracks a performance trajectory across iterations t ∈ {1, . . . , T} under a cumulative resource budget b_t ≤ B_max.The agent configuration is evaluated on tasks drawn from a held-out distribution D_eval.
- Reporting requirements: Reports should provide full learning curves under fixed budgets, including checkpoints, acceptance criteria, early-stopping rules, and initial and post-improvement performance.Reporting only a final peak score can obscure plateaus, regressions, or overfitting from unbounded iterations.
- Transfer evaluation: Held-out evaluation distributions, private task sets, or newly generated tasks test whether gains transfer beyond the optimization feedback rather than reflecting memorization.Evaluation should use D_eval that does not overlap with optimization data.
- Efficiency and supervision: Protocols should disclose compute, API tokens, wall-clock time, tool invocations, and human input to compare the real-world efficiency of improvement pipelines.Cumulative cost b_t determines practical evaluation and should be reported transparently alongside task success.
- Stability and safety: Evaluation should monitor regression rates, tail-risk indicators, and safety-policy violations across iterations, not only mean success rates.Iterative self-modification can produce goal drift, reward hacking, and compounding errors in memory or tool use.
- Benchmark design: Benchmarks should vary by update channel—foundation-model parameters θ or scaffolding Σ—and by interface, from single-shot outputs to interactive tool-, memory-, and environment-based behavior.Mechanism-focused suites isolate channels such as memory expansion or prompt refinement, while domain-focused suites test them in realistic interactive environments.
9 Discussion
The discussion frames self-improving agents as closed-loop systems whose evolution mechanisms—not static configurations—must be evaluated, governed, and secured. It highlights asymmetric scaffold and parameter updates, critic governance, layered safety, and six research directions for reliable continuous improvement.
- Evolution as the Evaluation Target: Self-improving agents are closed-loop dynamical systems, so evaluation should focus on the mechanisms that drive agent evolution rather than static agents.The formulation represents an agent as A_t = (θ_t, Σ_t) that generates signals and applies stable update rules.
- From Fast Exploration to Slow Consolidation: Scaffold edits enable fast, reversible adaptation, whereas parameter updates are slower, transfer capabilities across domains, and make regression credit assignment difficult.Prompt edits, memory writes, and tool adjustments have minimal computational overhead; weight modifications can obscure the source of failures.
- From Fast Exploration to Slow Consolidation: Noisy feedback favors validating scaffold updates through execution tests before consolidating stable behavior into parameters, because distillation lossy-compresses complex trajectories.The discussion recommends deferring distillation or finetuning until new behavior has proven stable.
- The Critic as Governed Infrastructure: An embedded critic is governed infrastructure and an attack surface: an agent’s capability ceiling is often limited by the critic’s exploit-resistance.Optimizing against the critic creates incentives to discover shortcuts.
- Safety through Layered Gating: Safe self-improvement requires treating agents as untrusted code in protected runtimes, using layered permission gating because persistent updates can turn transient exploits into architectural vulnerabilities.The risks are especially severe when agents can modify their own source code or operational logic.
- Future Research Directions: Future research targets reliable update cycles through test-time adaptation, active exploration, parametric distillation, resource-aware dynamics, cooperative co-evolution, and robustness to distribution drift.These directions address dynamic execution, sparse feedback, scaffold-to-parameter transfer, computational limits, shared artifacts, and changing real-world environments.
10 Conclusion
The survey frames modern self-improving systems through a unified systems lens, distinguishing foundation-model improvement from other improvement pathways. It identifies scalable and secure evaluation, alignment under continuous adaptation, and integration of fast and slow loops as pressing open challenges.
- Conclusion: The survey presents self-improvement as an enduring AI vision expanded in scale, versatility, and scope by large foundation models.It synthesizes this paradigm shift through a unified systems lens.
- Conclusion: It distinguishes foundation-model improvement as a parametric, slower loop from a second improvement pathway.The passage characterizes the first loop as driven by intrinsic generative demonstrations.
- Conclusion: Scalable and secure evaluation, alignment under continuous adaptation, and integration of fast and slow improvement loops remain pressing technical challenges.The survey describes these challenges as part of a decisive phase toward truly self-improving agents.
Notation
The notation guide defines recurring symbols for rewards, preferences, critiques, trajectories, environment dynamics, prompt optimization, memory improvement, and iteration indices used throughout the survey.
- Agent-environment interaction: Agent-environment notation represents interaction trajectories as τt, states as s and s′, actions as a, rewards as r, and policies as πθt,Σt.The learned world model is denoted W(sk+1, rk | sk, ak).
- Prompt improvement: Prompt improvement uses P for the search space, p for a candidate, p∗ for the best-scoring prompt, f (p) for its scalar score, and pt+1 for the updated prompt.Related operators include Refine(·), Critique(·), and IMPROVEp.
- Prompt and memory populations: Prompt-population notation includes Pt for the population, N for population size, Fit(p) for fitness, pchild for offspring, and Crossover(·) and Mutate(·) for variation operators.The i-th prompt in Pt is also represented, alongside textual gradients g(pt), ⊕ updates, and IMPROVEm for memory improvement.