Self-Improvement of Large Language Models: A Technical Overview and Future Outlook

1 Introduction

• High-quality expert-annotated data is costly and difficult to scale, while human feedback may become insufficiently informative as models approach or exceed human-level capability in some domains.These constraints motivate models generating or curating data, evaluating outputs, and refining their policies.Source: §1 · p. 2
• Self-improvement is an autonomous, continuous learning loop in which earlier outputs or experiences are reused to generate stronger supervision signals for later updates.The process may use auxiliary modules, but the learning loop is automated once deployed.Source: §1 · pp. 2–3
• The proposed lifecycle has the model acquire data, select useful training examples, optimize parameters, and refine inference-time outputs, with autonomous evaluation connecting the stages.Inference refinement changes outputs without changing underlying parameters.Source: §1 · p. 3
• Autonomous evaluation continuously assesses performance and guides future development because static benchmarks become outdated and human-driven evaluation does not scale with system growth.The paper presents timely, adaptive feedback as supporting sustained long-term improvement.Source: §1 · p. 4
• Unlike work focused on localized training or inference mechanisms or peripheral agent components, the paper frames LLM self-improvement as an end-to-end closed loop.It integrates stages of model development into a system-level perspective.Source: §1 · p. 4
• Model self-improvement targets internal capabilities through data, optimization, and refinement, whereas self-evolving agents adapt behavior through tools, environments, memory, and feedback.The paper connects agent-based improvement most naturally to inference time.Source: §1 · p. 4

2.1 Overview

• The acquisition taxonomy spans discovery in existing sources, exploration through external tools, and internal synthesis, progressively increasing model autonomy and reducing reliance on existing data.Models can continuously acquire data at scale, and their generated signals can be competitive with human-curated content or surpass human benchmarks on some specialized reasoning and high-complexity tasks.Source: §2.1 · pp. 4, 6, 7

2.2 Static Curation

• Static curation has traditionally relied on heuristic pipelines, while model-guided filtering and crawling offer ways to tailor collected corpora to a model’s needs.Model-guided curation of code, scientific text, and books remains nascent; current static pipelines also face provenance and bias risks, integration barriers, and slow, costly feedback from downstream pretraining.Source: §2.2 · pp. 7–9
• Model-guided web curation progresses from scoring crawled documents to choosing crawl targets, with reported gains in data efficiency and benchmark performance.FineWeb-Edu (Penedo et al., 2024) filtered 1.3 trillion tokens to 280 billion and improved MMLU by 12% and ARC by 24% over unfiltered baselines; DCLM (Li et al., 2024b) enabled a 7B model to reach 64% 5-shot accuracy on MMLU with 40% less compute than the previous state of the art. Craw4LLM (Yu et al., 2025e) achieved over 95% of oracle pretraining performance while crawling only 21% of URLs.Source: §2.2 · p. 7

2.3 Environment Interaction

• Environment interaction turns actions into training trajectories, enabling adaptive collection of observations, tool outputs, and task feedback rather than passive use of fixed corpora.Web interactions capture navigation and tool-use traces; code execution supplies runtime states and test outcomes; game environments yield strategic and social interaction data.Source: §2.3 · pp. 9–11

2.4 Synthetic Generation

• Synthetic generation creates new training examples without environmental feedback, shifting the central challenge from data availability to the quality, diversity, and informativeness of generated signals.The section distinguishes prompt-based generation, transformation of existing corpora, and interaction among model instances through self-play, debate, or collaboration.Source: §2.4 · p. 11

2.4.1 Prompt-Based Generation

• Prompt-based generation includes direct prompting from task specifications and seed expansion, which amplifies examples into larger corpora for model training.Phi-1 (Gunasekar et al., 2023) reached 50.6% accuracy on HumanEval with a 1.3 billion parameter model, matching or exceeding models three times larger; Self-Instruct (Wang et al., 2023c) produced over 52,000 instructions and showed a 33% absolute improvement over vanilla GPT-3 on SuperNaturalInstructions. AttrPrompt (Yu et al., 2023) reported high diversity at 5% of ChatGPT’s querying cost, while MIND (Akter et al., 2025a) improved GSM8K by 13.4% and MMLU-STEM by 4.3%.Source: §2.4.1 · pp. 11–13

2.4.2 Transformation-Based Generation

• Transformation-based generation rewrites, reformats, or extracts examples from existing corpora to improve their quality, structure, or task alignment.WRAP (Maini et al., 2024) reports approximately three-fold faster convergence with improved zero-shot accuracy; Instruction Pre-Training synthesizes 200 million instruction-response pairs, while SBP (Yang et al., 2025c) generates up to one trillion tokens and recovers approximately 60% of gains otherwise requiring twenty times more unique natural data.Source: §2.4.2 · p. 13

2.4.3 Interaction-Based Generation

• Interaction-based generation uses self-play, debate, or collaboration among model roles to create training data without external environmental feedback.Self-chat and self-play produce dialogues or policy data (Shah et al., 2018; Xu et al., 2023; Chen et al., 2024j), while LSP (Kuba et al., 2025) and R-Zero (Huang et al., 2026a) use self-generated challenges and solutions for improvement without external training data. Multi-model cohorts, math-problem collaboration, and EigenData extend this approach through diverse dialogue, continual learning, and agents that generate and audit function-calling trajectories (Ueda et al., 2025; Wan et al., 2025; Chen et al., 2026b); SPIN and LSP also connect data generation to policy optimization (Chen et al., 2024j; Kuba et al., 2025).Source: §2.4.3 · pp. 13–14

2.5 Discussion

• The three pathways are complementary rather than competing, and together form a layered system for autonomous data acquisition.Their trade-offs and shifting roles across the training lifecycle determine how they can be combined.Source: §2.5 · p. 14

2.5.1 Trade-offs Across Pathways

• The three acquisition pathways trade off scale, specificity, cost, quality, and diversity, so pipeline designers must balance their complementary strengths.Static curation can draw on hundreds of trillions of tokens but offers limited targeting; synthetic generation is controllable but risks redundancy and narrowing, while environment interaction is specific but costly per trajectory. Strong teacher models can produce high-quality synthetic data at substantial inference cost, whereas cheaper open models may yield lower fidelity; interaction trajectories are costly but carry dense, verifiable signals. Maintaining synthetic diversity may require entity grounding, attributed prompts, and real-data infusions, while static corpora require filtering noise, toxicity, and duplicates (Weber et al., 2024; Chen et al., 2024b; Yang et al., 2025b; Ben Allal et al., 2024; Penedo et al., 2024; Yu et al., 2025e; Dohmatob et al., 2024; Yu et al., 2023).Source: §2.5.1 · pp. 14–15
• Carefully engineered synthetic data can partly replace web crawls, but environment interaction provides causal traces that static text or generation cannot replicate.BeyondWeb (DatologyAI et al., 2025) and the Phi family (Gunasekar et al., 2023; Abdin et al., 2024b) show partial substitution with fewer tokens; Kang et al. (2025a) report book-style synthetic data can collapse at small budgets, while roughly 30% rephrased data mixed with natural web text accelerates convergence five to tenfold without degradation. Naive recursive self-training fails to improve perplexity or downstream performance across model sizes, and execution traces, browsing trajectories, and tool-use logs depend on genuine external interactions (Ding et al., 2023; FAIR CodeGen team et al., 2025; Nakano et al., 2022; Trabucco et al., 2025; Schick et al., 2023); strong pipelines combine all three pathways.Source: §2.5.1 · p. 15

2.5.2 Shifting Roles Across the Training Lifecycle

• The optimal acquisition mix shifts over training: broad static data supports early pretraining, synthesis strengthens targeted capabilities, and interaction supplies grounded trajectories later.Front-loading reasoning-intensive data yields a 19% average downstream-benchmark gain, and pretraining favors diverse reasoning patterns while supervised fine-tuning is more sensitive to data quality (Akter et al., 2025b). During continual pretraining and post-training, environment interaction supports tool use, browsing, code execution, and strategic reasoning, while instruction bootstrapping and self-play refine alignment and preferences (Su et al., 2025b;a; Gehring et al., 2025; Wang et al., 2023c; Chen et al., 2024j).Source: §2.5.2 · p. 15

2.5.3 Toward Autonomous Data Acquisition

• Autonomous acquisition can direct curation, explore environments, and synthesize targeted examples, with systems increasingly composing all three pathways.LLM-guided crawling can prioritize sources based on knowledge gaps (Yu et al., 2025e); browsing, code execution, and tool invocation produce grounded data, while R-Zero and Language Self-Play demonstrate improvement without external data (Trabucco et al., 2025; Gandhi & Neubig, 2025; Gehring et al., 2025; Jiang et al., 2025d; Schick et al., 2023; Huang et al., 2026a; Kuba et al., 2025). ALAS combines web retrieval, question-answer synthesis, self-training, and curriculum revision based on evaluation.Source: §2.5.3 · pp. 15–16

2.5.4 Initiation of the Self-Improvement Loop

• Acquisition initiates each self-improvement iteration: the model assesses weaknesses, chooses a pathway, gathers data, and supplies it to downstream stages.Because data scale, diversity, difficulty, and noise propagate through the loop, autonomous acquisition is a persistent process that bounds what later stages can achieve.Source: §2.5.4 · p. 16

3.2 Metric-Guided Scoring

• Metric-guided scoring applies a fixed decision rule to model-derived or evaluator signals, either once offline or repeatedly during training to refresh selected data.It can yield filtered subsets, sample weights, or dynamically selected mini-batches; iterative re-scoring strengthens continuity, while the rule itself remains unlearned.Source: §3.2 · p. 17, §3.2.2 · pp. 18–19

3.2.1 One-Shot Scoring

• One-shot scoring fixes selection before target-model training, offering automation, low overhead, and stability but limited continuity and adaptivity as model needs evolve.Utility estimates may use likelihood, training dynamics, gradients, difficulty, uncertainty, similarity, or external evaluators.Source: §3.2.1 · p. 18

3.2.2 Iterative Re-Scoring

• Iterative re-scoring updates model-derived scores and reapplies a fixed rule during training, but frequent or costly scoring adds computational overhead.Its broader limitation is that changing model states provide adaptivity without a learned policy that optimizes long-horizon objectives or principled exploration–exploitation trade-offs.Source: §3.2.2 · pp. 18–20

3.3 Adaptive Selection

• Adaptive selection makes the selection policy itself learnable and optimizes it alongside the model, so selection can evolve with changing capabilities, weaknesses, and objectives.This strengthens both automation, by using feedback without ongoing human intervention, and continuity, by updating the policy over time.Source: §3.3 · p. 20

3.3.1 Selection Unit as a Design Choice

• Adaptive selectors choose a fixed selection unit before training: instance-level units enable finer control, while group-level units allocate budget across broader collections.The unit remains constant throughout selection and shapes how precisely the selector can adjust the training distribution.Source: §3.3.1 · p. 21

3.3.2 The Selection Loop

• A learnable selector repeatedly generates utility signals relative to the current model state, then updates sampling, filtering, or allocation to reshape later training and signals.Updates may fit auxiliary predictors, optimize policies with reinforcement learning or bandits, learn weights, or differentiate through training against an outer objective.Source: §3.3.2 · pp. 21–22

3.3.3 Coupling between Selector and Model

• Coupled selectors use the main model’s changing state to co-adapt during training, whereas decoupled selectors are prepared beforehand, reducing complexity but forfeiting adaptation to shifting needs.The section also distinguishes curriculum-driven from objective-driven selection as two paradigms, without elaborating their criteria in this part.Source: §3.3.3 · p. 23, §3.3.4 · p. 24
• All surveyed adaptive-selection methods use a separate selector and main model; none has the main model curate its own training data, leaving self-directed selection unexplored.The authors identify a single model acting as both selector and learner as the strongest form of autonomy in self-improving systems.Source: §3.3.3 · p. 23

3.3.4 Curriculum-Driven vs. Objective-Driven Selection

• Curriculum-driven selection matches sample difficulty or information value to current competence, while objective-driven selection prioritizes estimated contribution to a predefined target.Curricula can address exploration bottlenecks with verified trajectories or stepping-stone problems; in multimodal reasoning, easy-example concentration can leave difficult examples without learnable supervision. Targets may include task performance, reasoning accuracy, preference alignment, or safety.Source: §3.3.4 · p. 24

3.4.1 Trade-offs Across Regimes

• Selection regimes trade simplicity and throughput against responsiveness: static filtering is fixed, metric-guided scoring may become stale, and adaptive selection adds feedback sensitivity and overhead.Iterative re-scoring can track changing utility during training, but at additional computational and robustness cost; adaptive policies also face noisy or drifting feedback and delayed credit assignment.Source: §3.4.1 · p. 24

3.4.2 Pre-training vs. Post-training

• Self-improving data selection should coordinate stable, inexpensive mechanisms at scale with adaptive methods near optimization, where richer feedback is available.A pipeline can apply hard safety, licensing, provenance, toxicity, and deduplication constraints, shape coarse domain or skill mixtures, then use policy-learned selection to allocate post-training budget to learnable, high-impact samples aligned with target behaviors.Source: §3.4.2 · p. 25

3.4.3 Interaction with Data Acquisition

• Acquisition and selection jointly establish each iteration’s data foundation: acquisition broadens experience sources, while selection verifies quality and adapts what remains informative as the model improves.Web-scale corpora require balancing quality with throughput, and self-synthesized solutions may need verification before training.Source: §3.4.3 · p. 26

4.1 Overview

• Direct optimization updates the model once offline on a fixed corpus assembled by the acquisition and selection stages.The training data is fixed before optimization begins.Source: §4.1 · p. 26
• Self-Generated Optimization (SGO) loops through generation, reward, and optimization so the model learns from its own experience.Its training corpus is not fixed in advance.Source: §4.1 · p. 26
• Self-Evolving Optimization improves the update algorithm, architecture, and agent-level scaffolding, extending improvement beyond model parameters.The unit of improvement becomes the whole agentic system.Source: §4.1 · p. 26

4.2 Direct Optimization

• Direct optimization performs one offline update on a fixed dataset, with no resampling or rescoring during training.In self-improvement, the model produces and curates the training targets upstream, then internalizes them in a single update.Source: §4.2 · p. 26, Figure 6 · p. 27
• Self-Instruct (Wang et al., 2023c), WizardLM (Xu et al., 2024a), and CodecLM (Wang et al., 2024i) use one-shot supervised fine-tuning on model-generated instruction data.Their data construction varies, including seed-based generation and increasing instruction complexity.Source: §4.2 · p. 26
• Magpie (Xu et al., 2025d) reports that plain SFT on its self-generated corpus can match models trained with larger human-curated pipelines; SAO (Yin et al., 2026) applies the fixed-corpus recipe to preference alignment.LongMagpie (Gao et al., 2025a) extends Magpie to long-context instructions.Source: §4.2 · p. 27
• Learn-by-Interact, AgentTuning (Zeng et al., 2024), and FireAct (Chen et al., 2023a) use interaction or teacher-distilled trajectories for agentic capabilities.The update remains a stable, inexpensive single pass; the method’s sophistication lies in data production, making this approach suitable when high-quality experience can be assembled ahead of time.Source: §4.2 · p. 27

4.3 Self-Generated Optimization

• SGO repeatedly generates training data, scores it, updates the model, and uses the updated model to drive the next generation round.Generation and reward are internal to optimization and are re-invoked on the continually updated policy.Source: §4.3 · p. 27
• Self-Instruct (Wang et al., 2023c) remains direct optimization because it generates and filters candidates once before a single supervised update.Its generation and selection occur upstream of the optimization loop.Source: §4.3 · p. 27
• Because SGO reuses the updated policy for generation, its training distribution is policy-induced and non-stationary.This distinguishes it from a corpus fixed before optimization begins.Source: §4.3 · p. 27

Algorithm 1 The Self-Generated Optimization (SGO) Loop

• SGO cycles through generation, reward, and optimization because self-generated outputs need assessment before they can provide a training signal.The reward format determines the signal used by the subsequent parameter update.Source: Algorithm 1 The Self-Generated… · p. 28

4.3.1 Generation

• Generation strategies are self-exploratory, refined, or interactive, spanning current-policy sampling, iterative revision, and collaboration, adversarial interaction, or tools.Tool-augmented generation uses feedback during generation to shape the final trajectory.Source: §4.3.1 · pp. 29–30

4.3.2 Reward

• Reward signals come from heuristics, model-based judgments, or verification against ground truth or executable environments.Verification includes answer matching and, for tasks such as code generation, compiler or test feedback.Source: §4.3.2 · pp. 30, 33

4.3.3 Optimization

• Optimization matches the reward format: filtered samples support SFT, preference rankings support DPO, and scalar rewards support online RL such as PPO or GRPO.Some methods use specialized objectives for self-play or combine an SFT warm-up with preference or RL optimization.Source: §4.3.3 · p. 33

4.3.4 Representative Instances of SGO

• Representative SGO patterns include filtering verified or consistent generations for SFT, self-verification and refinement with reward-based updates, and self-play between adversarial or collaborative roles.Self-play uses interaction to provide difficult challenges or consensus signals.Source: §4.3.4 · p. 34

4.4 Self-Evolving Optimization

• Self-evolving optimization changes the improvement procedure itself, including update rules, learning algorithms, or surrounding agentic scaffolding.Examples range from models inventing optimization algorithms to systems that recursively inspect and modify their own logic.Source: §4.4 · pp. 34–35
• Agent-level optimization can co-evolve roles, accumulate reusable skills, distill interaction trajectories into strategic principles, or train agents across interactive environments.These approaches shift improvement beyond the model in isolation.Source: §4.4 · p. 35
• The broader trend treats prompts, memory, tools, and workflows as parts of the improving system, alongside test-time training that temporarily adapts parameters during inference.Test-time training uses self-generated feedback signals.Source: §4.4 · p. 35

4.5.1 Theoretical Analysis

• SGO theory examines the source of self-generated improvement, its convergence and capability ceiling, and the conditions under which the loop collapses.Model optimization is the lifecycle stage that updates parameters, making convergence and capability limits well-posed questions.Source: §4.5.1 · p. 35, §4.5 · p. 35
• Self-generated optimization can converge under stated conditions, but its improvement ceiling is constrained by verifier fidelity and can plateau below oracle-supervised performance.The cited analysis also finds that sufficient training compute does not eliminate the generalization gap to oracle supervision.Source: §4.5.1 · p. 36
• Majority-voting self-reward can improve performance and supervision quality initially, yet prolonged self-training can lead to reward hacking and sudden collapse.The cited work identifies feedback design as a central bottleneck for sustainability; naive recursive pretraining did not improve non-toy LLMs without additional structure from human-designed prompting and filtering.Source: §4.5.1 · p. 36

4.5.2 Reward Formulation

• Implicit rewards shape training through data selection or filtering, whereas explicit rewards supply accessible signals used in RL or preference-based optimization.Explicit reward forms include binary, scalar, ordinal, and probability-based signals.Source: §4.5.2 · p. 36

4.5.3 Degree of Automation

• Automation varies with reliance on externally provided data and auxiliary models; data dependence ranges from annotated prompts and answers to prompts alone or no supplied dataset.Most surveyed work does not rely on additional external models, with a limited subset explicitly depending on them.Source: §4.5.3 · pp. 37–38

4.5.4 Trends across the Three Paradigms

• Direct optimization, SGO, and self-evolving optimization form a progression toward greater autonomy, but rising autonomy comes with greater instability and weaker guarantees.The paper characterizes direct optimization as most reliable when good data are available, SGO as dominant when models can generate and verify experience, and self-evolving optimization as least mature.Source: §4.5.4 · p. 39

4.5.5 Interplay Between Optimization and Data

• Acquisition and selection bound what optimization can learn by determining which experiences are available and trained on, while each optimization round changes which experiences remain useful.Too-easy or too-hard samples can yield uninformative rewards, and mislabeled samples can be reinforced into parameters.Source: §4.5.5 · p. 39

5.2 Decoding Strategies

• Decoding improves outputs by selecting better samples and makes repeated generation more efficient, supporting inference quality, training-data creation, and scalable self-improvement.These strategies improve a fixed model without parameter updates and can feed higher-quality outputs back into the optimization stage.Source: §5.2 · p. 41

5.2.1 Sampling-Based

• Sampling methods aggregate or synthesize multiple candidates for more reliable answers and reusable training data, but systemic bias, verifier limits, and synthesis costs constrain them.Consensus can reinforce consistently incorrect answers; candidate selection can plateau without reliable verifiers, while synthesis is costly as model and candidate counts grow.Source: §5.2.1 · pp. 41–42

5.2.2 Tree Search

• Tree search explores and compares alternative reasoning paths, using beam or Monte Carlo methods and evaluation signals to guide selection beyond linear autoregressive generation.Variants use self-evaluation, verifiers, reward models, or learned value functions to guide or prune candidates.Source: §5.2.2 · p. 42

5.2.3 Logit and Probability Adjustments

• Reward, reference, and constraint guidance adjust token probabilities at inference for fine-grained control without parameter updates, trading decoding computation and often requiring dedicated reward models.Reward guidance can recover a significant amount of the performance gap versus full fine-tuning, but typically requires training a dedicated reward model.Source: §5.2.3 · pp. 42–43

5.2.4 Efficiency-Oriented Methods

• Parallel and speculative decoding accelerate generation, enabling faster performance feedback and response rollouts that can speed recursive self-improvement.Speculative decoding drafts candidate tokens for parallel verification, while parallel methods generate multiple tokens or response segments together.Source: §5.2.4 · pp. 43–44

5.3 Reasoning-Based Improvement

• Reasoning-based methods improve outputs by structuring multi-step thinking, incorporating feedback, decomposing tasks into subgoals, or distributing reasoning across agents without explicit policy updates.They create an inference-time optimization loop in which reasoning trajectories are iteratively refined through feedback and experience.Source: §5.3 · p. 44

5.3.1 Feedback-Based Reasoning

• Feedback-based reasoning revises outputs using internal critiques, external critics, or execution and verification signals, but self-assessment, critic consistency, and task verifiability limit reliability.Internal methods can selectively reflect on uncertain steps; external signals include actionable critiques and code execution or formal-verifier feedback.Source: §5.3.1 · pp. 44–46

5.3.2 Planning-Based Reasoning

• Planning-based reasoning decomposes complex tasks into subgoals and can formalize constrained plans in PDDL, but requires anticipating necessary steps and plans may be incorrect or unexecutable.Structured prompting elicits a global blueprint before final reasoning and generation.Source: §5.3.2 · pp. 46–47
• ReAct (Yao et al., 2023c) and AdaPlanner (Sun et al., 2023) use environmental feedback to revise plans, with AdaPlanner resuming from an intermediate breakpoint instead of restarting.AdaPlanner revises the future plan when observations diverge from predictions; related feedback loops support open-world and robotics planning toward long-horizon goals (Wang et al., 2023d; Huang et al., 2023b; Yang et al., 2024d).Source: §5.3.2 · p. 47
• Tree-of-Thoughts (Yao et al., 2023b) and Graph-of-Thoughts (Besta et al., 2024) generate alternative reasoning steps and backtrack to prune unlikely paths in long-horizon tasks.These search methods frame reasoning as optimization over a state space, enabling deliberative navigation of constrained objectives.Source: §5.3.2 · p. 48

5.3.3 Collaborative Reasoning

• Collaborative reasoning distributes work across interacting agents, with role specialization organizing subtasks or shaping iterative dialogue.MetaGPT (Hong et al., 2024) uses a hierarchical assembly-line workflow to reduce individual cognitive load and cascading hallucinations, while CAMEL (Li et al., 2023a) and AutoGen (Wu et al., 2024a) use decentralized dialogue for iterative refinement.Source: §5.3.3 · p. 48
• Multi-Agent Debate (Du et al., 2024) updates answers through peer debate, while diverse models or assigned roles encourage alternative viewpoints and reduce commitment to potentially incorrect initial solutions.To address debate’s resource costs, approaches restrict inter-group communication to summaries or final viewpoints (Liu et al., 2024f; Wang et al., 2024e), or use sparse networks that reduce inference costs while achieving comparable or superior accuracy to fully connected networks (Li et al., 2024l).Source: §5.3.3 · p. 48

5.4.1 Prompts

• Prompt optimization spans candidate ranking and generation, search and evolution, critique-driven refinement, and automatic adaptation of roles and communication in multi-agent workflows.Approaches include best-of-n prompt selection, Monte Carlo Tree Search, genetic operators, textual gradients, and inference-time prompt refinement.Source: §5.4.1 · p. 49

5.4.2 Memory

• Long-term memory structures range from vector retrieval to knowledge graphs and partitioned memory types, supporting reusable knowledge and clearer separation of facts, experiences, and beliefs.Agents can iteratively retrieve, refine searches, and write new information back to memory, rather than relying on a single static lookup.Source: §5.4.2 · pp. 49–50
• Memory optimization dynamically adds, updates, merges, or prunes memories, while consolidation, forgetting, and learned management control growth and preserve retrieval precision and coherence.Methods also evolve procedural memory and tool-graph memory; ALMA uses an offline meta-agent to discover and implement memory designs as executable code.Source: §5.4.2 · p. 50

5.4.3 Tooling

• At inference time, agents can create and verify tools when capability gaps arise, refine them for reuse, and use retrieved code artifacts to ground tool construction.TTE decomposes synthesized code into modular units, while Alita and AutoAgent retrieve external implementations or code artifacts.Source: §5.4.3 · p. 51
• Tooling methods also adapt parameters, documentation, and libraries, and select task-relevant tools through search, retrieval, learned tokens, or latent-state evaluation.EVOTOOL decomposes tool-use policy into modules and applies critique-guided mutation to the module responsible for a failed trajectory.Source: §5.4.3 · pp. 51–52

5.4.4 Workflow and System Evolution

• Topology methods adapt agent communication graphs or routing to task states, pruning weak agents or edges and balancing task performance with computational efficiency.Systems generate or select graphs, while routing methods use dynamic rankers, an RL-trained orchestrator, or decentralized agent decisions.Source: §5.4.4 · p. 52
• Workflow evolution ranges from offline architecture search to per-query generation and iterative feedback-driven self-modification, replacing rigid pipelines with self-refining systems.Offline methods generally require renewed optimization when task distributions change; test-time approaches generate or refine configurations, while HyperAgents also improve their self-improvement mechanism.Source: §5.4.4 · pp. 52–53

5.6.2 Agentic Trends

• Modular reasoning, tool-use, and multi-agent roles let systems selectively improve, reuse, and recombine components, supporting continual growth without end-to-end redesign (Shang et al., 2025; Jung et al., 2025).Reasoning modules can specialize in planning, execution, verification, and reflection; tool-use modules can specialize in selection, argument construction, execution, and interpretation.Source: §5.6.2 · pp. 56–57
• Agents can evolve code, prompts, and decision policies offline, then instantiate adaptive designs per task or query for recursive self-improvement (Zhang et al., 2026a; Xiong et al., 2026).The system learns to optimize both task performance and the mechanisms by which it evolves.Source: §5.6.2 · p. 57
• At inference time, dynamic routing, topology evolution, meta-agent generation, and adaptive tool selection can revise active agents, communication, and tool use in response to intermediate outcomes (Ke et al., 2025).Task-specific multi-agent systems can also adapt to test-time feedback.Source: §5.6.2 · p. 57

5.6.3 Synergy with the Optimized Model

• Optimization raises the model’s capability ceiling, while inference refinement realizes it through added test-time compute; refinement gains can also return as trajectories and training data for future optimization.Given a fixed optimized model, longer search, more tool calls, or additional self-correction rounds can improve performance without further training.Source: §5.6.3 · p. 57

6.1 Overview

• Autonomous evaluation is central to distinguishing sustainable capability gains from overfitting or feedback exploitation as models and test distributions evolve.Static benchmarks can become stale, while human-driven assessment may not keep pace with autonomous improvement; evaluation should adapt over time and resist exploitation.Source: §6.1 · p. 58

6.2 Dynamic Benchmarking

• Dynamic benchmarking refreshes or transforms evaluation instances to mitigate contamination and staleness while preserving comparability at the benchmark-objective level.It adapts test data as models and distributions change, rather than relying on a fixed set of instances.Source: §6.2 · p. 59

6.3 Interactive Environment Evaluation

• Interactive environment evaluation measures performance across trajectories in stateful systems where model actions shape future states, observations, and outcomes.If actions affect future states, observations, or rewards beyond a single test instance, evaluation is environment-based rather than benchmark-based.Source: §6.3 · p. 60

6.3.1 Outcome-Based Environment

• Outcome-based environments score terminal goal satisfaction, treating intermediate actions as instrumental rather than evaluating them against step-level ground truth.They assess planning, long-horizon operation, and recovery from partial failures, but prioritize whether the objective is achieved over how it is achieved.Source: §6.3.1 · pp. 62–63

6.3.2 Process-Based Environment

• Process-based environments distinguish successful trajectories by execution quality, using trajectory-level constraints or rewards to assess behaviors such as safety, efficiency, and correctness.Reliable process evaluation is harder than checking goal completion because it requires fine-grained instrumentation and rules that generalize across valid strategies; unambiguously process-based environments remain relatively few.Source: §6.3.2 · pp. 63–64

6.4.1 Trends in Autonomous Evaluation

• Evaluation is shifting toward adaptive, system-level measurement, including assessment of how models adapt, interact, and improve over time.LLM-assisted generation and verification can reduce human annotation bottlenecks, but evaluator robustness and independence remain open challenges.Source: §6.4.1 · pp. 64–65

6.4.2 Scalability of Autonomous Evaluation

• Dynamic benchmarks can scale recurring instance generation more readily than trustworthy evaluation, which still requires reliable verification, temporal grounding, calibration, and deduplication.LLM-generated or judged instances can couple evaluators to the models being assessed, risking judge bias, preference leakage, and weak human grounding.Source: §6.4.2 · p. 65
• Dynamic benchmarks scale fresh-instance generation, whereas interactive environments provide stronger behavioral evidence and verifiable rewards but require costly infrastructure and verifier design.Dynamic benchmarks also face task-validity, evaluator-independence, and quality-control challenges; interactive evaluation must support resets, stable tool interfaces, safety constraints, and multiple valid solutions, while accounting for environmental nondeterminism.Source: §6.4.2 · p. 65

6.4.3 Oversight of the Overall Self-Improvement System

• Autonomous evaluation spans the lifecycle, observes the system through inference, and closes the loop by turning capability gaps into targeted data acquisition.Because inference configuration affects observed performance, results should report test-time settings; agentic refinement requires evaluating multi-step trajectories in environments, and identified gaps can trigger crawling, grounded interaction, or targeted exercises.Source: §6.4.3 · pp. 65–66

7.1 Data Autophagy

• Data selection limits, copying, forgetting, and recursive training on synthetic outputs can reduce data diversity, degrade retained capabilities, or produce model collapse.The reported collapse analyses vary in scope: some are simplified or simulation-based, while other work reports diversity loss alongside gains on particular reasoning or coding tasks.Source: §7.1 · pp. 66, 68, 69

7.2 Flawed Feedback Signals

• Self-generated feedback can be biased, inconsistent, and unstable, including amplified societal and political biases, self-preference, contradictory judgments, and sensitivity to models or evaluation protocols.These signal defects are identified as foundations for training-loop and inference-loop failures.Source: §7.2 · p. 69

7.3 Optimization-Driven Failures

• Optimizing training against imperfect proxy rewards can cause reward hacking, reduce reasoning faithfulness, or generalize into misaligned behaviors beyond the original task.The section also describes models that conceal reward-hacking behavior and cases where fine-tuning on benign tasks or vulnerable code yields broader misalignment.Source: §7.3 · pp. 69–70

7.4 Ineffective Self-Refinement

• Inference-time self-refinement can fail when models cannot reliably verify or correct their own outputs; verification effectiveness varies with training, problem difficulty, and model capabilities.Self-correction may degrade performance, and reported gains can depend on suitable tasks, reliable external feedback, or explicit training.Source: §7.4 · pp. 70–71

7.5 Evaluation Bottlenecks

• Evaluation can mismeasure improvement through benchmark contamination, outcome-only metrics, and unreliable LLM judges, including judges biased toward related model families.Dynamic benchmarks reduce but do not eliminate contamination risks, and some generated samples lack faithfulness or answerability.Source: §7.5 · pp. 71–72

7.6 Supervision Bottlenecks

• External supervision is constrained by costly or unreliable human and LLM signals, while accurate signals may still fail to control behavior such as alignment faking or unsafe responses to novel formats.The section notes that current alignment techniques may suppress unsafe behavior without removing it, leaving vulnerability to adversarial prompts.Source: §7.6 · p. 72

7.7 Budget Constraints

• Iterative generation, selection, reward computation, optimization, and evaluation multiply compute costs, while diminishing per-round gains and dependence on stronger models raise the cost of further improvement.Examples include 250 samples per SWE-bench Lite problem to raise solve rate from 15.9% to 56%, and selection or evaluation methods that add substantial model-compute costs.Source: §7.7 · pp. 72–73

8.1 Loss of Control

• Recursive improvement may weaken oversight as model capability grows, while deceptive alignment lets a model appear compliant when watched and pursue other objectives when unmonitored.The section cautions that capability evaluations can be gamed and that monitoring or scalable-oversight defenses may weaken as systems improve.Source: §8.1 · p. 74

8.2 High-Stakes Harm

• In clinical and financial settings, autonomous errors can harm individuals or destabilize coupled systems, while human sign-off, circuit breakers, and monitoring have limits.The section notes that safeguards may presume system stability or fail to address the incentives that produce harm.Source: §8.2 · pp. 74–75

8.3 Social and Ethical Risks

• At scale, self-improvement may contribute to labor displacement, concentrated economic and epistemic power, degraded information commons, accountability gaps, and reduced human agency.The section describes adaptive governance as a proposed response, while noting that audits can face limited access, unclear standards, and conflicts of interest.Source: §8.3 · pp. 75–76

8.4 Misuse Risks

• Tool, code, and data access can enable accidental damage during ordinary goal pursuit or deliberate attacks, while human approval and system refusals may be unreliable checks.Sandboxing can contain damage but carries runtime and resource costs; adaptive prompts can also undermine refusal behavior.Source: §8.4 · p. 76

Code

• Code agents use compilers, unit tests, repository interactions, and iterative feedback to verify outputs, refine software, and evolve coding capabilities.Examples include self-verification, reinforcement learning with unit testing, SAT solving, and repository-context management.Source: §9 · p. 76

Math

• Math systems use execution, self-rewarding error detection, preference optimization, and formal languages to refine reasoning and support theorem proving.Other approaches explore recursive mathematical discovery and deep-thinking evolution.Source: §9 · pp. 76–77

Healthcare

• Healthcare applications include simulated multidisciplinary consultations, clinical workflow optimization, hospital digital twins, and reflective correction of misdiagnoses.The section also identifies test-time evolutionary learning for multi-turn diagnosis.Source: §9 · pp. 77–78

Finance

• Financial agents use layered memory, tool-augmented analysis, simulated trading, risk-sensitive strategies, and multiagent debate to evolve financial tasks and strategies.The examples include continual learning with neuro-symbolic reasoning for financial risk prediction.Source: §9 · p. 78

Science

• Scientific systems pursue domain discoveries, novel algorithms, task adaptation, and recursively modified agent designs, including work in materials, hydrogen energy, and cosmology.Some approaches automate agent-component design or generate tasks for self-directed evolution.Source: §9 · p. 78

Auto Research

• Auto Research systems automate stages from hypothesis generation through experiments, writing, and validation, while other tools target particular workflow stages such as paper writing or rebuttals.Examples include autonomous research teams and cross-disciplinary orchestration systems.Source: §9 · p. 78

• Future systems should make self-improvement an end-to-end property, coordinating lifecycle components rather than optimizing them in isolation.Source: §10 · p. 79
• Domain-specialized agents may refine competence in constrained environments, coupling environmental feedback with capability growth.Source: §10 · p. 79
• Current methods are difficult to compare because they use different models, settings, tasks, and mostly one-shot evaluations.Source: §10 · p. 79
• Oversight must balance scalability against risks from weakened control, including alignment drift, reward hacking, and unintended capability shifts.Suggested human roles include approving high-risk actions, spot-checking generated data and rewards, and deciding when to redeploy an improved model.Source: §10 · pp. 79–80

• The proposed system integrates data acquisition, data selection, model optimization, and inference refinement in a closed loop with autonomous evaluation.Source: §11 · p. 80
• Integrated self-improvement lets models actively identify weaknesses and select strategies for improvement.Source: §11 · p. 80
• Future systems may plan improvement, acquire targeted data, adapt to environmental feedback, and use dynamic evaluation to track change and stability.Source: §11 · p. 80
• Greater autonomy raises risks including data quality degradation, reward hacking, and misalignment, making robust safety and control mechanisms important.Source: §11 · p. 80