Source-linked AI summary
A Survey on Self-Evolution of Large Language Models
Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, Jingren Zhou
TL;DR
LLMs face costly supervision and possible performance ceilings as tasks become more complex and diverse. This survey proposes a four-phase self-evolution framework, categorizes objectives and methods, and identifies challenges and future directions. It presents self-evolution as a framework for models to acquire, refine, update, and evaluate experiences autonomously.
Problem
Current LLM training relies on costly human or external-model supervision and may face performance ceilings as task complexity and diversity increase.
Method
The survey organizes self-evolution into iterative cycles of experience acquisition, refinement, updating, and evaluation, while synthesizing objectives, methods, challenges, and future directions.
Results
The survey provides a comprehensive framework and taxonomy for understanding self-evolving LLMs, covering their objectives, process stages, current challenges, and prospective directions.
Takeaways & Limitations
Self-evolution is presented as a potential training paradigm for enabling LLMs to adapt, learn, and improve autonomously.
Takeaways & Limitations
The field lacks solid theoretical grounding, with mixed evidence on whether LLMs can self-correct reasoning errors without external feedback.
Abstract
from arXiv · showhide
Large language models (LLMs) have significantly advanced in various fields and intelligent agent applications. However, current LLMs that learn from human or external model supervision are costly and may face performance ceilings as task complexity and diversity increase. To address this issue, self-evolution approaches that enable LLM to autonomously acquire, refine, and learn from experiences generated by the model itself are rapidly growing. This new training paradigm inspired by the human experiential learning process offers the potential to scale LLMs towards superintelligence. In this work, we present a comprehensive survey of self-evolution approaches in LLMs. We first propose a conceptual framework for self-evolution and outline the evolving process as iterative cycles composed of four phases: experience acquisition, experience refinement, updating, and evaluation. Second, we categorize the evolution objectives of LLMs and LLM-based agents; then, we summarize the literature and provide taxonomy and insights for each module. Lastly, we pinpoint existing challenges and propose future directions to improve self-evolution frameworks, equipping researchers with critical insights to fast-track the development of self-evolving LLMs. Our corresponding GitHub repository is available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/Awesome-Self-Evolution-of-LLM
1 Introduction
LLMs face challenges on increasingly complex tasks because existing training paradigms require difficult modeling, annotation, and evaluation, while further scaling real-world data may be limited. The survey organizes self-evolution as an iterative cycle for autonomously acquiring, refining, updating, and evaluating experiences.
- Existing training paradigms struggle with sophisticated tasks because modeling, annotation, and evaluation are difficult, while scaling real-world data may face limits.
- Self-evolution offers a path beyond intensive human supervision by enabling models to learn through autonomously generated experiences, inspired by self-play successes such as AlphaZero.
- Research spans self-instruct, self-play, self-improving, and self-training, but their relationships lacked systematic organization and analysis.
- The survey frames self-evolution as iterative experience acquisition, refinement, updating, and evaluation, with evaluation measuring progress and setting new objectives.
2 Overview
The survey conceptualizes self-evolution as a cyclical process that mirrors human learning: models acquire task experiences, refine them, update themselves, and evaluate progress to guide the next objective.
- 2.2 Conceptual Framework: The framework organizes self-evolution into repeated cycles focused on an evolution goal, relevant tasks, experience optimization, model updating, and progress evaluation.
- Experience Acquisition: Experience acquisition generates tasks, solutions, and environmental feedback under objective E_t, producing new experiences (T_t, Y_t, F_t).
- Experience Refinement: Experience refinement discards incorrect data and improves imperfect data, yielding refined task-solution pairs (˜T_t, ˜Y_t).
- Updating and Evaluation: Updating integrates refined experiences into the model, while evaluation assesses performance externally and informs the next objective E_t+1.
3 Evolution Objectives
Evolution objectives specify what ability should improve and in which direction, covering both LLM capabilities and agent skills across diverse self-evolution methods.
- An evolution objective combines an evolving ability with an evolution direction, formalized as E_t = (A_t, D_t).For example, reasoning is the ability and accuracy improving is the direction.
- The taxonomy distinguishes targeted abilities of LLMs from abilities of LLM-based agents, including instruction following, reasoning, planning, tool use, embodied control, and communication.
- Table 1 categorizes methods across evolutionary stages and tracks directions including feedback adaptation, knowledge-base expansion, safety, ethics, bias reduction, and performance enhancement.
- Performance improvement can extend models toward diagnostic dialogue, social skills, and role-playing, illustrating diverse evolution directions.
4 Experience Acquisition
Experience acquisition is the exploratory first phase of self-evolution, in which models create tasks, develop solutions, and optionally obtain environmental feedback to improve future behavior.
- Experience acquisition helps models adapt to new tasks, overcome knowledge limitations, and improve solution effectiveness through exploration.
- Experience includes encountered tasks, developed solutions, and feedback received from task performance.
- The phase comprises task evolution, solution evolution, and optional feedback collection from interaction with the environment.
4.1 Task Evolution
Task evolution generates or selects tasks aligned with the current evolution objective, using external knowledge, the model itself, or selective sampling from existing tasks.
- 4.1 Task Evolution: Task evolution maps the current objective and model to an evolved task, categorized as Knowledge-Based, Knowledge-Free, or Selective.The survey formalizes this process as T_t = f_T(E_t, M_t).
- Knowledge-Based: Knowledge-Based methods use structured or unstructured external information to improve task relevance and factual validity.Examples include topic and character knowledge, scientific texts, and mined events.
- Knowledge-Free: Knowledge-Free methods generate tasks from the evolution objective and model without requiring additional external knowledge.Self-prompting, task rewriting, mutation, and backtranslation are representative strategies.
- Selective: Selective methods choose relevant tasks from an existing pool instead of generating new ones, reducing curation requirements.Selection can be random, diversity-oriented, objective-based, or guided by solution-related metrics.
4.2 Solution Evolution
Solution evolution develops task solutions that are relevant to the evolution objective, using positive or negative experiences and strategies ranging from rationales to interaction, self-play, and grounding.
- 4.2 Solution Evolution: Solution evolution maps the evolved task, objective, and model to a solution strategy, while addressing the risk of objective-irrelevant outputs.The survey categorizes solutions as positive or negative according to correctness and intended use.
- Positive Solutions: Positive solution methods include rationale-based, interactive, self-play, and grounded task solving to obtain correct and objective-aligned outputs.Rationales make the objective explicit, interaction supplies environmental feedback, self-play creates closed-loop feedback, and grounding uses rules or prior experiences.
- Grounded: Grounding solutions in predefined rules or previous experiences provides explicit guidance for solving tasks and retrieving relevant knowledge.Systems may use principles, workflows, question-answer records, induced rules, or user-portrait knowledge.
- Negative Solutions: Negative solutions support self-improvement through trial and error by contrasting desirable and undesirable outputs or intentionally perturbing solutions.Contrastive methods compare solution pairs, whereas perturbative methods deliberately create harmful or incorrect outputs for avoidance learning.
- Perturbative: Perturbative approaches generate negative solutions through harmful instructions or negative contexts, including incorrect persona conversations.These negatives are used to teach models what responses or behaviors to avoid.
4.3 Feedback
Feedback evaluates task solutions during or after solving, either through model-generated judgments or signals from external environments.
- 4.3 Feedback: Feedback provides correctness information that enables reflection and subsequent model updating in the self-evolution cycle.The survey formalizes feedback acquisition as F_t = f_F(T_t, Y_t, E_t, M_t; ENV).
- Model Feedback: Model feedback consists of self-generated scores or textual critiques used to assess and improve solutions.Scores can represent correctness or confidence, while critiques provide multidimensional natural-language guidance.
- Environment Feedback: Environment feedback comes from code interpreters, tool execution, embodied settings, or other agents and can provide precise guidance for updating.Execution traces and error messages can identify information needed for improved code generation.
- Environment Feedback: Communication feedback enables multi-agent systems to correct and support one another through co-evaluation.This feedback is common in systems where agents interact through debate, principles, or role-based communication.
5 Experience Refinement
Experience refinement improves acquired tasks and solutions before updating by filtering unreliable outputs and correcting imperfect experiences through critique-based or critique-free methods.
- 5 Experience Refinement: Experience refinement transforms acquired tasks and solutions into refined experiences before model updating, using filtering or correcting methods.The survey formalizes refined experiences as (T~_t, Y~_t) = f_R(T_t, Y_t, F_t, E_t, M_t).
- Filtering: Filtering retains reliable, high-quality experiences using metric-based or metric-free criteria.Metric-based methods use external rewards or task metrics, while metric-free methods rely on consistency or model-inherent criteria.
- Metric-Based Filtering: Metric-based filtering applies correctness, accuracy, completed-subgoal, or similarity criteria to select training experiences.Examples include binary rewards, F1-score, accuracy, subgoal completion, and ROUGE-L-based removal of repeated instructions.
- Metric-Free Filtering: Metric-free filtering selects outputs using internal agreement or verification, but designing internal measures that reflect quality remains challenging.Self-consistency, verification scores, and code-test agreement are representative criteria.
- Correction: Correction methods use critiques or objective signals to improve experiences, with critique-based approaches offering detailed guidance and critique-free approaches simplifying implementation.Critiques may come from the model or tools, whereas critique-free methods use correctness signals, execution results, refined prompts, or input evidence.
6 Updating
The survey classifies updating methods into in-weight and in-context learning, covering strategies for preserving prior capabilities while acquiring new ones and for adapting agents through memory.
- Updating approaches divide into in-weight learning, which changes model weights, and in-context learning, which updates external or working memory.
- In-weight learning: Preventing catastrophic forgetting while improving capabilities is the central challenge of iterative self-evolution, addressed through replay, regularization, and architecture-based strategies.
- In-weight learning: Replay methods preserve prior knowledge by mixing old data with new outputs or generating synthetic rehearsal data for subsequent updates.
- In-weight learning: Regularization constrains behavioral or weight changes, while architecture-based methods isolate task-specific parameters or merge models to limit forgetting and improve performance.
- In-context learning: In-context updating adapts models without parameter training by collecting, retrieving, reflecting on, and forgetting experiences in external or working memory.
- In-context learning: Agent memory integration refines states or beliefs from past experiences, supporting improved performance and adaptability across tasks and environments.
7 Evaluation
Evaluation measures an evolved model’s current performance and guides its next learning objective, but accurate assessment and improvement direction remain underexplored. The survey contrasts quantitative metrics with qualitative analyses and highlights scalable LLM-based judging.
- Accurately assessing evolved models and identifying directions for future improvement remain crucial but underexplored problems.
- Evaluation produces both a performance score and an evolving goal for the next iteration through quantitative or qualitative assessment.
- Quantitative Evaluation: Traditional automatic metrics struggle with increasingly complex tasks, while human evaluation is costly or unsuitable for autonomous self-evolution.
- Quantitative Evaluation: LLM-as-a-judge offers a cost-effective, scalable evaluation approach that can closely match human judgment through pairwise, single-answer, or reference-guided grading.
- Qualitative Evaluation: Qualitative evaluation uses case studies, reasoning, debate, and past-error analysis to provide guidance for subsequent iterations.
8 Open Problems
The survey identifies unresolved challenges spanning objective coverage, autonomy, theory, iterative learning, knowledge retention, evaluation, and safety in self-evolving LLMs.
- 8.1 Objectives: Diversity and Hierarchy: Existing evolution objectives cover only a small fraction of human needs, motivating frameworks for diversified and hierarchical real-world objectives.Tool-use taxonomies suggest decomposing broad objectives into manageable sub-goals.
- 8.2 Level of Autonomy: From Low to High: Most current frameworks remain low-level, requiring user-designed modules and objective-specific human effort, while medium- and high-level autonomy remain urgent needs.High-level autonomy requires the model to diagnose deficiencies and construct its own improvement methods.
- 8.3 Experience Acquisition and Refinement: From Empirical to Theoretical: Self-evolution lacks solid theoretical grounding, with mixed evidence on whether LLMs can self-correct reasoning errors without external feedback.One study observed self-correction in models exceeding 22 billion parameters, whereas another found difficulty correcting reasoning errors without external feedback.
- 8.3 Experience Acquisition and Refinement: From Empirical to Theoretical: Self-generated training data may reduce linguistic diversity and cause model collapse, while current methods often stop improving after more than three self-evolution rounds.A proposed explanation is that the self-critic has not co-evolved with the evolving objective.
- 8.4 Updating: Stability-Plasticity Dilemma: Iterative self-evolution must balance retaining prior knowledge with adapting to new tasks, because conventional approaches may not resolve the stability-plasticity dilemma efficiently.Training from scratch could mitigate catastrophic forgetting but becomes highly inefficient as model parameters grow.
- 8.5 Evaluation and 8.6 Safety and Superalignment: Dynamic benchmarks are needed because static tests can become obsolete or unreliable when evolving models access test data through environments, while scalable alignment remains necessary for safety.Sotopia exemplifies an environment-based evaluation approach, and Superalignment combines scalable training, validation, oversight, robustness, interpretability, and adversarial testing.
9 Conclusion
The survey frames self-evolution as a shift toward LLMs that learn through iterative experience acquisition, refinement, updating, and evaluation rather than relying primarily on human annotation or teacher models. It synthesizes objectives, methods, challenges, and future directions toward more autonomous and capable systems.
- 9 Conclusion: Self-evolution offers a framework for LLMs to acquire, refine, update, and evaluate experience while reducing reliance on human annotation and teacher models.The survey organizes current methods and objectives within these iterative cycles.
- 9 Conclusion: The survey identifies challenges and future directions intended to accelerate more dynamic, intelligent, and efficient self-evolving models.Its broader stated goal is progress toward systems capable of surpassing human performance on complex real-world tasks.