PaperlayerView exact revision on arXiv

Source-linked AI summary

The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback

Hangfan Zhang, Siyuan Xu, Zhimeng Guo, Huaisheng Zhu, Shicheng Liu, Xinrun Wang, Qiaosheng Zhang, Yang Chen, Peng Ye, Lei Bai, Shuyue Hu

arXiv:2510.02752v1cs.CL

TL;DR

RL can improve LLM reasoning, but expert-dependent data curation is costly. Self-aware RL alternates task generation and solving while selecting tasks by predicted difficulty and seeking outside guidance at capability limits. The paper reports improvements across code and math benchmarks, including a 53.2% relative gain using less than 1% of the extra data required by conventional methods.

  • Problem

    RL can enhance LLM reasoning, but training often requires costly, time-consuming expert creation of tasks and annotations.

    Source: §1 · p. 1

  • Method

    Self-aware RL alternates task generation and solving, using predicted difficulty to select challenging solvable tasks and requesting external guidance for valuable tasks beyond current capability.

    Source: §1 · pp. 2–3

  • Results

    Self-aware RL improves performance on out-of-distribution code generation and out-of-domain mathematical reasoning; the conclusion reports a 53.2% relative gain using less than 1% of the extra data required by conventional methods.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    The findings indicate the potential of self-evolving agents to reduce reliance on large-scale human annotation.

    Source: §6 · p. 11

Abstract

from arXiv · show

Reinforcement learning (RL) has demonstrated potential in enhancing the reasoning capabilities of large language models (LLMs), but such training typically demands substantial efforts in creating and annotating data. In this work, we explore improving LLMs through RL with minimal data. Our approach alternates between the LLM proposing a task and then attempting to solve it. To minimize data dependency, we introduce two novel mechanisms grounded in self-awareness: (1) self-aware difficulty prediction, where the model learns to assess task difficulty relative to its own abilities and prioritize challenging yet solvable tasks, and (2) self-aware limit breaking, where the model recognizes when a task is beyond its capability boundary and proactively requests external data to break through that limit. Extensive experiments on nine benchmarks showing a 53.8% relative improvement with less than 1.2% extra data demonstrate the efficacy of self-aware RL and underscore the promise of self-evolving agent training.

1 Introduction

RL can improve LLM reasoning, but expert-created tasks and annotated solutions make data-intensive training costly. Self-aware RL addresses task selection and capability-boundary stagnation by having models gauge task difficulty and request outside guidance when needed.

1 Introduction

  • Expert task creation and solution annotation make RL data curation costly and time-consuming, creating a bottleneck for LLM progress.Self-generated tasks and attempted solutions can form a self-improving RL loop that reduces reliance on manually curated data.Source: §1 · pp. 1–2
  • Self-generated tasks can be too easy to prompt deeper reasoning or too difficult to provide useful feedback, and their difficulty must adapt as capabilities evolve.The task generator may also stop proposing sufficiently challenging tasks as the model reaches its current knowledge limits.Source: §1 · p. 2
  • The paper frames self-awareness as understanding changing strengths, weaknesses, and capability boundaries so task difficulty can track the model’s current state.Its contributions connect this self-awareness to improving LLMs while reducing reliance on external data.Source: §1 · pp. 2–3
  • The approach combines difficulty prediction, which prioritizes challenging but solvable tasks, with limit breaking, which requests external guidance for valuable tasks beyond current capability.Difficulty estimates are aligned with actual success rates; limit breaking uses predicted difficulty and task novelty.Source: §1 · p. 2
  • Self-aware RL significantly improves both out-of-distribution code generation and out-of-domain mathematical reasoning, with a 53.8% average relative gain on mathematical reasoning benchmarks.The experiments use Qwen2.5-Coder-3B across nine benchmarks.Source: §1 · p. 2

2 Related Works

Related work covers verifiable-reward RL and self-evolving agents as routes to LLM improvement, alongside curriculum methods for selecting suitable training tasks. The paper identifies capability-unaware task generation and the combination of curriculum learning with self-evolving RL as open concerns.

Reinforcement Learning with Verifiable Reward (RLVR)

  • RL with verifiable rewards avoids human-crafted imitation responses but still needs many high-quality tasks to encourage diverse and effective reasoning.The section surveys RLVR training and stabilization methods, including GRPO, DAPO, GSPO, and Dr.GRPO.Source: §2 · p. 3

Self-evolving RL

  • Self-evolving approaches generate tasks or environments to reduce data dependence, but the surveyed methods do not account for the agent’s own capabilities when setting task difficulty.Examples include Absolute Zero Reasoner (Zhao et al., 2025), WebEvolver (Fang et al., 2025), WebRL, and ZeroGUI (Yang et al., 2025).Source: §2 · p. 3

Curriculum RL in LLM Reasoning

  • Curriculum methods dynamically select, filter, or structure tasks by difficulty, while combining curriculum learning with self-evolving RL remains underexplored.The paper explores combining these approaches and states that curriculum design is especially important in self-evolving RL training.Source: §2 · p. 4

3 Preliminaries

The preliminaries define the policy and task-response notation, then introduce REINFORCE++ and its normalized advantage term.

3 Preliminaries

  • The paper denotes the LLM parameterized by θ as policy model πθ, also using that term for an agent driven by the policy.A task is x and its solver response is y; both are token sequences.Source: §3 · p. 4
  • REINFORCE++ is presented as an advanced RL algorithm for LLM post-training, with a learning objective on a sampled task-response pair.The passage identifies REINFORCE++ (Hu et al., 2025) as the algorithm.Source: §3 · p. 4
  • The normalized advantage term is denoted Ât.The passage introduces it in connection with the learning objective.Source: §3 · p. 4

4 Method

Self-aware RL alternates task generation and solving, using predicted difficulty and selective external guidance to support self-evolution with minimal data. Its pipeline combines rewards for difficulty prediction, task outcomes, and response format.

4 Method

  • The self-evolving loop pairs a generator that proposes tasks with a solver that attempts them, using verifiable code outcomes for RLVR training.The framework is implemented for code generation with a Python interpreter, and can apply in other environments that verify responses.Source: §4 · pp. 4–5, Figure 1 · p. 5

4.1 Self-aware difficulty prediction

  • The generator predicts how many of N solver rollouts will be correct, then receives reward for aligning its estimate with the observed success rate.This difficulty estimate is intended to help the generator produce tasks suited to the model’s current capabilities.Source: §4.1 · pp. 5–6

4.2 Self-aware limit breaking

  • When all solver paths fail, limit breaking checks task utility and obtains guidance from a stronger external solver for valuable tasks.The method selects failed tasks based on difficulty and novelty; novelty uses the solver’s token-level perplexity and can indicate unfamiliarity with a task.Source: §4.2 · pp. 6–7
  • External-guidance selection is based on each task’s utility relative to recent tasks, with higher-utility tasks assigned higher selection probability.A fixed-size FIFO buffer stores recent tasks; the utility z-score sets p(x), while τ controls its expected value and γ its sharpness.Source: §4.2 · p. 7

4.3 Self-aware RL training pipeline

  • The generator is trained with difficulty-prediction and format rewards, while the solver receives binary outcome and format rewards under a compatible RL pipeline.The implementation uses REINFORCE++ for verification.Source: §4.3 · pp. 7–8

5 Experiment

Across nine benchmarks, self-aware RL improved performance while using external guidance on only a small fraction of tasks. The experiments analyze training behavior and test the contributions of limit breaking, guidance frequency, and utility ranking.

5.1 Experimental Setup

  • Evaluation used six mathematical datasets and three coding benchmarks, with Qwen2.5-Coder-3B as the base and Qwen2.5-Coder-32B as the external policy.Limit breaking was disabled for the first 50 steps because early difficulty predictions were inaccurate.Source: §5.1 · p. 8

5.2 Experimental Results

  • 53.8% average improvement on mathematical benchmarks and 5.3% on coding benchmarks came with external guidance on 157 of 12,800 tasks (1.23%).The comparison covers nine benchmarks.Source: Table 1 · p. 8, §5.2 · p. 9

5.3 Analysis

  • After 50 steps, self-aware RL's reward surpassed AZR and difficulty-prediction accuracy rose from 0.2 to over 0.6, so limit breaking began then.The initial reward was lower because the agent was adapting to a more complicated dialogue template.Source: §5.3 · p. 9, Figure 2 · p. 9, Figure 3 · p. 9
  • Rollout accuracy fell as generated tasks grew harder and stabilized around 0.6; selected tasks had significantly higher utility than unselected tasks.The results indicate that the generator adapted task difficulty and the filter distinguished higher-utility tasks.Source: §5.3 · p. 10, Figure 4 · p. 10, Figure 5 · p. 10

5.4 Ablation study

  • Without limit breaking, self-aware RL still outperformed AZR by 25.6% on mathematical reasoning benchmarks.This ablation isolates performance retained with difficulty-prediction self-awareness alone.Source: Table 2 · p. 10, §5.4 · p. 11
  • τ=0.1 yielded the highest performance gain, while randomized utility ranking delivered only 28.0% to 28.4% on math and 53.3% to 53.6% on coding.Larger τ made guidance gains ignorable and may corrupt the solver's original reasoning pattern; without utility ranking, external queries produced no obvious improvement.Source: §5.4 · p. 11, Table 3 · p. 11, Table 4 · p. 11

6 Conclusion

The conclusion presents self-aware RL as a self-evolving framework that uses self-awareness to guide learning and request minimal external data. The reported experiments and proposed future direction frame its potential scope.

6 Conclusion

  • Self-aware RL is a self-evolving framework in which agents guide their own learning by recognizing capability limits and proactively requesting minimal external data.The paper describes this as a way to address common failure modes of naive self-training.Source: §6 · p. 11
  • The conclusion reports a 53.2% relative performance gain while using less than 1% of the extra data required by conventional methods.The paper connects these findings to reducing reliance on large-scale human annotation.Source: §6 · p. 11
  • Future research could extend the self-aware learning framework to more complex, interactive domains.Source: §6 · p. 12

A.1 Training setup

The training setup reports the compute environment, run length, code environment, and a table of hyperparameter configurations.

A.1 Training setup

  • Experiments ran for 200 steps on a server with four NVIDIA H-200 GPUs.Source: §A.1 · p. 13
  • The code environment was implemented using the QWQ Python executor.Source: §A.1 · p. 13
  • Table 5 lists the hyperparameter configurations.Source: Table 5 · p. 13

A.2 Chat templates

The appendix provides chat templates for generating reasoning tasks and predicting their difficulty, with task generation adapted from AZR (Zhao et al., 2025).

A.2 Chat templates

  • The task-generation template is similar to AZR’s because part of the implementation is based on AZR (Zhao et al., 2025).Source: §A.2 · p. 13
  • The difficulty prompt asks the model to assess code complexity, solution steps, creativity, and the current solver’s strength, targeting medium difficulty at 4 or 5 on a 0–8 scale.The prediction is placed in designated review and difficulty-prediction tags.Source: §A.2 · p. 13
  • The task-generation prompt asks for a deterministic Python function and one input whose output is given, creating a puzzle that requires deep algorithmic reasoning.It specifies a multi-step task and restricts random operations, I/O, external state, and execution time.Source: §A.2 · p. 14
  • The templates specify Python and input formatting, including a function example and a sample input format.The expected response format includes think, answer, review, and difficulty-prediction sections.Source: §A.2 · pp. 13–14
  • Task evaluation criteria include executability, reasoning difficulty, creativity relative to reference snippets, and restrictions on specified keywords and packages.The prompt also directs the model to plan the snippet before writing the code and input.Source: §A.2 · p. 15

B The Use of Large Language Models(LLMs)

The authors report using an LLM only as a writing assistant for polishing expressions, not for substantive research work.

B The Use of Large Language Models(LLMs)

Loading 2510.02752v1…