PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation

Enci Zhang, Xingang Yan, Wei Lin, Tianxiang Zhang, Qianchun Lu

arXiv:2505.08364v2cs.AI

TL;DR

Static curricula can misalign as model competence changes, and Zero-RL can produce no learning signal when all rollouts receive zero reward. The paper combines adaptive difficulty re-estimation with expert-guided solution reformulation. Together, these strategies improve performance over standard Zero-RL by 10% on AIME24 and 16.6% on AIME25.

  • Problem

    Static curricula can misalign as competence changes, while Zero-RL rollouts can all receive zero reward on problems beyond current capability, leaving no policy-update signal.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    ADCL periodically re-estimates and re-sorts upcoming batches; EGSR guides models to reconstruct expert solutions in their own conceptual frameworks rather than imitate them.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    10% on AIME24 and 16.6% on AIME25 over the standard Zero-RL baseline with ADCL and EGSR combined.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The strategies are presented as complementary refinements to Zero-RL, with EGSR intended to foster reasoning capabilities beyond the model’s initial scope.

    Source: §1 · p. 2

    Evidence is primarily from one model architecture and size and mathematical-reasoning datasets; generalization across model types, scales, and domains remains to be studied.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite impressive progress in areas like mathematical reasoning, large language models still face significant challenges in consistently solving complex problems. Drawing inspiration from key human learning strategies, we propose two novel strategies to enhance the capability of large language models to solve these complex problems. First, Adaptive Difficulty Curriculum Learning (ADCL) is a novel curriculum learning strategy that tackles the Difficulty Shift phenomenon (i.e., a model's perception of problem difficulty dynamically changes during training) by periodically re-estimating difficulty within upcoming data batches to maintain alignment with the model's evolving capabilities. Second, Expert-Guided Self-Reformulation (EGSR) is a novel reinforcement learning strategy that bridges the gap between imitation learning and pure exploration by guiding models to reformulate expert solutions within their own conceptual framework, rather than relying on direct imitation, fostering deeper understanding and knowledge assimilation. Extensive experiments on challenging mathematical reasoning benchmarks, using Qwen2.5-7B as the base model, demonstrate that these human-inspired strategies synergistically and significantly enhance performance. Notably, their combined application improves performance over the standard Zero-RL baseline by 10% on the AIME24 benchmark and 16.6% on AIME25.

1 Introduction

The paper proposes ADCL and EGSR to address static-curriculum mismatch and limits of on-policy exploration, drawing on human learning strategies. Their combination improves results over standard Zero-RL on AIME24 and AIME25.

  • Static difficulty definitions can misalign with a model’s changing needs, while on-policy exploration limits exposure to reasoning patterns beyond its pretraining knowledge.Source: §1 · p. 1
  • The strategies draw on the Zone of Proximal Development and self-explanation: suitably challenging tasks and active reformulation are presented as supports for learning.Source: §1 · pp. 1–2
  • ADCL periodically re-estimates difficulty to align training batches with the model’s evolving capabilities, while EGSR guides reconstruction of expert solutions within the model’s conceptual framework.Source: §1 · p. 2
  • The paper identifies Difficulty Shift as undermining static curricula and reports that EGSR enables more stable on-policy knowledge integration than direct imitation.Source: §1 · p. 2
  • 10% on AIME24 and 16.6% on AIME25: combining ADCL and EGSR improves performance over the standard Zero-RL baseline.Source: §1 · p. 2

2 Related Work

The related work describes curriculum learning for LLMs and contrasts on-policy and off-policy reinforcement learning. It positions ADCL as dynamic curriculum adjustment and EGSR as a way to use expert demonstrations while generating on-policy trajectories.

2.1 Curriculum Learning for LLMs

  • Curriculum learning typically orders training data from easier to harder; in LLM reinforcement fine-tuning, existing static curricula predetermine difficulty, motivating ADCL’s periodic re-estimation.Source: §2.1 · p. 2

2.2 Reinforcement Learning for LLMs

  • On-policy methods learn from current-policy data, whereas off-policy methods use data from varied policies and may introduce instability and complexity.Source: §2.2 · pp. 2–3
  • EGSR uses expert demonstrations to guide the current policy toward improved, more on-policy trajectories, addressing the challenge of integrating off-policy data with on-policy stability.Source: §2.2 · p. 3

3 Method

The section introduces ADCL, which re-estimates difficulty in upcoming batches as the model changes, and EGSR, which uses expert guidance to generate trajectories in the model’s own policy. The reported analyses show reduced difficulty deviation with ADCL and greater policy alignment for guided trajectories than for expert solutions.

3.1 Adaptive Difficulty Curriculum Learning

  • Difficulty Shift occurs because the model’s difficulty perception changes during training, making the initial fixed ranking increasingly inaccurate and potentially misaligning the curriculum.The paper attributes increasing rank deviation to the model’s growing deviation from its initial state.Source: §3.1 · pp. 3–4
  • ADCL sorts the dataset into sequential batches, then re-estimates and re-sorts the next batch using the model updated on the current batch.This localized process continues through training rather than re-evaluating the entire dataset.Source: §3.1 · p. 4
  • ADCL’s repeated re-estimation of upcoming-batch difficulty significantly corrects deviation compared with predefined curriculum learning.The reported Figure 1 analysis finds increasingly dispersed difficulty ranks under the predefined curriculum as training proceeds.Source: §3.1 · p. 4

3.2 Expert-Guided Self-Reformulation

  • When all G on-policy rollouts receive zero reward, advantage estimates vanish and provide no policy-improvement update, motivating expert guidance.The passage links this situation primarily to problems exceeding the model’s current capabilities.Source: §3.2 · p. 4
  • EGSR uses expert demonstrations to guide the current policy in generating trajectories, incorporating expert knowledge while keeping samples aligned with the model’s capabilities.Its objective combines M guided trajectories with G − M standard rollouts, particularly when initial exploration yields zero rewards.Source: §3.2 · pp. 5–6
  • Guided trajectories τs,a and τa have substantially lower PPL than expert solutions s and are closer to unguided outputs τq.The paper interprets this as greater alignment with the current policy.Source: Figure 2 · p. 5, §3.2 · p. 6

4 Experiments

ADCL and EGSR improve mathematical reasoning within the Zero-RL training setup, with their combination outperforming standard RL on three reported benchmarks. The experiments compare training strategies using curated reasoning datasets and five benchmarks.

4.1 Setup

  • BaseSet-7K contains 6,894 problems with base-model accuracy rates of 10%–90%, and AugSet-10K adds 3,000 problems with 0% accuracy.Difficulty estimates used Qwen2.5-7B with 32 rollouts per problem; the added problems provide more instances for EGSR guidance.Source: §4.1 · p. 6
  • Experiments used Qwen2.5-7B, five mathematical reasoning benchmarks, and pass@8 for AIME24 and AIME25 to mitigate variance on the smaller sets.ADCL used BaseSet-7K with four difficulty-based batches and three difficulty re-estimations; EGSR used AugSet-10K and tested answer-only and solution-plus-answer guidance.Source: §4.1 · p. 6

4.2 Main Results

  • Standard +RL improved performance more consistently than +SFT, which decreased performance versus the base model on AIME24 and AIME25.ADCL consistently outperformed PCL; the authors attribute this to better alignment with the model’s evolving state.Source: §4.2 · p. 7
  • EGSR(a) and EGSR(s, a) surpassed direct off-policy guidance, with EGSR(s, a) achieving the most favorable results among the guidance methods.EGSR(s, a) guides reformulation using both the expert solution process and final answer, while answer-only guidance could yield correct answers with flawed or incomplete reasoning.Source: §4.2 · p. 7
  • 10% on AIME24, 16.66% on AIME25, and 7.73% on Minervamath: combined ADCL and EGSR improved over the +RL baseline.The combination uses ADCL with EGSR(s, a).Source: §4.2 · p. 7

5 Analysis

Analysis reports gains across two additional model architectures, describes how ADCL’s reward dynamics differ from static curricula, and evaluates expert guidance against capability boundaries using pass@32.

5.1 Generalization to Other Architectures

  • ADCL and EGSR consistently and significantly outperformed the RL baseline on Qwen2.5-1.5B and Llama3.1-8B-Instruct.These experiments test whether the strategies generalize across architectures and scales.Source: §5.1 · p. 7, Table 2 · p. 8

5.2 ADCL Training Dynamic

  • ADCL periodically re-estimates difficulty and re-sorts batches, producing steeper reward increases after re-estimation points than PCL’s static ranking.ADCL and PCL reward can decline as task difficulty rises, yet remain above raw accuracy at corresponding difficulty levels; NoCL shows a steady increase in accuracy reward.Source: §5.2 · p. 8, Figure 3 · p. 8

5.3 Impact of Expert Guidance on Capability Boundaries

  • EGSR(s, a) increased pass@32 accuracy on AIME25 by 16.67% over the base model, the largest reported improvement in capability boundaries.The analysis characterizes pass@32 as offering more opportunities to explore diverse solution paths than pass@1 or pass@8.Source: §5.3 · pp. 8–9, Table 3 · p. 8

6 Conclusion

The conclusion presents ADCL as a response to changing difficulty perception and EGSR as a way to reformulate expert solutions, reporting that the strategies outperform baselines, especially in combination.

6 Conclusion

  • ADCL periodically re-estimates problem difficulty, while EGSR guides models to reformulate expert solutions within their own conceptual framework.The conclusion frames these strategies as addressing Difficulty Shift and supporting deeper knowledge assimilation rather than direct imitation.Source: §6 · p. 9

Limitations

The limitations concern the scope of the evaluation and assumptions in the training strategies. Broader tests are needed to assess generalizability and the effects of imperfect expert demonstrations.

Limitations

  • The findings are primarily demonstrated on mathematical reasoning with a specific model architecture and size, leaving broader generalizability to be assessed.The authors call for studies across more model types, scales, and application domains.Source: Limitations · p. 9
  • ADCL is evaluated within RL; its prevalence across training paradigms such as SFT and across domains remains for future investigation.The authors also note that varying ADCL's fixed number of difficulty re-estimations and resortings warrants further study.Source: Limitations · p. 9
  • EGSR assumes high-quality expert demonstrations, and the effects of errors or biases in those demonstrations remain undetermined.Source: Limitations · p. 9

A Pseudocode for ADCL

The ADCL pseudocode estimates sample difficulty, trains on sequential batches, and re-estimates upcoming samples as the model changes. The EGSR pseudocode sets up question, solution, and answer data for iterative policy training with sampled trajectories and rewards.

A Pseudocode for ADCL

  • ADCL first estimates difficulty for every dataset sample using the initial policy, then sorts the samples and partitions them into sequential batches.Source: A Pseudocode for ADCL · p. 12
  • After training on batch Bk, ADCL re-estimates difficulty for samples in the next batch before continuing training.Source: A Pseudocode for ADCL · p. 12

B Pseudocode for EGSR

  • EGSR's dataset consists of question, expert-solution, and answer triples, and training proceeds over iterations using sampled batches.Source: B Pseudocode for EGSR · p. 12
  • For each sampled question, EGSR samples policy trajectories, computes their rewards, and then computes token-level advantages from mixed trajectories.Source: B Pseudocode for EGSR · p. 12

C Training Hyperparameters

The section points readers to the training hyperparameters and training curves for ADCL, EGSR, and SFT. The supplied passages identify these materials but do not report specific hyperparameter values or curve outcomes.

C Training Hyperparameters

D Quantifying ADCL Re-sorting Dynamics

The ADCL re-sorting analysis uses the Normalized Inversion Rate to quantify changes in sample order after difficulty re-estimation. The EGSR prompt asks the model to reconstruct expert reasoning in its own words rather than copy it.

D Quantifying ADCL Re-sorting Dynamics

E Prompts for EGSR

  • The EGSR prompt asks the model to develop its own step-by-step reasoning toward the expert solution's conclusion instead of copying it verbatim.Source: E Prompts for EGSR · p. 14

F Case Study: Emergence of Flawed Reasoning Steps under Answer-Only Guidance in EGSR

The case study contrasts a correct Cauchy–Schwarz solution with flawed reasoning under answer-only guidance. Despite lower output perplexity, the model can appear to force its reasoning toward the provided answer.

F Case Study: Emergence of Flawed Reasoning Steps under Answer-Only Guidance in EGSR

  • Answer-only guidance can yield flawed reasoning even when the final answer is correct, with the model appearing to force a path to that answer.The passage describes this behavior as reminiscent of reward hacking.Source: F Case Study: Emergence of Flawed… · pp. 14–15
  • The flawed τ(a) example cites a generally incorrect formula that ignores d, yet arrives at the correct answer.The example notes that the attempted Lagrange-multiplier system was complex to solve.Source: F Case Study: Emergence of Flawed… · p. 15
  • The correct τ(s, a) example uses Cauchy–Schwarz to bound the expression and states when equality makes the maximum achievable.It assumes positive x and y, then gives the bound 14(x^2 + y^2 + 1) ≥ (x + 2y + 3)^2.Source: F Case Study: Emergence of Flawed… · pp. 14–15

Related papers

Loading 2505.08364v2…