PaperlayerView exact revision on arXiv

Source-linked AI summary

Ladders of Thought: A Self-Evolving Curriculum of Progressively Simplified Reasoning Traces

Minghui Liu, Thomas Magelinski, Dehao Yuan, Qi Yu, Furong Huang

arXiv:2609.25643v1cs.AIcs.LG

TL;DR

Small- and mid-scale LLMs remain brittle reasoners despite knowledge distillation, while their reasoning generalization beyond specific datasets is limited. LoT trains them with progressive question rewrites and a self-evolving curriculum. Across the reported domains and models, it outperforms knowledge-distillation and curriculum baselines, with gains that vary by task.

  • Problem

    Small- and mid-scale LLMs remain brittle reasoners, and their ability to generalize learned reasoning beyond a specific dataset is limited.

    Source: §1 · p. 2

  • Method

    LoT combines semantically faithful easier question rewrites, step-based difficulty buckets, and an adaptive self-evolving bandit scheduler.

    Source: §1 · p. 2

  • Results

    LoT consistently outperforms strong knowledge-distillation and curriculum baselines across math and multi-hop reasoning, with substantial out-of-distribution arithmetic gains and dataset-dependent multi-hop benefits.

    Source: §4.7 · p. 8, §6 · p. 10

  • Takeaways & Limitations

    LoT accelerates convergence compared with flat or staged curricula, and the reported findings favor adaptive progression with moderate rewrite depth.

    Source: §4.7 · p. 8, §6 · p. 10

    Evaluation covers 1–8B models and English math and text-only multi-hop benchmarks; benefits on multi-hop tasks are dataset-dependent.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) excel at reasoning when scaled to hundreds of billions of parameters, but small- and mid-scale models remain brittle reasoners even with knowledge distillation (KD). We present Ladders-of-Thought (LoT), a framework that improves reasoning by combining progressive question rewrites with a self-evolving curriculum. LoT automatically generates semantically faithful but easier variants of reasoning problems, organizes them into difficulty buckets using step-based measures, and employs a self-evolving bandit scheduler to allocate training adaptively. Evaluated on two reasoning domains, math and multi-hop reasoning, across 1-8B models from different families, LoT consistently improves over KD. It delivers large gains on arithmetic tasks (e.g., +32 percentage points on AddSub, +25pp on SVAMP), +2-8pp improvements on in-domain test splits, and strong though dataset-dependent benefits on multi-hop reasoning (e.g., +16pp on QASC, +25pp on StrategyQA). LoT also converges faster than staged curricula, highlighting the value of adaptive progression. These results show that progressive rewrites coupled with adaptive curricula provide a simple yet effective recipe for strengthening reasoning in smaller LLMs.

1 Introduction

Small- and mid-scale LLMs remain brittle reasoners, and supervised reasoning curricula are underexplored partly because reasoning difficulty is hard to define. LoT combines easier, semantically faithful question rewrites with step-based difficulty buckets and adaptive bandit scheduling.

1 Introduction

  • Prior work highlights explicit reasoning traces as a way to guide LLMs toward more accurate and robust answers.Examples include tree-of-thoughts, cumulative reasoning, and DUP.Source: §1 · p. 2
  • Reasoning distillation remains challenging for smaller models, which often fail to generalize learned reasoning beyond a specific dataset.Large student–teacher gaps can lead to reliance on surface heuristics rather than transferable reasoning skills.Source: §1 · p. 2
  • Curriculum learning for supervised reasoning remains underexplored, partly because length, inferential steps, and information structure complicate difficulty measurement.The cited passage distinguishes adaptive curricula, which dynamically select training examples, from simply ordering examples easy to hard.Source: §1 · p. 2
  • LoT combines semantically faithful easier rewrites, step-based difficulty buckets, and a self-evolving bandit scheduler to allocate training adaptively.The framework aims to improve reasoning in small- and mid-scale LLMs.Source: §1 · p. 2
  • The paper introduces progressive rewrites that generate semantically faithful but easier variants, forming a principled difficulty ladder.This is one of the paper’s stated contributions.Source: §1 · p. 2
  • LoT improves pass@5 accuracy, accelerates convergence, and strengthens out-of-distribution generalization for small- and mid-scale LLMs.The introduction reports these benefits across extensive experiments.Source: §1 · p. 2

2 Background

The background frames LoT around chain-of-thought distillation, curriculum learning, and multi-armed bandit scheduling. Together, these concepts describe rationale-based student training, difficulty-ordered sampling, and adaptive selection of training buckets.

2 Background

  • Chain-of-thought distillation trains a student on teacher-generated rationales and final answers, encouraging step-by-step reasoning.The student autoregressively generates the rationale and answer and is optimized with negative log-likelihood.Source: §2 · p. 3
  • Curriculum learning structures training data into difficulty buckets and defines sampling distributions that can change over training.Fixed curricula move gradually from easy to hard, while self-evolving curricula adjust sampling based on model progress.Source: §2 · p. 3
  • In a multi-armed bandit curriculum, each bucket is an arm; the scheduler samples from a selected bucket and receives a reward such as validation improvement.The sampling probabilities are adapted online, balancing exploration with exploitation.Source: §2 · p. 3

3 Method: Ladders-of-Thought (LoT)

LoT turns reasoning traces into easier rewrites, labels examples by estimated reasoning steps, and uses those labels to build staged or adaptive curricula.

3.1 Progressive Rewrites

  • Each rewrite replaces premises used in a reasoning step with their entailed conclusion, creating easier, semantically faithful versions of the same task rather than distinct subproblems.A capable instruction-tuned LLM identifies the premises and generates each simplified question; the rewriting model may differ from the teacher.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 Difficulty Labeling via Step Definition

  • Difficulty is the model-estimated minimum number of reasoning steps, aligned with rewrite depth rather than potentially verbosity-inflated chain-of-thought length.The authors manually spot-check examples for monotonic decreases in step counts before using them as interpretable difficulty labels.Source: §3.2 · p. 4

3.3 Curriculum Construction

  • Adjacent step-count levels are grouped into difficulty buckets, and the staged baseline trains on one ordered bucket at a time for a fixed number of steps.Example buckets group 1–3, 4–5, and 6+ steps as easy, medium, and hard.Source: §3.3 · pp. 4–5
  • The adaptive scheduler treats each bucket as a bandit arm, rewards validation-accuracy gain over a baseline, and samples buckets with Boltzmann or ϵ-greedy policies.Its bucket focus changes dynamically toward levels with the greatest marginal improvement.Source: §3.3 · p. 5

4 Experiments

Across math and multi-hop reasoning, LoT improves over knowledge distillation, with the strongest math gains on out-of-distribution arithmetic and more variable multi-hop results. Ablations show that moderate rewrite depth and adaptive curriculum scheduling matter, while added compute is limited.

4.1 Setup

  • Experiments test pass@5 accuracy on math and multi-hop reasoning, comparing LoT with base models and CoT knowledge distillation across OPT and Pythia model families.Math training uses GSM8K and multi-hop training uses EntailmentBank; evaluations include their test splits and out-of-distribution benchmarks.Source: §4 · p. 5, §4.1 · p. 5

4 Experiments

  • The experiments examine whether LoT improves reasoning generalization and how rewrite depth and curriculum scheduling affect performance.The evaluation focuses on two questions: gains over baselines across models and domains, and the effects of depth and scheduling choices.Source: §4 · p. 5

4.2 Main Results

  • LoT consistently outperforms CoT KD across four model checkpoints and both domains, with especially large out-of-distribution arithmetic gains and dataset-dependent multi-hop results.For OPT-2.7B, AddSub rises from 8.26 to 40.37 (+32.11 percentage points) and SVAMP from 19.06 to 44.15 (+25.09 percentage points); QASC gains +4–16 and StrategyQA +17–25, while some OpenBookQA and MuSiQue results regress.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7

4.3 Larger Students: Qwen2.5–7B and Llama3.1–8B

  • At 7–8B, LoT matches or slightly improves over KD on GSM8K and yields larger out-of-distribution gains, including AddSub and SVAMP improvements across both models.Qwen2.5-7B improves +20.2 on AddSub and +11.0 on SVAMP; Llama3.1–8B improves +31.2 on AddSub and +10.4 on SVAMP.Source: §4.3 · p. 6

4.4 Ablation: Rewrite Depth

  • Shallow rewrites produce the largest single accuracy jump (+28.48 percentage points), gains continue through depth 3, and using all rewrites can reduce performance.Increasing rewrite depth skews training toward low-step, easier examples; the results favor shallow-to-intermediate ladders that retain a range of difficulty.Source: §4.4 · pp. 6–7, Table 4 · p. 8

4.5 Ablation: Curriculum Scheduling

  • Self-evolving scheduling converges fastest and reaches the highest final accuracy, while Easy→Hard beats flat sampling and Hard→Easy performs worst.Flat sampling learns reasonably early but plateaus lower, whereas the self-evolving curriculum combines early learning efficiency with stronger final performance.Source: §4.5 · p. 8, Figure 3 · p. 9, Figure 4 · p. 9

4.6 Overhead of LoT

  • The online MAB scheduler adds 3–8% of total runtime, while rewrite generation occurs once before training and remaining compute matches standard supervised fine-tuning.The paper characterizes LoT's practical computational overhead as minimal.Source: §4.6 · p. 8

4.7 Discussion

  • The discussion recommends faithful easier rewrites in a moderate-depth ladder with adaptive exposure, while noting that excessive depth can hurt and curriculum direction affects outcomes.The authors describe this recipe as supporting sample efficiency and generalization with minimal computational overhead.Source: §4.7 · pp. 8–9

5 Related Works

LoT combines progressive rewrites, step-based difficulty estimation, and adaptive scheduling for supervised reasoning fine-tuning. It differs from prior work by simplifying the original question into difficulty-aligned variants rather than adding auxiliary tasks or refining generated outputs.

Question Decomposition

  • Unlike decomposition methods such as Least-to-Most Prompting (Zhou et al., 2022) and Self-Ask (Press et al., 2023), LoT simplifies the same question without adding auxiliary tasks.It replaces antecedent premises with entailed intermediate conclusions, preserving semantic equivalence while tying difficulty to minimal reasoning depth.Source: §5 · p. 9

Rationale Refinement and Process-Supervision

Curriculum Learning

  • Prior curriculum work spans in-context learning and reinforcement learning, while Chen et al. (2025) study self-evolving curricula in RL rather than supervised fine-tuning.LoT’s setting is supervised fine-tuning of reasoning.Source: §5 · p. 10

Difficulty Estimation

  • LoT measures difficulty by minimal reasoning steps, aligning the measure with progressive rewrites rather than noisy proxies such as raw chain-of-thought length.Prior measures include MCTS heuristics, dataset difficulty labels, and model hit rates.Source: §5 · p. 10

Positioning

  • LoT integrates progressive rewrites, step-based difficulty estimation, and adaptive scheduling into a curriculum for small- to mid-scale LLMs.The positioning distinguishes this supervised fine-tuning setting from work focused on large models, heuristic difficulty proxies, reinforcement learning, or in-context learning.Source: §5 · p. 10

6 Conclusion

LoT combines progressive rewrites with a self-evolving curriculum to improve reasoning in small- to mid-scale LLMs. Across math and multi-hop reasoning, it outperforms strong distillation and curriculum baselines, with gains varying by task and faster convergence than flat or staged curricula.

6 Conclusion

  • LoT combines progressive rewrites with a self-evolving curriculum to improve reasoning in small- to mid-scale LLMs.The conclusion describes the training signals as semantically faithful rewrites organized into adaptive curricula.Source: §6 · p. 10
  • +32 percentage points on AddSub, +25pp on SVAMP, and +25pp on StrategyQA accompany LoT’s gains over strong knowledge-distillation and curriculum baselines.The conclusion also reports modest but robust in-domain improvements on GSM8K and EntailmentBank, and dataset-dependent multi-hop benefits.Source: §6 · p. 10
  • LoT converges faster than flat or staged curricula, balancing training efficiency with final performance.The conclusion presents adaptivity as valuable in balancing these outcomes.Source: §6 · p. 10

Limitations

The study’s evidence is bounded by its model scale, rewrite generation, and evaluation settings. Multi-hop benefits also vary by dataset.

Limitations

  • The study evaluates 1–8B models, so scalability to larger foundation models remains untested.Source: Limitations · p. 10
  • LoT depends on a capable generator; low-quality or unfaithful rewrites may add noise, and the fidelity–diversity balance remains unexplored.Source: Limitations · p. 10
  • Evaluation is limited to English math and text-only multi-hop benchmarks, leaving multilingual, multimodal, and interactive domains for future study.Source: Limitations · p. 10
  • Mixed results on some multi-hop tasks indicate that benefits are dataset-dependent, leaving open which reasoning settings gain most.Source: Limitations · p. 10

Reproducibility Statement

The authors report that datasets, model architectures, and training and evaluation details are available or specified, with scripts and rewrite datasets planned for release.

Reproducibility Statement

  • All datasets used in the work are publicly available.Source: Reproducibility Statement · p. 11
  • Preprocessing, rewrite generation, and filtering rules are detailed in Appendix A.3, while training hyperparameters, curriculum schedules, and evaluation settings are specified in Section 4 and Appendix A.3.Source: Reproducibility Statement · p. 11
  • OPT and Pythia architectures are open-source, and the authors say they will release training scripts, the scheduler implementation, and rewrite datasets.Source: Reproducibility Statement · p. 11

Impact Statement

The work aims to strengthen smaller models’ reasoning, with potential benefits for cost, sustainability, and access, while noting possible misuse risks.

Impact Statement

  • Strengthening small and mid-scale models may reduce energy consumption and computational cost compared with relying on very large models.The stated aim also supports more sustainable deployment.Source: Impact Statement · p. 11
  • Stronger open or locally deployable models may reduce dependence on massive proprietary systems and broaden access to advanced reasoning.Source: Impact Statement · p. 11
  • Improved reasoning could be misused for harmful planning or deception, motivating responsible deployment and complementary safety measures.Source: Impact Statement · p. 11

A.1 Algorithms

LoT combines progressive rewrites, difficulty-based buckets, and adaptive training; balanced validation informs the scheduler’s updates.

A.1 Algorithms

  • Training examples are rewritten into simpler variants that preserve the original solution, grouped into difficulty buckets, and sampled for student fine-tuning.The student is optimized with a chain-of-thought loss.Source: §A.1 · p. 17
  • The scheduler uses bucket Q-values reflecting recent accuracy improvements to allocate batches, balancing learning progress with exploration.It selects buckets using Boltzmann exploration or an ϵ-greedy policy and updates Q-values and running accuracy baselines from validation rewards.Source: §A.1 · p. 17
  • Balanced validation samples equally from each bucket and returns per-bucket accuracies for scheduler reward and learning-signal updates.Source: §A.1 · p. 17

A.2 Dataset Statistics

This appendix documents rewrite quality, step-based bucketing, verification and evaluation procedures, additional analyses, and progressive-rewrite prompt designs for both reasoning domains. The GSM8K prompt describes progressively simplified problem versions that insert intermediate results to reduce required reasoning steps at each level.

A.2 Dataset Statistics

  • In a 500-rewrite audit, 99.2% were valid and solvable, 98.0% showed decreased difficulty, and 98.6% preserved the original answer.The audit assessed question validity, difficulty decrease, and answer preservation.Source: §A.2 · p. 19

A.3.3 Bucketing by Step Count

  • Step-count buckets separate 0-, 1-, 2-, and 3-step questions while grouping examples requiring 4+ steps to retain short-chain granularity and avoid fragmenting sparse long-tail examples.Curriculum schedules and sampling strategies are applied over these buckets.Source: §A.3.3 · p. 20

A.3.4 Answer Verification Procedure

  • The validation loop uses a four-stage verification process for free-form reasoning outputs; for arithmetic datasets, math_verify compares numeric expressions and treats mathematically equivalent forms as correct.Multiple-choice benchmarks are converted to free-form generation and evaluated against the correct-option text using the verification pipeline; contextual tasks use sentences marked relevant by dataset annotations. Results report pass@5 accuracy and standard errors across evaluation runs.Source: §A.3.4 · p. 20, §A.3.5 · pp. 20–21

A.4 Additional Experiment Results

  • The appendix reports pass@5 benchmark accuracies for math and multi-hop reasoning and greedy-decoding pass@1 accuracies in Tables 7–10.The tables cover GSM8K and EntailmentBank test splits plus four math and four multi-hop benchmarks; supplied passages do not provide their cell values.Source: §A.4 · pp. 21–22, Table 7 · p. 21, Table 8 · p. 21

A.5 Rewrite Depth Shifts the Difficulty Distribution.

  • 0–1-step questions rise from 0.1% at rewrite depth 0 to 45.1% at depth 4+, while 6+ step examples become increasingly rare; moderate depth increases exposure to easier patterns without collapsing the full difficulty range.Table 13 reports the distribution of minimal reasoning steps by maximum rewrite depth.Source: §A.5 · pp. 22–23, Table 13 · p. 23

A.6 Ablation: Empirical Difficulty vs. Step-Based Difficulty.

  • Lower minimal-step counts correlate with higher model success, but a curriculum bucketed by Qwen2.5-7B empirical success rates performs markedly worse than LoT.Empirical buckets span success ≥0.8 through success <0.2; the authors suggest this measure mixes structural difficulty with surface-level artifacts, unlike step-based buckets that target inferential depth.Source: §A.6 · pp. 23–24

A.7 Ablation: Effect of Rewriter Model Scale

  • LoT performance depends on rewrite quality: Qwen rewriters used as both rewriter and teacher generally lag GPT-5-mini, and fixing GPT-5-mini as teacher helps only the Qwen-7B rewriter, leaves Qwen-14B comparable, and Qwen-72B degrades significantly.The findings indicate rewriter quality has a stronger downstream influence than teacher strength when rewrites contain semantic inconsistencies; LoT still outperforms standard CoT distillation with a weaker rewriter.Source: §A.7 · pp. 25–26, Table 17 · p. 26

A.8 Progressive Rewrite Prompts

  • The EntailmentBank and GSM8K prompts create progressively easier versions by moving intermediate conclusions or results into the input and reporting monotonically non-increasing minimum reasoning-step counts.Both prompts define step-count rules, including zero steps when the answer is stated in the input; the EntailmentBank prompt also directs the model to stop once the problem is trivial.Source: §A.8.1 · p. 26, §A.8.2 · p. 27

A.8.2 GSM8K Prompt

  • The prompt starts with the original problem and generates easier versions by inserting intermediate results from its worked solution into the problem statement.The number of versions is chosen automatically based on the length of the reasoning chain.Source: §A.8.2 · p. 27
  • Each version must include the modified question, final numeric answer, reasoning, minimum required steps, and a note explaining the step count.Source: §A.8.2 · pp. 27–28
  • The goal is a set of GSM8K problems that require fewer reasoning steps at each successive level.Source: §A.8.2 · p. 28

Related papers

Loading 2609.25643v1…