PaperlayerView exact revision on arXiv

Source-linked AI summary

LADDER: Self-Improving LLMs Through Recursive Problem Decomposition

Toby Simonds, Akira Yoshiyama

arXiv:2503.00735v3cs.LGcs.AI

TL;DR

Reinforcement learning depends on verifiable tasks matched to a model’s abilities, a challenge for complex reasoning. LADDER recursively generates simpler variants for learning, and TTRL applies reinforcement learning to variants of test problems. The paper reports improved integration results, including 90% on the MIT Integration Bee with TTRL.

  • Problem

    Reinforcement learning needs verifiable tasks matched to a model’s current capabilities, but obtaining such tasks is challenging, especially for complex reasoning.

    Source: §1 · p. 1

  • Method

    LADDER recursively generates and solves simpler variants, while TTRL applies reinforcement learning to variants of each test problem at inference time.

    Source: §1 · pp. 1–2, §3.1.5 · p. 6

  • Results

    90% on the MIT Integration Bee with TTRL, alongside LADDER improvements from 1% to 82% on undergraduate integration problems and from 50% to 73% on the exam.

    Source: §5 · p. 10, §6 · p. 12

  • Takeaways & Limitations

    The results show that self-directed learning can improve mathematical reasoning without architectural scaling or human supervision.

    Source: §6 · p. 12

    The comparison with o1 is a general baseline, not a strict head-to-head evaluation, because o1 lacks access to a numerical checker.

    Source: §4.3 · p. 10

Abstract

from arXiv · show

We introduce LADDER (Learning through Autonomous Difficulty-Driven Example Recursion), a framework which enables Large Language Models to autonomously improve their problem-solving capabilities through self-guided learning by recursively generating and solving progressively simpler variants of complex problems. Unlike prior approaches that require curated datasets or human feedback, LADDER leverages a model's own capabilities to generate easier question variants. We demonstrate LADDER's effectiveness in the subject of mathematical integration, improving Llama 3.2 3B's accuracy from 1% to 82% on undergraduate-level problems and enabling Qwen2.5 7B Deepseek-R1 Distilled to achieve 73% on the MIT Integration Bee qualifying examination. We also introduce TTRL (Test-Time Reinforcement Learning), where we perform reinforcement learning on variants of test problems at inference time. TTRL enables Qwen2.5 7B Deepseek-R1 Distilled to achieve a state-of-the-art score of 90% on the MIT Integration Bee qualifying examination, surpassing OpenAI o1's performance. These results show how self-directed strategic learning can achieve significant capability improvements without relying on architectural scaling or human supervision.

1 Introduction

LADDER addresses the difficulty of obtaining verifiable reinforcement-learning tasks matched to a model’s capabilities by recursively generating simpler problem variants. The introduction reports improved integration performance with LADDER and further gains from test-time reinforcement learning.

1 Introduction

  • Reinforcement learning needs verifiable tasks matched to a model’s current capabilities; tasks that exceed them can stall training or cause catastrophic collapse.The challenge is especially acute in complex-reasoning domains with substantial gaps between simple and advanced tasks.Source: §1 · p. 1
  • LADDER recursively generates and solves progressively simpler variants, creating a difficulty gradient that supplies stepping stones toward harder problems.The framework uses reinforcement learning with GRPO and systematic generation and verification of variants.Source: §1 · pp. 1–2
  • The framework is presented as improving problem solving beyond standard pass@k sampling, including on problems previously beyond a model’s capabilities.The introduction characterizes the approach as self-bootstrapping through generated variants and their solutions.Source: §1 · p. 1
  • LADDER uses the model’s existing capabilities and verifiable rewards rather than curated datasets or human feedback, with numerical integration checking solutions.The model can assess its progress and guide its learning trajectory without human intervention.Source: §1 · p. 1
  • 90% on the MIT Integration Bee with TTRL, up from LADDER’s 73%, accompanies Llama 3B’s rise from 1% to 82% on undergraduate integration problems.The introduction also reports that LADDER’s 7B model achieved 73% on the exam, compared with GPT-4o at 42% and typical human performance of 15–30%.Source: §1 · p. 2
  • TTRL extends LADDER to inference time by generating variants of each test problem and applying reinforcement learning to refine the model’s solution.The introduction describes this as scaling test-time compute through strategic problem decomposition.Source: §1 · p. 2

2 Related Work

Related work covers self-play and self-generated training examples, inference-time reasoning and search, and reinforcement-learning approaches to self-improvement. LADDER is positioned as adding explicit difficulty progression and learning from targeted variants at test time.

2 Related Work

  • Haluptzok et al. (2023) use a code-focused model to invent and solve programming puzzles, verify solutions with Python, and fine-tune on correct results.LADDER builds on that generate–solve–verify–learn cycle with a more explicit curriculum for step-by-step improvement.Source: §2 · p. 2
  • STaR learns from model-generated chains of thought, while this work transforms unsolvable problems into tractable subproblems without human annotation or intervention.The passage characterizes prior methods as automatically curating examples that progressively reduce task complexity.Source: §2 · p. 2
  • Prior inference-time methods allocate computation to verification, reasoning paths, longer outputs, or solution-space exploration; TTRL instead learns through practice on related problems.The paper contrasts static inference-time techniques with dynamic adaptation through targeted learning.Source: §2 · p. 2
  • The related-work discussion describes o1 refining chain-of-thought reasoning through reinforcement learning and DeepSeek R1 generating and self-evaluating training data with minimized human supervision.These are presented as examples of reinforcement learning for language-model self-improvement.Source: §2 · pp. 2–3
  • Cited work reports stronger RL generalization than supervised fine-tuning and out-of-distribution generalization in competitive mathematics.The passage says these findings suggest reinforcement learning offers a promising path to self-improvement compared with supervised fine-tuning.Source: §2 · p. 3
  • The section frames these approaches as a shift from externally curated curricula and human-labeled data toward autonomous, feedback-driven self-improvement.Source: §2 · p. 3

3.1 Algorithm Design (LADDER & TTRL)

LADDER trains on recursively generated integral variants, while TTRL applies a focused variant-based reinforcement-learning step to each test question. The design combines variant generation, numerical verification, and reinforcement learning.

3.1.1 LADDER

  • LADDER generates variant trees for training integrals, then trains a base model on those variants with reinforcement learning before answering new integrals.The number of variants per question is an experimental hyperparameter.Source: §3.1.1 · p. 3

3.1.2 Variant Generation

  • Variant generation uses sampled transformations, batches of 10, varied temperatures, and rotating mathematical personas, then recursively builds simpler variants to a depth of three.The authors found these diversity techniques addressed repetitive outputs; naive requests for easier variants often produced repetitive, insufficiently diverse results.Source: §3.1.2 · p. 4

3.1.3 Solution Verification

  • Verification compares candidate antiderivatives with numerical integration at five sampled points using intervals of length 0.1, with adaptive quadrature and checks for singularities, precision, timeouts, and degenerate answers.The numerical verifier may not be fully accurate; final MIT benchmark results are checked against official solutions.Source: §3.1.3 · p. 5

3.1.4 Reinforcement Learning Protocol

  • GRPO estimates its baseline from group scores without a separate critic, while rule-based accuracy and answer-format rewards guide training.The accuracy reward uses the numerical verifier; the format reward encourages answers in <ANSWER> tags.Source: §3.1.4 · pp. 5–6

3.1.5 Test-Time Reinforcement Learning

  • TTRL generates a variant tree for each test question, fine-tunes a base model on that question’s variants, answers with the tuned model, then restores its original parameters.The number of variants per test question is varied across experiments.Source: §3.1.5 · p. 6

3.2 Experiments

The experiments test whether recursively generated variants support reinforcement learning on integration problems, first with Llama 3.2 3B and then on the MIT Integration Bee. TTRL extends the approach by training on variants of individual test questions.

3.2.1 Llama 3B Experiments

  • The Llama 3.2 3B experiment uses 110 indefinite integrals calibrated as solvable by GPT-4o but beyond the base model’s capabilities, split into 10 training and 100 test problems.Approximately 500 variants were generated per training problem, with only five exact matches among 5,000 variants and the test set.Source: §3.2.1 · p. 6
  • The Llama experiments compare reinforcement learning on the 10 original training questions with learning on variants generated from those questions, using identical hyperparameters until performance plateaued.Solutions were expressed in SymPy algebraic notation without integration constants to standardize verification.Source: §3.2.1 · p. 7

3.2.2 MIT Integration Bee (LADDER)

  • The MIT Integration Bee experiments use Qwen 2.5 7B and 9,000 variants from 2010–2024 qualifying exams, with recursion capped at two levels.Third-level variants were capped because preliminary experiments found them trivially simple and less relevant to the original problems.Source: §3.2.2 · p. 7
  • For the MIT exam variants, the authors settled on a 70% easier and 30% equivalent difficulty mix after finding that a skew toward easier problems created more effective learning trajectories.Some equivalent variants introduced novel mathematical patterns while retaining similar difficulty.Source: §3.2.2 · p. 7

3.2.3 MIT Integration Bee (LADDER + TTRL)

  • TTRL applies 100 reinforcement-learning steps to roughly 800 two-level variants of each problem LADDER leaves unsolved, counting a problem as solved if any solution is verified correct.The variants are generated for the test question itself, creating a problem-specific learning process.Source: §3.2.3 · pp. 7–8

4 Results

LADDER and TTRL improve integration performance on the Llama 3B test set and the MIT Integration Bee. TTRL further raises the LADDER-trained 7B model’s score, while the paper cautions that its comparison with o1 is not head-to-head.

4.1 Llama 3B Experiments

  • 82% test accuracy with LADDER exceeds base pass@1 at 1% and pass@10 at 2%; RL without variants peaked below 3% before collapsing to 0% within 30 steps.The training curve rises rapidly and plateaus around 82%.Source: Figure 4 · p. 8, §4.1 · p. 8
  • The remaining unsolved integrals involve combinations of techniques and novel substitutions, indicating difficulty synthesizing methods for more intricate problems.The paper notes that variant generation teaches individual techniques, while their synthesis remains a challenge.Source: §4.1 · p. 8
  • Performance continued rising as variant counts increased, with improvement slowing beyond 6,000 variants but no definitive plateau and diminishing returns relative to earlier scaling.The authors describe a trade-off between computational cost and incremental gains.Source: §4.1 · pp. 8–9

4.2 MIT integration Bee (LADDER)

  • On the 2025 MIT Integration Bee qualifying examination, LADDER raised Deepseek-R1 Qwen 2.5 7B base accuracy from 50% to 73%, outperforming GPT-4o (42%) and meeting the 14/20 qualification threshold.LADDER’s 73% accuracy remained below o1’s 80%.Source: §4.2 · p. 9

4.3 MIT Integration Bee (LADDER + TTRL)

  • 90% accuracy follows TTRL’s 100 steps on problems LADDER missed, up from 73%; the paper reports 3–4 additional problems solved.Figure 6 compares TTRL with LADDER pass@100; the authors report TTRL outperforming OpenAI’s o1 at pass@1, while noting their comparison is not head-to-head.Source: §4.3 · pp. 9–10, Figure 6 · p. 10
  • TTRL on the pre-LADDER base model solved none of the questions that LADDER+TTRL solved; gains occurred only when TTRL followed LADDER training.The authors suggest LADDER exposes the model to the integral distribution needed for later test-time learning.Source: §4.3 · p. 10

5 Discussion

The discussion presents recursive, verifier-guided learning as a route to stronger mathematical reasoning, including further gains from learning at test time. It considers parallel test-time exploration and possible extensions beyond integration, while identifying variant difficulty and curriculum sequencing as open challenges.

5 Discussion

  • 90% on MIT Integration Bee with TTRL, versus 73% after LADDER; LADDER also raises undergraduate integration results from 1% to 82%.The discussion also reports LADDER’s MIT Integration Bee result rising from 50% to 73%, without human feedback or model scaling.Source: §5 · p. 10
  • LADDER decomposes beyond-capability problems into simpler variants, guiding self-directed learning through formal verification rather than human oversight.The authors frame recursive decomposition and solution verification as a way for models to direct their learning trajectory.Source: §5 · pp. 10–11

5.1 A Form of Test-Time Compute Scaling

  • TTRL offers an alternative to longer sequential outputs: independent variants could be explored across compute units, although the current implementation explored them sequentially.The discussion identifies sequential token generation as a bottleneck for difficult problems and describes distributed variant exploration as a potential speedup.Source: §5.1 · p. 11

5.2 Extension to Other Verifiable Domains

  • Beyond integration, the authors propose applying verified variant learning to domains with a generator-verifier gap, including unit-tested programming, Lean-checked proofs, and planning tasks.For planning, they suggest building from simpler tasks that models successfully execute.Source: §5.2 · pp. 11–12

5.3 Future Work

  • LADDER’s variant difficulty and curriculum sequencing remain calibration challenges; the current approach uses static parameters, while adaptive difficulty is proposed for future work.The authors identify redundant or inappropriately difficult examples, and the balance between exploring novel patterns and exploiting known strategies, as optimization concerns.Source: §5.3 · p. 12

6 Conclusion

The conclusion presents LADDER and its test-time extension, TTRL, as approaches to improving problem-solving through recursive decomposition and verified reinforcement learning. It reports substantial integration results and suggests that the principles could extend to other domains with clear verification mechanisms.

6 Conclusion

  • 1% to 82% on undergraduate integration problems and 90% accuracy on MIT Integration Bee through TTRL are the conclusion’s headline results.The conclusion attributes these results to recursive problem decomposition and reinforcement learning with verifiable rewards, without architectural scaling or human supervision.Source: §6 · p. 12
  • The conclusion highlights recursive decomposition, verifiable rewards, and strategic test-time practice as insights for AI development beyond integration.It argues that how models interact with their environment matters alongside architectural innovations.Source: §6 · p. 12
  • The authors suggest the approach could extend to domains with clear verification mechanisms, including program synthesis and formal theorem proving.They frame carefully constructed learning trajectories as a way for models to bootstrap capabilities in increasingly complex domains.Source: §6 · p. 12

Appendix: MIT Integration Bee Failed Questions

Two questions on the 2025 MIT Integration Bee qualifying exam remained unanswered correctly after LADDER and TTRL; the paper describes them as among the exam’s most complex.

Appendix: MIT Integration Bee Failed Questions

  • Two 2025 MIT Integration Bee qualifying questions remained incorrect after LADDER and TTRL, and were among the exam’s most complex and very difficult for undergraduates.The passage does not identify the questions in readable text.Source: Appendix: MIT Integration Bee Failed… · p. 13

Related papers

Loading 2503.00735v3…