PaperlayerView exact revision on arXiv

Source-linked AI summary

ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates

Ling Yang, Zhaochen Yu, Bin Cui, Mengdi Wang

arXiv:2502.06772v2cs.CLcs.AIcs.LG

TL;DR

Complex reasoning requires efficient search over reasoning paths, yet existing template and retrieval methods face integration and scalability challenges. ReasonFlux addresses this with a structured library of thought templates, hierarchical reinforcement learning over template trajectories, and adaptive inference-time scaling. It reports state-of-the-art mathematical reasoning performance, including 91.2% on MATH and 56.7% on AIME 2024.

  • Problem

    Existing template-based reasoning methods can struggle with complex problems that require integrating multiple templates, while retrieval scalability also remains challenging.

  • Method

    ReasonFlux combines around 500 structured thought templates with hierarchical reinforcement learning over template trajectories and adaptive inference-time template scaling.

  • Results

    ReasonFlux-32B consistently outperforms frontier and open-source reasoning models on challenging mathematical benchmarks, achieving 91.2% accuracy on MATH and 56.7% on AIME 2024.

  • Takeaways & Limitations

    The results support ReasonFlux as a generalizable and scalable framework for improving complex mathematical reasoning across diverse challenging benchmarks.

Abstract

from arXiv · show

We present that hierarchical LLM reasoning via scaling thought templates can effectively optimize the reasoning search space and outperform the mathematical reasoning capabilities of powerful LLMs like OpenAI o1-preview and DeepSeek V3. We train our ReasonFlux-32B model with only 8 GPUs and introduces three innovations: (i) a structured and generic thought template library, containing around 500 high-level thought templates capable of generalizing to similar or relevant reasoning problems; (ii) performing hierarchical reinforcement learning on a sequence of thought templates instead of long CoTs, optimizing a base LLM to plan out an optimal template trajectory for gradually handling complex problems; (iii) a brand new inference scaling system that enables hierarchical LLM reasoning by adaptively scaling thought templates at inference time. With a template trajectory containing more explainable reasoning structures than DeepSeek-R1 and o3-mini, our ReasonFlux-32B significantly advances math reasoning capabilities to state-of-the-art levels. Notably, on the MATH benchmark, it achieves an accuracy of 91.2% and surpasses o1-preview by 6.7%. On the USA Math Olympiad (AIME) benchmark, ReasonFlux-32B solves an average of 56.7% of problems, surpassing o1-preview and DeepSeek-V3 by 27% and 45%, respectively. Code: https://github.com/Gen-Verse/ReasonFlux

1. Introduction

ReasonFlux addresses complex reasoning by organizing high-level thought templates into trajectories learned with hierarchical reinforcement learning and scaled adaptively at inference time. The framework combines a structured template library with hierarchical planning to simplify reasoning-path search and improve mathematical reasoning.

  • 1. Introduction: ReasonFlux uses hierarchical reinforcement learning to plan optimal, generalizable trajectories of high-level thought templates for input problems.The approach optimizes template sequences rather than long chain-of-thought trajectories.
  • 1. Introduction: The framework is designed to simplify reasoning-path search for complex problems while improving reasoning capabilities over prior approaches.The introduction positions this as the central motivation and contribution of ReasonFlux.
  • 1. Introduction: A structured library contains around 500 compact thought templates for retrieving and adapting relevant reasoning strategies.The templates support detailed reasoning across a series of simpler sub-problems.
  • 1. Introduction: ReasonFlux adaptively retrieves and scales template sequences at inference time, dynamically selecting high-level templates for individual sub-problems.This design targets a better exploration-exploitation trade-off during problem solving.

2. Related Work and Discussions

Prior work explores deliberate search, reward-guided reasoning, retrieval, and inference scaling, but template-based methods face retrieval and scalability challenges for complex problems. ReasonFlux develops structured template retrieval and hierarchical reasoning to address these limitations.

  • 2. Related Work and Discussions: Embedding-similarity retrieval becomes less effective as document size grows, motivating a structured and compact template library for complex reasoning.The proposed library is intended to improve retrieval precision and reduce scalability challenges.
  • 2. Related Work and Discussions: Inference-scaling research studies how additional tokens and strategies such as majority voting or best-of-N affect reasoning performance.This line of work expands or compares reasoning trajectories at inference time.

3. ReasonFlux: Scaling Thought Templates for Hierarchical LLM Reasoning

ReasonFlux builds a structured thought-template library and trains a navigator to configure template trajectories for complex mathematical problems. At inference time, it retrieves and instantiates templates dynamically, with feedback between planning and execution refining the reasoning path.

  • 3.1. Constructing Structured Thought Template Library: The library addresses scalability challenges in template-based retrieval through structured, targeted organization of diverse mathematical reasoning templates.Templates are generated from challenging problems by summarizing solution strategies and identifying common patterns.
  • 3.1. Constructing Structured Thought Template Library: Each template records a name, keyword tags, description, problem scope, application steps, and examples for retrieval and reuse.The library is represented as a collection of templates with structured fields supporting efficient application.
  • 3.1. Constructing Structured Thought Template Library: Template metadata enables quick keyword-based retrieval of relevant templates for particular problem characteristics.Names and tags are used to search the structured library, making relevant templates more readily available.
  • 3.2. Hierarchical Reinforcement Learning on Thought Template Trajectory: Hierarchical reinforcement learning trains a navigator to plan sequences of high-level templates that decompose problems into simpler sub-problems.The process first fine-tunes a base LLM to associate template identifiers with descriptions and scopes, then uses trajectory preferences for further optimization.
  • 3.2. Hierarchical Reinforcement Learning on Thought Template Trajectory: Trajectory quality is evaluated by solving similar problems with instantiated templates, using average accuracy as the reward for preference optimization.Multiple trajectories are sampled for each input, evaluated through their template-trajectory rewards, and used to construct optimization pairs.
  • 3.3. Inference Scaling with Scaling Thought Templates: ReasonFlux combines its optimized navigator, the template library, and an inference LLM in a multi-round system that dynamically scales thought templates.The system uses automatically planned trajectories and dynamically retrieved templates during inference.
  • 3.3. Inference Scaling with Scaling Thought Templates: For each problem, ReasonFlux generates a dynamically tailored trajectory, links its steps to template metadata, retrieves matching templates, and guides the inference LLM.The trajectory is not fixed in advance; its number of retrieved templates matches the configured reasoning steps.
  • 3.3. Inference Scaling with Scaling Thought Templates: Intermediate results feed back into planning, allowing ReasonFlux to adjust the trajectory as reasoning proceeds and constraints or opportunities emerge.The paper frames this as an iterative planning-and-execution interaction for complex problem solving.

4. Experiments

ReasonFlux-32B is evaluated across challenging mathematical benchmarks, template generalization tests, reasoning-flow comparisons, and inference-scaling experiments. It reports strong benchmark performance, broad template transfer, and lower exploration cost than alternative search strategies.

  • Evaluation Setup: The system uses approximately 500 thought templates and evaluates performance across MATH, AIME 2024, AMC 2023, OlympiadBench, and Gaokao.
  • Results on Challenging Reasoning Benchmarks: 56.7% accuracy on AIME 2024 surpasses o1-preview and DeepSeek-V3 by 27% and 45%, respectively.
  • Results on Challenging Reasoning Benchmarks: ReasonFlux-32B achieves 63.3% on OlympiadBench and 83.6% on Gaokao, surpassing DeepSeek-V3 by 14% and o1-mini by 7%, respectively.
  • Generalization Ability of Structured Template Library: Template-augmented reasoning improves different base LLMs over direct reasoning on generated variants, with smaller guided models sometimes outperforming larger models using direct reasoning.
  • Reasoning Flows over Planned Template Trajectory: Reasoning-flow examples show planned template trajectories guiding an inference LLM through interactive, context-conditioned steps.
  • Inference Scaling Laws for Template-Augmented Reasoning: Inference scaling adaptively increases retrieved templates and interplay rounds as problem complexity increases.
  • Better Exploration-Exploitation Trade-off: ReasonFlux maintains lower and more stable exploration cost across difficulty levels than MCTS and Best-of-N.

5. Conclusion

ReasonFlux presents hierarchical reasoning that scales structured thought templates to simplify complex mathematical search and outperform powerful LLMs on challenging math benchmarks.

  • ReasonFlux adaptively scales fundamental thought templates to simplify the search space of complex reasoning.
  • The framework combines a structured compact thought-template library, hierarchical reinforcement learning over template trajectories, and inference-time scaling.
  • ReasonFlux outperforms OpenAI o1-preview and DeepSeek-V3 on challenging MATH and AIME benchmarks.
  • The reported findings include scaling laws for template-augmented reasoning and a superior exploration-exploitation trade-off over previous reasoning strategies.

A. More Examples of Structured Template Library

The structured template library organizes reusable mathematical knowledge and problem-solving procedures into metadata-rich templates, illustrated across inequalities, extremum problems, and trigonometric analysis.

  • Template Structure: Each template contains metadata and content, with metadata describing its name, knowledge tags, description, and application scenarios.
  • Problem-Solving Methods: The absolute-value inequality template uses zero points, interval partitioning, rewriting, interval solving, endpoint verification, and solution-set combination.
  • Problem-Solving Methods: For |x −1| > |2x + 3|, the squaring method produces −(3x + 2)(x + 4) > 0 and solution set (−4, −2 3).
  • Secondary Mathematical Conclusions: The three-variable and n-variable inequality template requires positive variables, applies arithmetic-geometric mean inequalities, transforms expressions, and checks equality conditions.
  • Secondary Mathematical Conclusions: For the example with abc = 1, the equality condition is reported as a = b = c = 1.
  • Property Theorem: The extremum-value template links constant products to minimum sums and constant sums to maximum products, with equality when x = y.
  • Knowledge Application: The trigonometric template uses reduction formulas to standardize functions before analyzing parity and symmetry axes or centers.
Loading 2502.06772v2…