PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning

Zhipeng Chen, Xiaobo Qin, Wayne Xin Zhao, Youbin Wu, Ji-Rong Wen

arXiv:2602.00759v1cs.CLcs.AI

TL;DR

RLVR’s limited information can leave models exploring blindly and failing on challenging problems. A2D trains a same-model decomposer to generate sub-questions that guide a reasoner during RLVR, and reports improved performance with guidance that supports exploration while finer prompts better support exploitation.

  • Problem

    RLVR provides limited information beyond outcome-level rewards, leaving models in largely blind exploration and limiting continued capability growth.

  • Method

    A2D trains a decomposer via RLVR to generate sub-questions, then uses them as guidance for a reasoner trained under RLVR without relying on external models.

  • Results

    A2D achieves better performance across mathematical tasks and RLVR algorithms, with coarse-grained hints aiding exploration and fine-grained prompts aiding exploitation.

  • Takeaways & Limitations

    Sub-question guidance provides additional information for RLVR exploration, while guidance granularity corresponds to different exploration and exploitation benefits.

  • Takeaways & Limitations

    The compatibility between decomposer and reasoner, and extension to online settings, remain future research questions.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has shown great potential to enhance the reasoning ability of large language models (LLMs). However, due to the limited amount of information provided during the RLVR process, the model can only engage in largely blind exploration, which often results in failure on challenging problems. To provide additional information for the RLVR process without relying on a teacher model, we propose A$^2$D, an Adaptive Ability Decomposing method for enhancing the effectiveness of RLVR. Specifically, we first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions. Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance. To better understand A$^2$D, we first compare its performance with competitive baselines, showing its effectiveness. Next, we observe that our method functions as a plug-and-play module that can be applied to different RLVR algorithms. Furthermore, we conduct an analysis of the decomposer, revealing how the RLVR process affects its performance and behavior, and which type of guidance is better suited for enhancing the reasoner's exploration and exploitation abilities.

1 Introduction

RLVR can improve LLM reasoning but offers limited information for exploration, leaving models vulnerable to blind failure on challenging problems. A2D addresses this by training a decomposer without external models to provide sub-question guidance, improving exploration across RLVR algorithms and revealing different benefits from coarse versus fine-grained hints.

  • Motivation: RLVR struggles to raise the upper bound of model performance because outcome-level supervision provides limited guidance for effective exploration.This can lead to failed exploration, punishment through RLVR, conservative reasoning strategies, and reduced further exploration.
  • Motivation: Prior approaches add information through distilled or manually annotated data, but teacher-generated guidance incurs higher costs and can limit students to the teacher’s performance ceiling.These methods remain within an imitation-learning paradigm, making it difficult for the student model to surpass the teacher’s upper bound.
  • Method: A2D trains a decomposer with RLVR to split complex questions into simpler sub-questions, then uses those sub-questions as additional information for training a reasoner.The decomposer and reasoner start from the same model, avoiding reliance on external models.
  • Method: A2D uses in-context distillation loss to learn from experiences obtained through sub-question-guided exploration and operates as a plug-and-play component across RLVR algorithms.This distinguishes the method from approaches that simply insert prompt-generated answers into the training dataset.
  • Results: A2D improves performance across eight mathematical tasks, while sub-question guidance enables more effective exploration across various RLVR algorithms.The paper also reports that specific, fine-grained hints improve Pass@1, whereas abstract, coarse-grained hints improve Pass@k.

2 Approach

A2D trains a decomposer with RLVR, uses its sub-question annotations as guidance, and integrates in-context distillation into reasoner RLVR training. The approach combines quality and format rewards for decomposition with selective guidance for difficult rollouts.

  • 2.2 Training Decomposer via RLVR: A2D first trains a decomposer through RLVR to split complex questions into simpler sub-questions for reasoner guidance.The decomposer is trained without SFT data annotation, using question–answer training instances and self-exploration.
  • 2 Approach: The approach is presented as a comparison between vanilla RLVR and A2D, with decomposer training preceding annotation and guided reasoner training.The pipeline is summarized through Figure 2 and Algorithm 1.
  • 2.2 Training Decomposer via RLVR: The decomposer receives a quality reward when a proxy reasoner solves the original question and a format reward for correctly structured sub-questions.The final decomposer reward is R = R_Q × R_F, encouraging both valid formatting and useful assistance while reducing reward hacking.
  • 2.3 Improving RLVR Effectiveness with Sub-question Guidance: The trained decomposer annotates each training question with sub-questions, forming triples that pair the original question and answer with potential solution steps.These sub-questions are treated as ideas for solving the original problem sequentially.
  • 2.3 Improving RLVR Effectiveness with Sub-question Guidance: A2D integrates in-context distillation loss into RLVR so the reasoner learns from positively rewarded hinted solutions while still generating directly from questions at test time.The method aims to internalize problem-solving strategies because sub-question hints are unavailable during real testing.
  • 2.3 Improving RLVR Effectiveness with Sub-question Guidance: The guidance loss targets questions whose rollout average reward falls below a threshold, and its weight is balanced against the RLVR objective by α.The selected positive responses are used for the auxiliary loss, with q = min(N_pos, k2 × n_rollout).

3 Experiments

The experiments evaluate A2D across eight mathematical benchmarks using SFT- and RLVR-based baselines. The setup uses Qwen2.5-7B-Instruct with GRPO as the backbone algorithm throughout training.

  • Experimental Overview: The experimental section evaluates effectiveness first and then analyzes the features of A2D.The paper organizes the experiments into settings, evaluation results, and feature analysis.
  • Datasets: The evaluation covers eight mathematical benchmarks, including AIME24, AIME25, AMC23, BeyondAIME, MATH500, Minerva, OlymMATH-Easy, and OlymMATH-Hard.The same prior-work dataset is used to train both the decomposer and reasoner.
  • Baselines: The baselines include SFT with chain-of-thought, SFT with chain-of-thought and sub-questions, and RLVR methods LUFFY and Scaf-GRPO.The RLVR baselines use external teacher-model guidance, motivating a fair comparison with A2D’s internally trained decomposer.
  • Implementation Details: The implementation uses Qwen2.5-7B-Instruct as the backbone and GRPO as the RLVR algorithm for the whole training process.Each question receives 32 rollout attempts during RLVR training.

3.2 Main Results

A2D consistently improves mathematical reasoning across backbone models and surpasses SFT- and RLVR-based baselines. However, directly training on decomposer-generated sub-questions does not improve performance and can degrade it in some settings.

  • A2D consistently yields substantial improvements across different backbone models and achieves superior performance on mathematical reasoning tasks.
  • The method surpasses both SFT-based and RLVR-based baselines, including GRPO, LUFFY, and Scaf-GRPO.
  • A2D remains effective across models from different families, scales, and capability levels.
  • Adding decomposer-generated sub-questions directly to SFT training does not improve performance and can degrade results on tasks such as AIME25 and AMC23.The paper attributes this to decomposition and solving being dissimilar abilities, and to sub-questions overlooking cases in incomplete reasoning processes.
  • Direct training on teacher-generated solutions without thinking processes is unlikely to improve reasoning and may impair it through overfitting.LUFFY partially alleviates this issue by adding distilled data only when the model struggles, but requires an external, capable teacher model.

3.3 Detailed Analysis

Detailed analyses show that A2D benefits from reward design, selective and diverse guidance, and compatibility with multiple RLVR algorithms. Its decomposer produces concise, coarse-grained hints that support exploration, while finer-grained hints better support exploitation.

  • Ablation Study: Format Reward trains the decomposer to produce formal subproblem lists, while Pass@k Reward encourages sub-questions that guide the reasoner toward correct answers.Using both rewards improves the decomposer’s assistance to subsequent RLVR training.
  • Ablation Study: Removing either Selection or Diversity Prompt reduces performance in RLVR with sub-question guidance.Selection avoids unnecessary guidance on problems the model can already solve, while diverse prompts mitigate external-knowledge influence when guidance is needed.
  • Compatibility with RLVR Algorithms: A2D further improves traditional RLVR algorithms on downstream tasks, including challenging AIME25 and BeyondAIME problems where vanilla RLVR struggles.The paper attributes vanilla RLVR’s difficulty to limited model capability for discovering correct solutions through self-exploration.
  • Effect of Prompt Guidance: Adding decomposer-generated sub-questions to prompts improves both Pass@1 and Pass@k scores.This supports using decomposition to guide more effective exploration and reasoning.
  • Effect of Prompt Guidance: GRPO + Prompt w/ SQ performs better during the first 100 training steps, but later reliance on hints limits its advantage over GRPO + A2D.The early benefit comes from hints directly providing solution strategies.
  • Decomposer Analysis: The decomposer generates about two sub-questions per question using approximately 60 tokens on average.These statistics indicate a relatively high-level, coarse-grained reasoning process.
  • Decomposer Analysis: Generated sub-questions avoid final-answer keywords, providing guidance without revealing solutions or fully constraining exploration.The decomposer therefore offers guidance and inspiration while retaining the reasoner’s exploratory capability.
  • Case Study: Coarse-grained hints improve Pass@k, whereas fine-grained hints better improve Pass@1 by supporting exploitation.The case study contrasts high-level conceptual decomposition with step-by-step reasoning solutions.

4 Related Work

The related work covers score-based RLVR, natural-language guidance, and compositional generalization as routes to stronger reasoning. A2D focuses on using decomposed subproblems to help models combine simple knowledge for complex tasks during RLVR.

  • Improving Exploration Ability of LLMs via Value-based Guidance: RLVR uses reward scores, reward models, advantage shaping, and sampling mechanisms to balance exploration and exploitation during reasoning.
  • Compositional Generalization Ability of LLMs: A2D investigates whether subproblems can help LLMs transfer subproblem-solving ability to complex questions and improve exploration efficiency in RLVR.
  • Test-time Scaling via Natural Language Guidance: Natural-language guidance includes teacher responses, self-reflection, supervised fine-tuning, and parallel reasoning to support exploration and reasoning performance.
  • Compositional Generalization Ability of LLMs: Compositional generalization is the ability to combine known simple knowledge to solve complex problems, but LLMs remain limited in this ability.

5 Conclusion

A2D trains a decomposer with RLVR to generate subquestions that guide a reasoner during RLVR. The analysis finds that guidance granularity relates differently to exploration and exploitation, while decomposer–reasoner compatibility and online extension remain future directions.

  • A2D trains a decomposer to split complex questions into subquestions, then uses those subquestions to assist the reasoner’s RLVR training.
  • Additional information from decomposed subquestions enables the reasoner to explore more effectively during RLVR.
  • Abstract, coarse-grained hints such as subquestion guidance benefit exploration, whereas concrete, fine-grained prompts better support exploitation.
  • Future work should examine compatibility between decomposers and reasoners, evolving hint style and granularity, and extending the algorithm to online settings.

A Prompt Templates

The prompt templates define separate formats for decomposer training, vanilla reasoning, subquestion-guided reasoning, and IDN Loss training. They specify placeholders, reasoning instructions, answer formatting, and subquestion tags.

  • Decomposer Prompt: The decomposer prompt replaces {QUESTION} with each question and requests step-by-step decomposition into tagged subquestions without detailed reasoning or solutions.
  • Vanilla Reasoner Prompt: The vanilla reasoner prompt replaces {QUESTION} with the question and requests step-by-step reasoning with the final answer inside boxed formatting.
  • Sub-question Guidance Prompt: The guided reasoner prompt supplies the original question and decomposed subquestions as tips for solving the problem.
  • IDN Loss Prompt: The IDN Loss prompt asks the reasoner to explore different approaches step by step and place the final answer inside boxed formatting.

B Details of the Format Reward for Decomposer Training

The format reward checks whether decomposer outputs follow required structural constraints. Valid responses must begin with the subquestion tag, avoid repeated tags within a subquestion, and exceed ten characters.

  • Generated decomposer responses must begin with the “<subquestion>” tag.
  • Each subquestion may contain no more than one “<subquestion>” tag.
  • The generated response must contain more than 10 characters.

C Pseudo Code of A2D

A2D trains a decomposer through RLVR, uses it to annotate training questions with simpler sub-questions, and then trains a reasoner with sub-question guidance under RLVR.

  • Training Decomposer: A2D takes a backbone model and dataset of question–answer pairs as input, producing a trained reasoner.The pipeline initializes both the decomposer and proxy reasoner from the backbone model.
  • Training Decomposer: The decomposer generates sub-questions for training examples, which a proxy reasoner evaluates with a quality reward.A format reward is also computed, and the decomposer is optimized using their product.
  • Decompose the Complex Questions: The optimized decomposer converts each complex question into several simpler sub-questions.These sub-questions are collected as S_i for each training question x_i.
  • Training Reasoner: The resulting triples ⟨x_i, y_i, S_i⟩ form the reasoner’s training dataset for RLVR with sub-question guidance.The reasoner is initialized from the backbone model before guided training begins.
  • Training Reasoner: The reasoner generates solutions both from the original question and from the question paired with sub-questions.The unguided outputs are used for the RLVR loss, while guided outputs are processed for an in-context distillation loss.
  • Training Reasoner: A2D optimizes the reasoner using the RLVR objective together with the in-context distillation loss, yielding the final reasoner.The pseudocode combines J(θ_R) and L′_IDL(θ_R) in the optimization step.
Loading 2602.00759v1…