RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems

Yuxiao Qu; Anikait Singh; Yoonho Lee; Amrith Setlur; Ruslan Salakhutdinov; Chelsea Finn; Aviral Kumar

Source-linked AI summary

RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problems

Yuxiao Qu, Anikait Singh, Yoonho Lee, Amrith Setlur, Ruslan Salakhutdinov, Chelsea Finn, Aviral Kumar

arXiv:2510.02263v1Oct 2, 2025cs.AIcs.CLcs.LG

TL;DR

Long-chain reinforcement learning often increases reasoning depth without consistently discovering or reusing procedures, leading to verbose and degenerate exploration. RLAD jointly trains an abstraction generator to propose concise procedural and factual abstractions and a solution generator to use them. RLAD achieves an average 44% improvement over DAPO on AIME 2025, while using reasoning abstractions improves performance by 30% on average over 37 tasks.

Problem
Long-chain reinforcement learning often increases reasoning depth without consistently discovering or reusing procedures, leading to verbose and degenerate exploration.
Method
RLAD jointly trains an abstraction generator to propose concise procedural and factual abstractions and a solution generator to use them.
Results
RLAD achieves an average 44% improvement over DAPO on AIME 2025, while using reasoning abstractions improves performance by 30% on average over 37 tasks.
Takeaways & Limitations
Generating diverse abstractions is a complementary way to scale test-time compute and can yield greater gains than increasing solution sampling alone.
Takeaways & Limitations
A single model trained to both generate and solve with abstractions quickly lost its ability to propose abstractions during RL training.

Abstract

from arXiv · show

Reasoning requires going beyond pattern matching or memorization of solutions to identify and implement "algorithmic procedures" that can be used to deduce answers to hard problems. Doing so requires realizing the most relevant primitives, intermediate results, or shared procedures, and building upon them. While RL post-training on long chains of thought ultimately aims to uncover this kind of algorithmic behavior, most reasoning traces learned by large models fail to consistently capture or reuse procedures, instead drifting into verbose and degenerate exploration. To address more effective reasoning, we introduce reasoning abstractions: concise natural language descriptions of procedural and factual knowledge that guide the model toward learning successful reasoning. We train models to be capable of proposing multiple abstractions given a problem, followed by RL that incentivizes building a solution while using the information provided by these abstractions. This results in a two-player RL training paradigm, abbreviated as RLAD, that jointly trains an abstraction generator and a solution generator. This setup effectively enables structured exploration, decouples learning signals of abstraction proposal and solution generation, and improves generalization to harder problems. We also show that allocating more test-time compute to generating abstractions is more beneficial for performance than generating more solutions at large test budgets, illustrating the role of abstractions in guiding meaningful exploration.

1. Introduction

The paper addresses degenerate, depth-focused reasoning by training models to discover and use abstractions that support broader exploration of solution strategies.

The approach targets long-chain reasoning that often increases depth while drifting into logic switches and degenerate exploration instead of diverse strategies.The paper frames breadth—exploring multiple strategies—as a complementary goal to longer chains of thought.
Reasoning abstractions compress shared procedures, intermediate results, and factual knowledge into natural-language guidance for difficult problems.They can include useful lemmas, intermediate steps, or failed approaches that clarify what to do or avoid.
RLAD trains an abstraction generator to propose reasoning abstractions and a solution generator to use them when solving problems.The two models are jointly trained with reinforcement learning, and the abstraction generator is rewarded through improvements in solution accuracy.
Across AIME 2025, DeepScaleR Hard, and AMC 2023, RLAD improves over DAPO by an average of 44% on AIME 2025 and benefits from diverse abstractions over solution sampling.These results are presented as evidence that abstraction-guided exploration can improve reasoning performance.

2. Related Work

Related work studies scaling reasoning through parallel sampling, sequential trace editing, and reusable textual guidance; RLAD combines complementary search ideas through learned abstractions and RL.

Parallel sampling explores multiple reasoning rollouts, whereas sequential methods iteratively edit one trace and can become trapped on harder problems.Sequential search can nevertheless perform better on easier and medium-difficulty problems.
RLAD provides a hybrid of sequential and parallel sampling by proposing abstractions that guide subsequent solution generation.Unlike concurrent work that distills interleaved sampling structure, RLAD uses RL to optimize the abstraction-guided procedures.
Prior approaches also use examples, plans, prompts, retrieval, and input-specific edits as reusable textual guidance for LLM behavior.RAG generally assumes a static human-written corpus, while prompt-learning methods modify or refine textual guidance.

3. Preliminaries and Notation

The preliminaries define reasoning as generating a token stream for a problem and evaluating it with correctness-based accuracy and diversity-sensitive pass@k.

An LLM receives a problem x and generates a token stream y ending in an answer, which is evaluated against a ground-truth solution.Correctness is represented by a rule-based binary reward Acc_x(y, y⋆).
Training uses problem–solution pairs from D_train, while the objective is high reward on a test distribution P_test.Models are evaluated by average accuracy under the test distribution.
Pass@k counts a problem as solved when any of k sampled solutions is correct, coupling accuracy with response diversity.The metric is largest when the model finds diverse, good responses; n ≥ k samples are used to reduce estimation variance.

4. Reasoning Abstractions and Why They Are Useful

Reasoning abstractions summarize procedural and factual knowledge to broaden solution strategies, and their benefits depend on abstraction quality and solver capability across several domains.

4.1. Proposing Good Reasoning Abstractions by Summarizing Solution Attempts: Useful abstractions identify shared substructures in a reasoning graph, including convergent strategies, recurring errors, lemmas, heuristics, and cautionary examples.They are generated by summarizing patterns from multiple solution attempts, using a stronger model to extract useful insights.
4.2. Results and Observations: Conditioning solutions on abstractions improves math accuracy only when abstractions are sufficiently informative and generated by a strong model, while the solver can interpret them.The reported conditions include nontrivial abstraction length, o4-mini generation, and sufficiently capable Qwen3 solution models.
4.2. Results and Observations: ARC-AGI experiments report consistent gains in both pass@k accuracy and max@k coverage when using abstractions.The evaluation covers 90 puzzles drawn evenly from ARC-AGI 1, ARC-AGI 2, and BARC.
4.2. Results and Observations: The ability to generate or leverage useful abstractions does not arise naturally, motivating explicit training for both capabilities.The paper therefore treats abstraction proposal and abstraction use as skills requiring dedicated training.
4.2. Results and Observations: Proposed abstractions improve base math generators by 30% and improve performance by 30% on average across 37 healthcare, behavior, legal, and web-security tasks.The abstractions contain different balances of procedural and factual knowledge across domains.

5. RLAD: Learning to Propose Reasoning Abstractions

RLAD jointly trains an abstraction generator to propose useful procedural or factual guidance and a solution generator to solve problems conditioned on it. Its cooperative rewards, safeguards, warmstarting, and RL implementation are designed to make abstraction use effective and trainable.

RLAD training objective: RLAD jointly trains an abstraction generator and an abstraction-conditioned solution generator in a cooperative two-player RL setup.The abstraction generator proposes variable-length facts or procedures, while the solution generator produces responses conditioned on them.
RLAD training objective: An abstraction is rewarded according to the expected accuracy of solutions that the solution generator produces for the problem when conditioned on it.Solution rewards use binary accuracy against the ground-truth answer, and abstraction rewards average that outcome over generated solutions.
Reward design: Naïve rewards can fail when the abstraction generator or solution generator is too weak or too strong, or when the solution generator ignores the abstraction.These cases can drown out one player’s learning signal or reward abstractions that add no useful information.
Reward design: RLAD mixes abstraction-augmented and unaugmented prompts, zeroing rewards on unaugmented traces so the solution generator learns to use abstractions carefully.This modified reward design is intended to address the failure modes of the naïve objective while preserving ordinary behavior on prompts without abstractions.
Warmstarting: The abstraction generator is warmstarted with filtered, supervised examples before RLAD, while the solution generator is not warmstarted because base Qwen3 models already used abstractions well.Seed abstractions are generated by o4-mini, filtered for improved solution success, and used for five SFT epochs.
Practical training: After SFT, RLAD fine-tunes both models using batched offline RL for abstraction generation and a DAPO-based recipe with an easy-to-medium curriculum for solution generation.The hard DeepScaleR split is held out for evaluation, with an 8K token budget used initially on easy problems.

6. Experimental Evaluation

RLAD improves mathematical reasoning by generating and using multiple abstractions, and its benefits persist across inference settings and matched compute budgets. Experiments also show that abstraction conditioning increases solution diversity and adherence, making abstraction generation a valuable test-time scaling axis.

6.1. Main Performance Results on Math Reasoning Benchmarks: RLAD consistently outperforms base and DAPO models across abstraction-conditioned and no-abstraction settings on three math benchmarks.Table 2 evaluates AIME 2025, DeepScaleR Hard, and AMC 2023 with no abstractions, average performance across four abstractions, and the best abstraction.
6.1. Main Performance Results on Math Reasoning Benchmarks: Training with abstractions improves performance even when no abstraction is provided during inference, with similar trends on AIME 2024 and HMMT 2025.The authors attribute this observation cautiously to exposure to diverse abstractions during training and also report effectiveness at 8K and 16K token budgets.
6.2. Understanding Properties of RLAD: As inference budgets grow, allocating more compute to abstraction generation is more efficient than allocating it to solution generation across normalization offsets k_0.Iso-compute curves distribute budget C using m abstractions and k solutions, with m × (k − k_0) = C for nonzero offsets.
6.2. Understanding Properties of RLAD: At equal compute, abstraction conditioning reaches pass@k 0.71 versus 0.65 at n=16 and 0.87 versus 0.82 at n=256.The comparison is between n abstractions with n solutions per abstraction and n^2 solutions without abstractions.
6.2. Understanding Properties of RLAD: Different abstractions produce semantically more diverse solutions because the solution generator more faithfully follows the abstraction it receives.Figure 6 measures solution semantic similarity and abstraction adherence; the adherence analysis evaluates 200 abstraction–solution pairs.

7. Discussion and Perspectives on Future Work

RLAD frames natural-language reasoning abstractions as a complementary way to broaden strategies and scale test-time reasoning. Its evaluation is limited to mathematical reasoning, while jointly training one model to propose and use abstractions remains an open direction.

7. Discussion and Perspectives on Future Work: Reasoning abstractions are concise natural-language representations of procedural and factual knowledge that broaden the strategies available to LLMs.RLAD operationalizes them through a two-player framework with an abstraction generator and an abstraction-conditioned solution generator.
7. Discussion and Perspectives on Future Work: RLAD provides an orthogonal test-time compute axis alongside longer chains of thought and parallel solution sampling.The paper argues this matters as standard chain-of-thought length scaling shows limited gains, while diverse abstraction generation yields greater gains than solution sampling alone.
7. Discussion and Perspectives on Future Work: Evaluation covers mathematical reasoning only, leaving the usefulness of abstractions for open-ended reasoning unexplored.This is the paper’s stated scope boundary.
7. Discussion and Perspectives on Future Work: A single model trained to both propose and use abstractions quickly lost abstraction-proposal ability during RL, motivating targeted mid-training or active interventions.The authors identify this as an open direction rather than a resolved limitation.

A. Experimental Details

RLAD is evaluated against base and GRPO-enhanced models on AIME 2024 and HMMT 2025, where it achieves the best performance across both datasets and generalizes beyond abstraction-conditioned evaluation.

A. Experimental Details: The RLAD procedure alternates updates to the abstraction policy and solution policy across separate sampled problem sets.
A. Experimental Details: RLAD achieves the best performance across both AIME 2024 and HMMT 2025 compared with the base and GRPO-enhanced models.Table 5 reports abstraction-conditioned performance for the three model settings.
A. Experimental Details: RLAD also generalizes better when evaluated without abstractions, indicating that it learns to leverage high-level procedural guidance rather than merely fitting the abstraction format.

B.2. Design Choice Ablations

The ablations isolate curriculum training, no-abstraction prompts, and reward masking, finding that their full combination provides the strongest abstraction-conditioned performance.

B.2. Design Choice Ablations: Curriculum training improves average performance from 0.38 to 0.41 and best-case performance from 0.43 to 0.48 versus non-curriculum training.It stages training from simpler problems toward harder ones.
B.2. Design Choice Ablations: Including a small fraction of no-abstraction prompts is intended to condition the solution generator on abstractions when present, but can encourage ignoring them as a shortcut.Reward masking addresses this risk by zeroing the policy advantage for completions on no-abstraction prompts.
B.2. Design Choice Ablations: The full combination of curriculum training, no-abstraction prompts, and reward masking significantly outperforms alternative configurations.The full method achieves the strongest performance under abstraction-conditioned evaluation.

B.3. Full Results for Abstractions in non-math domains

Abstraction conditioning improves performance across diverse non-math domains, with average and best abstractions outperforming standard prompting across 37 tasks.

B.3. Full Results for Abstractions in non-math domains: For ARC-AGI, the abstraction generator is warmstarted with synthetically augmented human annotations from BARC, while the solution generator is frozen in the reported experiment.
B.3. Full Results for Abstractions in non-math domains: Figure 7 reports examples in non-math domains and states that adding an abstraction to GPT-4o-mini consistently improves performance on unseen instances.

D. Qualitative Examples of Math Reasoning Abstractions

The qualitative analysis categorizes abstractions by their cognitive function and illustrates reusable mathematical procedures, strategic launchpoints, structural shortcuts, and warnings against invalid solutions.

D. Qualitative Examples of Math Reasoning Abstractions: RLAD abstractions are categorized as caution alerts, productive launchpoints, blind-follow trajectories, and structural shortcuts to characterize their reasoning functions.Figure 8 presents the categorization as a diverse characterization of generated abstractions.
Examples of (C) Blind-follow trajectory: Blind-follow trajectories provide repeatable sequential procedures, such as using floor(log10 y) + 1 to count the integer digits of y.For y = x^7, the example computes floor(7 * log10 x) + 1.
Examples of (A) Caution alert: Caution alerts preserve solution validity by checking forbidden denominator values and rejecting roots that fail the original equation.The example excludes x = ½ for (x+2)/(2x–1) = x–3 and verifies accepted roots.
Examples of (A) Caution alert: The examples also emphasize preserving consistent units after extracting lengths from areas, such as converting 100 cm to 1 m before subsequent calculations.
Examples of (B) Productive launchpoint: Productive launchpoints reframe relationships into equations, such as translating “twice as many” and “one less than” into x = 2y and y = z - 3.This captures the core relationship in a solvable form.
Examples of (B) Productive launchpoint: Another launchpoint selects one variable as a parameter so the remaining variables can be expressed through it, reducing independent symbols and simplifying substitutions.With p/q = 3 and r/q = 2, choosing q gives p = 3q and r = 2q.
Examples of (D) Structural shortcut: Structural shortcuts eliminate variables using constraints, including substituting c = P-a-b under a+b+c=P to reduce Heron’s formula to two variables.The example becomes A(a,b)=sqrt[5 * (5-a) * (5-b) * (a+b-5)] for P=10.
Examples of (D) Structural shortcut: Geometric structural reasoning can equate two algebraic forms of the same distance, solve for the unknown, and back-substitute to determine the desired length.A perimeter-based shortcut similarly reduces a three-variable geometry expression to two variables.