Source-linked AI summary
Concise Reasoning via Reinforcement Learning
Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang, Kartik Talamadupula
TL;DR
The paper addresses why RL-trained reasoning models become unnecessarily verbose and whether concision can be improved without sacrificing accuracy. It analyzes PPO and GRPO dynamics, relates incorrect answers to lengthening, and proposes a two-phase RL procedure. Across experiments, conciseness correlates with correctness, while a short secondary stage reduces response length and preserves or improves accuracy; GRPO nevertheless has collapse modes.
Problem
The paper studies why RL post-training can produce excessively long reasoning and whether models can become more concise without losing accuracy.
Method
The authors analyze PPO and GRPO loss dynamics and use a two-phase RL procedure whose secondary stage trains on occasionally solvable problems.
Results
Concise reasoning correlates with higher accuracy across reasoning and non-reasoning models, while the two-phase strategy produces more concise reasoning while preserving or improving accuracy.
Takeaways & Limitations
Response verbosity can reflect RL loss dynamics rather than deeper reasoning, and a brief secondary training phase can target concision without sacrificing reported accuracy.
Takeaways & Limitations
GRPO has structural collapse modes: its advantage becomes zero when groups are entirely correct or entirely incorrect, and fully solvable problems can cause its policy loss to vanish.
Abstract
from arXiv · showhide
A major drawback of reasoning models is their excessive token usage, inflating computational cost, resource demand, and latency. We show this verbosity stems not from deeper reasoning but from reinforcement learning loss minimization when models produce incorrect answers. With unsolvable problems dominating training, this effect compounds into a systematic tendency toward longer outputs. Through theoretical analysis of PPO and GRPO, we prove that incorrect answers inherently drive policies toward verbosity \textit{even when} $γ=1$, reframing response lengthening as an optimization artifact. We further uncover a consistent correlation between conciseness and correctness across reasoning and non-reasoning models. Building on these insights, we propose a two-phase RL procedure where a brief secondary stage, trained on a small set of solvable problems, significantly reduces response length while preserving or improving accuracy. Finally, we show that while GRPO shares properties with PPO, it exhibits collapse modes, limiting its reliability for concise reasoning. Our claims are supported by extensive experiments.
1. Introduction
The paper argues that verbosity during RL post-training is an optimization artifact rather than a reasoning strategy. It links conciseness with correctness and proposes analyzing PPO and GRPO dynamics to reduce unnecessary reasoning length.
- The authors argue that longer outputs arise from loss minimization after incorrect answers, rather than deeper reasoning.
- A two-phase RL strategy uses a secondary stage on solvable problems to reduce response length while preserving or improving accuracy.
- Concise reasoning strongly correlates with higher accuracy in both reasoning and non-reasoning models.
- PPO analysis shows that incorrect answers tend to drive longer responses, whereas correct answers encourage brevity.
- GRPO can encourage length under negative advantage and conciseness under positive advantage, but collapse modes limit its effectiveness.
2. Response Length vs. Accuracy
Across reasoning and non-reasoning models, shorter responses are strongly associated with correctness across benchmarks of varying difficulty. The paper therefore questions whether long chains of thought are inherently necessary for accuracy.
- Brevity and accuracy are strongly correlated across reasoning and non-reasoning models, with or without RL training.
- The correlation persists across benchmarks with varying difficulty, so it is not attributed to problem difficulty alone.
- RL post-training can produce significantly shorter responses while preserving or improving correctness, especially early in training.
- Longer chains of thought are not inherently necessary for higher accuracy, motivating analysis of when RL increases response length.
3. Each Reasoning Problem is an MDP
The paper models each reasoning problem as an MDP in which token generation proceeds through deterministic transitions and correctness is rewarded only at the terminal step. It argues that online RL can remain effective with very small datasets.
- Each reasoning problem is formalized as an MDP with token states, vocabulary actions, deterministic transitions, and terminal correctness rewards.
- Problem difficulty is measured by the probability that at least one sampled response is correct, distinguishing occasionally solvable, fully solvable, and unsolvable problems.
- Adding problems changes the initial-state distribution and reward function while retaining the underlying MDP structure.
- Online RL can remain effective with extremely small training sets because sampling diversity and continual updates support exploration and reduce stagnation.
4. PPO Impact on Response Length
The PPO analysis shows that terminal reward signs and GAE determine how response length affects loss, even with γ=1. Negative rewards favor longer responses under λ<1, while λ=1 introduces substantial instability and bias.
- PPO uses GAE to estimate advantages for policy and value losses, with γ=1 and λ=0.95 common in practice.
- With γ=1 and λ<1, PPO favors shorter responses for positive terminal rewards and longer responses for negative terminal rewards.
- The PPO loss tracks the mean advantage under bounded ratios and estimation deviations, including agreement in sign when advantages share one sign.
- When the terminal term dominates, negative residual reward favors longer responses, while positive residual reward favors shorter responses.
- For λ=1, average error scales linearly with response length, causing slower, inefficient, or unstable training and making long responses difficult to enforce.
- Using γ<1 affects the loss and TD errors, potentially degrading training, so the authors recommend using λ instead.
5. GRPO Impact on Response Length
GRPO encourages longer responses for negative advantages and can favor termination for positive advantages, but collapse modes can make its conciseness effect unreliable. Compared with PPO, GRPO is less robust when all responses are unsolvable or fully solvable.
- GRPO loss encourages longer responses for negative advantages and shorter responses for positive advantages.The analysis derives these opposing effects from GRPO’s loss dynamics.
- A negative advantage is sufficient to encourage longer responses, whereas a positive advantage increases the chance of termination at the current position.Positive advantage alone does not establish preference among multiple correct completions with different lengths.
- GRPO’s conciseness result is invariant to softmax temperature rescaling, and experiments indicate the Jacobian-dependent constant has only a minor effect.The reported inequality continues to support conciseness under the examined conditions.
- When two correct responses differ in length, GRPO generally reinforces the shorter response under similar conditions.The shorter response receives stronger reinforcement because the length ratio is below one.
- On unsolvable problems, GRPO can converge to fewer than 80 tokens because zero advantage leaves the KL term dominant.The authors attribute the short-output collapse to lower KL penalties for shorter responses.
- On fully solvable problems, accuracy reaching one makes the policy loss vanish, so KL dominance stalls progress and removes GRPO’s conciseness-inducing effect.This failure mode is especially relevant to small or skewed datasets dominated by fully solvable problems.
6. A Two-Phase Reinforcement Learning Strategy
The paper proposes a two-phase RL strategy: first train on challenging problems to build problem-solving capacity, then use occasionally solvable problems to enforce conciseness. The second phase can preserve or improve accuracy and may improve robustness to lower sampling temperatures, though a very small dataset may reduce generalization.
- A two-phase RL procedure delays conciseness training until after challenging-problem training has built problem-solving capacity.The first phase is expected to increase response length because wrong answers predominate.
- Excessive unsolvable problems can significantly delay and increase the cost of the transition from longer responses to conciseness.
- Training on occasionally solvable problems in the second phase enforces conciseness while preserving or even enhancing accuracy.The second phase uses problems with non-zero probability of being solved.
- The second phase substantially improves robustness to lower temperature, maintaining performance with limited sampling.
- A small problem set can support effective RL in the second phase, but it may reduce generalization.PPO can be applied to only a few problems after the model has already developed generalization capabilities.
7. Experimental Results
Experiments across difficulty levels, model sizes, benchmarks, and RL methods show that a two-phase strategy can shorten responses while preserving or improving accuracy, robustness, and performance. Minimal RL training also substantially improves non-reasoning models, especially without prior RL refinement.
- Overall results: The two-phase strategy produced significant improvements across models, including more concise reasoning and preserved or improved accuracy.The experiments also evaluated robustness under reduced sampling intensity and minimal RL improvements for non-reasoning models.
- Difficulty and response length: Across all problem sets and PPO or GRPO runs, accuracy improvements coincided with shorter response lengths.The three training sets had average pa values of 0.375, 0.25, and 0.16 from easiest to hardest.
- Response-length reduction: PPO training on eight MATH examples substantially reduced response length across AIME 2024, AMC 2023, and other evaluation benchmarks while maintaining reasonable accuracy.The setup used four samples per query at temperature 0.6 and top-p 0.95; GRPO was omitted from these easy-problem experiments because of collapse concerns.
- Performance and robustness: At temperature 0, the post-trained model significantly outperformed the baseline, indicating improved robustness under deterministic sampling.The authors contrast this with the severe accuracy degradation that temperature zero can cause in reasoning models.
- Minimal RL training: Using only four MATH examples, RL produced an improvement of up to 30% in the 1.5B model.The largest gains occurred for a model trained solely with token completion and without earlier RL-based refinement.
8. Concluding Remarks
The paper concludes that response length and performance should be optimized together through a two-phase RL strategy. Its practical guidance emphasizes concise reasoning without sacrificing accuracy, while noting unresolved scope and stability limitations.
- Main results: The method reduced response length by over 54% for R1 1.5B and 40% for R1 7B while preserving accuracy and improving low-temperature performance.These reductions were reported for the post-trained R1 models.
- Limitations: The main limitation is that the work does not test larger base models or domains beyond math and MMLU-STEM.Both extensions remain future research directions.
- Practical takeaways: Minimal RL training can substantially improve non-reasoning models, and conciseness can increase without compromising accuracy.The paper presents both as practitioner-relevant takeaways.
- Training guidance: PPO robustness depends critically on setting λ < 1, while training data should include occasionally solvable problems to reinforce conciseness.The conclusions also report that GRPO may train faster but becomes unstable as more problems are solved.
- Two-phase strategy: The proposed two-phase strategy first improves reasoning and generalization, then enforces conciseness while preserving established accuracy.The first phase may allow prolixity, whereas the second phase specifically targets shorter responses.
A.1. Applying Penalty and Discounting
The paper contrasts per-step penalties with discounting as ways to encourage brevity in reinforcement learning. Both have drawbacks: penalties can create premature termination, while discounting can shorten the effective decision horizon and harm accuracy.
- Penalty-based brevity: Per-step penalties can create local optima that encourage premature termination instead of complete solutions.The penalty must be large enough to influence decisions without overwhelming the positive reward for correctness.
- Penalty-based brevity: Finding one effective penalty is impractical because different problems require different response lengths.
- Discounting: Discounting avoids misleading local optima and does not alter the model’s fundamental reasoning process.
- Discounting: Discounting can dilute distant rewards by shortening the effective decision horizon, potentially degrading accuracy on long chains of thought.The adverse effect depends on the discount factor.
A.2. Value Behaviour of PPO
The value network evolves under opposing pressures during training. KL regularization pulls it toward previous near-zero values, while regression pushes it toward the target value, producing an intermediate equilibrium.
- KL regularization drives the value network toward zero to remain close to previous values.
- Regression loss pushes the value network toward Vg as value training progresses.
- The opposing losses create an equilibrium between zero and Vg where their effects balance.
A.3. Problems from OlympiadBench and AIME Used for Training
This material specifies training examples, develops advantage and loss analyses, and characterizes how PPO and GRPO respond to terminal-token probabilities. The reported results connect advantage sign with termination behavior and quantify the effect of removing normalization terms in Dr.GRPO.
- Training problems: OlympiadBench training used four problems, while AIME24 training used four example sets with different average accuracies.The datasets and problem identifiers are specified in the implementation description.
- Advantage analysis: For binary samples, standard deviation is minimized when all samples match and maximized when ones and zeros are nearly balanced.
- Dr.GRPO normalization: Dr.GRPO’s loss decreases by nearly 3× for group size 8 and up to 10× for group sizes 64–128 when division by σ is omitted.These are the reported loss changes from removing the standard-deviation normalization.
- Dr.GRPO normalization: Removing response length from the denominator increases loss magnitude from about 1K to 30K.
- GRPO termination behavior: Under a positive advantage, GRPO raises terminal-token probability and discourages longer sequences.
- GRPO termination behavior: Under a negative advantage, GRPO lowers terminal-token probability and promotes longer sequences.