PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Optimal Reasoning Length for RL-Trained Language Models

Daisuke Nohara, Taishi Nakamura, Rio Yokota

arXiv:2602.09591v3cs.CLcs.AIcs.LG

TL;DR

The relationship between reasoning length and accuracy in RL-trained language models remains unclear despite length-control methods intended to manage rising computational costs. This paper trains length-controlled policies across models and tasks, finding that sample accuracy peaks at an intermediate length while mode accuracy keeps improving as longer outputs become more dispersed.

  • Problem

    It remains unclear whether non-monotonic length–accuracy relationships arise across RL-trained policies and what mechanism drives them.

  • Method

    The paper trains matched RL policies with several length-control methods at varying penalty strengths and compares accuracy across output lengths.

  • Results

    Across three models and two task domains, sample accuracy peaks at an intermediate length while mode accuracy continues rising as longer outputs become increasingly dispersed.

  • Takeaways & Limitations

    The non-monotonic length–accuracy relationship is explained by dispersion around an increasingly correct distributional center.

  • Takeaways & Limitations

    The study could not reproduce GFPO’s reported length-reduction effect and therefore focuses on Token Avg and ALP in remaining settings.

Abstract

from arXiv · show

Reinforcement learning substantially improves reasoning in large language models, but it also tends to lengthen chain-of-thought outputs and increase computational cost. Although length-control methods have been proposed, the length-accuracy relationship they induce remains unclear. We train policies with several length-control methods on multiple base models in a controlled setup and find that, across both mathematical reasoning and code generation, accuracy is non-monotonic in output length, peaking at an intermediate value. Mode accuracy, however, continues to improve with length even in settings where sample accuracy plateaus or declines, indicating that the non-monotonic length-accuracy relationship is driven by dispersion around an increasingly correct center.

1. Introduction

RL improves reasoning but lengthens outputs and raises computational costs. Across controlled RL-trained policies, accuracy peaks at an intermediate length because longer outputs improve modal correctness while increasing dispersion.

  • Motivation: RL substantially improves language-model reasoning, but RL-trained models tend to produce longer outputs, increasing training and inference costs.These costs motivate length-control methods that penalize verbose outputs.
  • Length–accuracy relationship: Accuracy varies non-monotonically with average output length, peaking at an intermediate length before plateauing or declining.The pattern holds across three models, four mathematical reasoning benchmarks, and two code generation benchmarks.
  • Mechanism: The divergence between mode and sample accuracy reflects improved correctness of the distributional center alongside greater sample dispersion.Dispersion is measured as the average fraction of samples outside the modal group.
  • Mechanism: Mode accuracy continues improving with output length even when sample accuracy plateaus or declines.The distributional center is correct on more problems, while individual samples increasingly deviate from it.

2. Related Work

Prior work shows that reinforcement learning improves LLM reasoning but lengthens outputs and increases costs, motivating RL-based length-control methods. Other studies find that longer reasoning can hurt accuracy through increased dispersion and a non-monotonic sample-level relationship.

  • RL and reasoning length: RL improves LLM reasoning, but extended chain-of-thought outputs increase training and inference costs.The passage attributes these trends to recent reasoning models and methods including GRPO and DAPO.
  • Length-control methods: RL-based length-control methods either penalize longer correct responses or avoid explicit reward shaping through length weighting and sample filtering.Examples include RLOO-LP, ALP, DRPO, and GFPO.
  • Non-monotonic reasoning length: Prior analyses show that longer reasoning is not uniformly beneficial: increased output dispersion and excessively short or long generations can reduce correctness.These findings come from a simplified probabilistic analysis and empirical observations at the sample level.

3. Experimental Setup

The experiments train three language models on mathematical reasoning and code generation under a controlled DAPO-based setup, evaluating performance across standard math and coding benchmarks. They compare several length-control methods before focusing subsequent experiments on Token Avg and ALP.

  • Training setup: Three models are trained: DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-1.7B, and Qwen3-4B-Base.All three models are trained for mathematical reasoning; only DeepSeek-R1-Distill-Qwen-1.5B is trained for code generation.
  • Training setup: The DAPO-based configuration disables overlong reward shaping and matches PPO mini-batch size to generation batch size for fully on-policy updates.Length penalties are studied separately from overlong reward shaping.
  • Training setup: Training covers DAPO-Math-17k for mathematical reasoning and DeepCoder-Preview for code generation.The mathematical dataset is used with all three models, whereas code-generation training uses DeepSeek-R1-Distill-Qwen-1.5B.
  • Evaluation setup: Evaluation uses AIME 2024, AIME 2025, AMC, and MATH-500 for mathematics, plus HumanEval and MBPP+ for code generation, with multiple responses sampled per problem.The multi-response protocol follows DeepSeek-R1.
  • Length-Control Methods: The comparison includes Sample Avg and Token Avg baselines alongside explicit length-control methods, but later settings focus on Token Avg and ALP after GFPO was not reproduced.Sample Avg is the original GRPO objective, Token Avg is the DAPO objective, and ALP reduces to Token Avg when β = 0.

4. Results

Across length-controlled RL policies, accuracy rises to an intermediate output length before plateauing or declining across mathematical and code-generation tasks. This pattern reflects increasingly correct distributional centers alongside greater sample dispersion at longer lengths.

  • 4.1 Accuracy vs. Length: Accuracy rises, peaks, then plateaus or declines across length-control sweeps, showing a non-monotonic relationship with output length.The sweeps vary α for RLOO-LP, β for ALP, and λ for DRPO to produce policies with different average lengths.
  • 4.1 Accuracy vs. Length: The peak-then-decline pattern persists across three methods, multiple Qwen models, and math and code-generation benchmarks.On code, accuracy peaks and then declines on HumanEval, MBPP+, and their average.
  • 4.2 Mechanism: Mode accuracy continues improving with length while sample accuracy plateaus or declines, creating a widening gap between distributional-center correctness and individual samples.This indicates that the answer-distribution center becomes more correct even when samples do not track that improvement.
  • 4.2 Mechanism: Mode leakage explains the divergence: it rises monotonically on Qwen3-4B-Base and is U-shaped on the other reported model-and-benchmark settings.For the U-shaped cases, leakage is elevated at short lengths from weak concentration and at long lengths from samples spreading around an increasingly correct mode.
  • 4.2 Mechanism: At long lengths, a more correct center and greater dispersion act in opposition, so dispersion eventually outweighs center gains and lowers sample accuracy.The results extend the variance-based account by showing that longer outputs shift the center toward correctness while also increasing spread.

5. Conclusion · A. Experimental Details

The study evaluates how output length affects RL-trained reasoning models across multiple length-control settings, models, and task domains. It finds that sample accuracy is non-monotonic with length, whereas mode accuracy continues rising for long outputs.

  • 5. Conclusion: The study examines output length as a determinant of RL-trained reasoning-model performance.Policies were trained with several length-control methods at varying penalty strengths and compared by average output length.
  • 5. Conclusion: The experiments vary penalty strengths across several length-control methods.These settings generate policies spanning different average output lengths.
  • 5. Conclusion: The evaluation covers three models and two task domains.The length-performance relationship was assessed across both model and domain variation.
  • 5. Conclusion: Sample accuracy is non-monotonic as output length increases.Accuracy does not improve steadily throughout the tested length range.
  • 5. Conclusion: Mode accuracy continues to rise into the long-output regime.This differs from the behavior observed for sample accuracy.
  • 5. Conclusion: The analysis decomposes the answer distribution into the correctness of its center and its dispersion.This decomposition is used to examine the sources of the observed accuracy patterns.

A.1. Training

The study trains three models across mathematical reasoning and code generation using standardized datasets and the verl framework. Training uses GRPO with normalized group-relative advantages, dynamic sampling, and fixed optimization settings, while response lengths and training budgets are controlled across runs.

  • Training setup: Three models are trained on mathematical reasoning, while DeepSeek-R1-Distill-Qwen-1.5B is additionally trained on code generation.Mathematical experiments use DAPO-Math-17K, code experiments use DeepCoder-Preview, and all experiments use verl.
  • Optimization: GRPO removes the KL penalty (β = 0) and computes group-relative advantages by normalizing rewards across sampled responses per prompt.DAPO-style dynamic sampling filters prompts with zero gradient signal.
  • Optimization: Optimization uses AdamW with a learning rate of 1 × 10−6, weight decay of 0.1, and 10 warmup steps.The prompt batch size is 64, with 16 responses sampled per prompt.
  • Training budget: All models and tasks use a maximum response length of 16K tokens, with fewer than 10% of initial rollouts exceeding the cap.Within each task setting, runs use the same number of steps and are budgeted around at least 576 GPU-hours for the slowest run under an 8-GPU setup.
  • Preliminary experiments: Preliminary Qwen3-1.7B-Base experiments use an 8K-token maximum response length, BF16 training, and truncated importance sampling.These settings apply to training-dynamics experiments and a GFPO reproduction attempt.

A.2. Evaluation … B.4. GFPO

The appendix specifies evaluation sampling and stable training configurations, then describes four length-control methods that promote shorter or more efficient reasoning through reward shaping, weighting, or filtering.

  • A.2. Evaluation: Evaluation samples 64 responses per problem for AIME 2024, AIME 2025, and AMC, and 16 for MATH-500 and code generation.Mathematical evaluation uses the training prompt template, while code evaluation uses a slightly different template with consistent answer extraction.
  • A.3. Batch Size Configuration: The 512/32 batch configuration caused decreasing response length and validation performance, whereas equal generation and PPO batch sizes maintained stable training.The mismatch also substantially increased the absolute difference between rollout and training token probabilities, implicating off-policy instability.
  • A.4. Precision and Truncated Importance Sampling: For DeepSeek-R1-Distill-Qwen-1.5B, BF16 with TIS diverged in two runs, while switching to FP16 substantially reduced the rollout-training probability gap and enabled stable training.An FP16 ablation found no noticeable probability-gap difference with versus without TIS.
  • A.4. Precision and Truncated Importance Sampling: ALP with β = 1e−4 remained unstable in a length-penalty experiment, with a temporary probability-gap spike causing failure before a restarted run succeeded.The instability occurred despite FP16 and was smaller than the BF16–FP16 gap but still sufficient to destabilize training.
  • B.1. RLOO-LP: RLOO-LP applies per-prompt-normalized length penalties to correct responses, with α = 0 recovering standard RLVR and larger α favoring shorter responses.Normalization uses online mean and standard deviation of correct response lengths, limiting excessive penalties for difficult problems.
  • B.2. ALP: ALP adjusts penalty strength using per-prompt online accuracy, applying stronger penalties to high-accuracy problems and weaker penalties to low-accuracy problems.The max(acc(x), 1/G) term ensures a minimum penalty even when accuracy is zero.
  • B.3. DRPO: DRPO assigns higher length-based weights to shorter correct responses while normalizing only among correct responses, preserving positive learning signals regardless of length.Smaller λ favors shorter responses more strongly, while λ →∞ recovers the DisCO objective.
  • B.4. GFPO: GFPO samples larger response groups, retains the top-k responses under a filtering metric, and computes updates only from the retained subset.Length filtering favors shorter responses, while token-efficiency filtering ranks reward per token to encourage concise, high-reward outputs.

B.4.1. GFPO REPRODUCTION ATTEMPT · C. GRPO and DAPO Objectives

The GFPO reproduction attempt failed to reduce output length: both tested models instead produced increasingly long outputs relative to DAPO. GRPO and DAPO use different length-normalization schemes, which can affect training stability when output-length variance is high.

  • B.4.1. GFPO REPRODUCTION ATTEMPT: GFPO’s simplest length-based filtering configuration selected the k = 8 shortest responses from G = 16 samples per prompt on two base models.The method was applied to Qwen3-1.7B-Base and DeepSeek-R1-Distill-Qwen-1.5B.
  • B.4.1. GFPO REPRODUCTION ATTEMPT: GFPO did not reproduce the reported length-reduction effect in the experimental setup.The authors were unable to reproduce the expected behavior and excluded GFPO from the main experimental comparisons.
  • B.4.1. GFPO REPRODUCTION ATTEMPT: Average output length increased throughout GFPO training for both models, particularly later, relative to the DAPO baseline without a length penalty.The increase was even more pronounced on the validation dataset.
  • B.4.1. GFPO REPRODUCTION ATTEMPT: GFPO’s apparent early-training length advantage over DAPO was a logging artifact caused by averaging only the filtered shortest outputs.The logging did not average across all sampled responses.
  • B.4.1. GFPO REPRODUCTION ATTEMPT: Because GFPO excludes longer outputs from training, it may also remove long incorrect responses as a learning signal and fail when verbose failures must be explicitly penalized.This is presented as a hypothesis about why GFPO may not control output length effectively.
  • C. GRPO and DAPO Objectives: GRPO eliminates a separate value model by computing group-relative advantages from multiple sampled responses per prompt and normalizing loss by each response’s length.G denotes the number of responses per prompt, while |y_i| denotes the length of response i.
  • C. GRPO and DAPO Objectives: DAPO instead normalizes by total token count, and these alternative schemes can affect training stability when output-length variance is high.On Qwen3-1.7B-Base, Sample Avg (GRPO) and DRPO also show large negative length bias, with incorrect responses tending to be longer than correct ones.

C.1. Analysis of Output Length Variance … D.1.1. WALL-CLOCK TIME COMPARISON

Training instability is linked to high within-prompt output-length variance, which makes per-sample length normalization produce uneven gradient updates. Robustness checks show that the non-monotonic performance–length relationship persists at equal wall-clock training time, so short-output degradation is not explained by fewer training tokens.

  • C.1. Analysis of Output Length Variance: On Qwen3-1.7B-Base, Sample Avg (GRPO) and DRPO show large negative length bias, indicating insufficient suppression of long incorrect responses.Length bias is the normalized difference between mean correct and incorrect response lengths; negative values mean incorrect responses tend to be longer.
  • C.1. Analysis of Output Length Variance: Table 1 reports overall, within-prompt, and between-prompt coefficients of variation for output lengths during the first 10 training steps.The within-prompt CV is averaged across prompts, while the between-prompt CV is computed from per-prompt mean lengths.
  • C.1. Analysis of Output Length Variance: Qwen3-1.7B-Base has substantially higher within-prompt CV than DeepSeek-R1-Distill-Qwen-1.5B, destabilizing length-normalized gradient updates.Sample Avg weights each sample by 1/|o|, so high within-prompt variance creates pronounced weight differences.
  • C.1. Analysis of Output Length Variance: DRPO exhibits similar instability because its underlying DisCO framework uses the same per-sample length normalization in its score function.This shared normalization is specified in Eq. 8.
  • C.1. Analysis of Output Length Variance: DeepSeek-R1-Distill-Qwen-1.5B has lower within-prompt length variance and therefore supports stable training for both Sample Avg and DRPO.The model was distilled from a reasoning model and already generates long chain-of-thought outputs.
  • D.1. Robustness of the Non-Monotonic Relationship: The robustness checks test whether confounding factors explain the non-monotonic relationship, using DeepSeek-R1-Distill-Qwen-1.5B.This identifies the model and scope used for the robustness analysis.
  • D.1.1. WALL-CLOCK TIME COMPARISON: At a fixed budget of 576 GPU-hours, strong length penalties still underperform despite enabling more training steps, and the non-monotonic relationship persists.Figure 8 compares score with average output length at equal wall-clock training time, showing that short-output degradation is not due to reduced training tokens.

D.1.2. EXTENDED CONTEXT LENGTH EVALUATION

Evaluating with a 64K context shows that the non-monotonic score–output-length relationship persists, ruling out response truncation as the primary cause of long-output degradation. The results support a genuine trade-off in reasoning length rather than an experimental artifact.

  • Extended context evaluation: The non-monotonic score–output-length pattern persists under 64K evaluation, confirming that long-output degradation is not caused by truncation.Figure 9 reports consistency with 32K evaluation for DeepSeek-R1-Distill-Qwen-1.5B.
  • Extended context evaluation: Truncation rates remain similar between 32K and 64K contexts across all benchmarks, indicating that most responses exceeding 32K also exceed 64K.Thus, extending the evaluation context does not substantially reduce truncation.
  • Extended context evaluation: The findings support interpreting the non-monotonic relationship as a genuine trade-off in reasoning length rather than an experimental setup artifact.The extended-context evaluation was designed to rule out responses exceeding the 32K-token evaluation limit as an explanation.

D.2. Full Dispersion Analysis

Across most additional math settings and both code benchmarks, mode accuracy improves more steadily with output length than score, while long outputs show increasing dispersion around an increasingly correct mode. Two AIME 2025 settings are exceptions because their scores continue increasing over the tested length range without reaching the saturation regime.

  • D.2. Full Dispersion Analysis: Across most additional math settings, mode accuracy rises more steadily than score, creating a gap as score saturates at long output lengths.The analysis covers AIME 2024, AIME 2025, and AMC across three models.
  • D.2. Full Dispersion Analysis: Mode leakage rises at long lengths and is U-shaped in many settings, consistent with samples dispersing away from an increasingly correct mode.
  • D.2. Full Dispersion Analysis: On Qwen3-1.7B, score increases from 0.274 to 0.356 on AIME 2025, while mode accuracy tracks the increase over the tested length range.The mode–sample divergence has not emerged because score has not yet saturated or declined at long lengths.
  • D.2. Full Dispersion Analysis: On Qwen3-4B-Base, score increases from 0.133 to 0.215 on AIME 2025, and mode accuracy likewise tracks the increase.The divergence regime has not yet been reached in this setting.
  • D.2. Full Dispersion Analysis: The same dispersion-analysis patterns observed on math also hold on HumanEval and MBPP for DeepSeek-R1-Distill-Qwen-1.5B.MBPP is included to confirm the trend also holds beyond MBPP+.
Loading 2602.09591v3…