PaperlayerView exact revision on arXiv

Source-linked AI summary

LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning

Weizhe Chen, Sven Koenig, Bistra Dilkina

arXiv:2510.01459v1cs.LGcs.CL

TL;DR

RLVR research has largely emphasized loss design, while dynamic sampling has mainly targeted training efficiency rather than final-model effectiveness. LSPO dynamically selects prompts using average response length, and experiments report consistently improved final models across settings.

  • Problem

    Recent RLVR progress has focused largely on loss design, while dynamic sampling has primarily aimed to improve training efficiency rather than final policy effectiveness.

    Source: §1 · p. 1

  • Method

    LSPO filters prompts by average response length, retaining prompts from the shortest and longest ranges for training with an RLVR algorithm.

    Source: §1 · p. 1, §4.2 · p. 6

  • Results

    LSPO consistently improves final model effectiveness across multiple base models, datasets, and RLVR algorithms.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The paper presents response length as a dynamic-sampling criterion that can complement gradient-based criteria in RLVR.

    Source: §7 · p. 10

    LSPO's advantage is not theoretically guaranteed for algorithms with strongly length-dependent loss functions, and its applicability may fail under fixed-token-length training.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

Since the release of Deepseek-R1, reinforcement learning with verifiable rewards (RLVR) has become a central approach for training large language models (LLMs) on reasoning tasks. Recent work has largely focused on modifying loss functions to make RLVR more efficient and effective. In this paper, motivated by studies of overthinking in LLMs, we propose Length-aware Sampling for Policy Optimization (LSPO), a novel meta-RLVR algorithm that dynamically selects training data at each step based on the average response length. We evaluate LSPO across multiple base models and datasets, demonstrating that it consistently improves learning effectiveness. In addition, we conduct a detailed ablation study to examine alternative ways of incorporating length signals into dynamic sampling, offering further insights and highlighting promising directions for future research.

1 INTRODUCTION

The paper proposes LSPO, a dynamic RLVR sampling method that filters prompts by average response length to address a gap in work focused mainly on loss design. Experiments report consistently better final models, and the paper examines alternative filtering choices.

1 INTRODUCTION

  • Recent RLVR progress has centered on loss design, while dynamic sampling strategies have primarily targeted training efficiency rather than final policy effectiveness.Source: §1 · p. 1
  • LSPO adaptively filters prompts by average response length, retaining a fixed ratio from the shortest and longest ranges after accuracy-based filtering.Source: §1 · p. 1, Figure 1 · p. 2
  • The paper's ablations examine LSPO hyperparameters, fixed-percentile versus value-based retention, and alternatives such as accuracy-based filtering.Source: §1 · p. 2
  • Across multiple base models, datasets, and RLVR algorithms, LSPO consistently improves final model effectiveness over baseline approaches and remains effective under the same training time despite additional sampling to match standard RL batch sizes.Source: §1 · pp. 1–2

2 RELATED WORKS

Prior RLVR work has explored alternative algorithms, loss functions, and sampling strategies. This paper positions response-length-based dynamic sampling as a complementary direction focused on final training effectiveness.

2 RELATED WORKS

  • RLVR work includes alternative algorithms and loss functions, alongside sampling and clipping changes intended to improve exploration, stability, or sample efficiency.Source: §2 · p. 2
  • Prior response-length research has studied shorter outputs, length prediction, and adaptive compute allocation; this work instead uses length to dynamically sample RLVR training data.Source: §2 · p. 3
  • Dynamic RLVR sampling has mainly targeted efficiency, whereas LSPO uses response length to select training data and improve final policy effectiveness.Prior approaches include dataset subset selection, reward-variance trajectory downsampling, zero-gradient sample removal, and useful-gradient prediction.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe GRPO and DAPO as RLVR algorithms and motivate response length as a potentially useful signal in reasoning tasks.

3.1 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD

  • GRPO computes each response's advantage relative to the average reward of a sampled group for the same prompt, while DAPO removes the KL term and uses asymmetric clipping.Source: §3.1 · p. 3
  • DAPO filters prompts whose current-batch accuracy is zero or one, resampling until the batch is filled, and adds a buffer-length penalty for overlong responses.Source: §3.1 · pp. 3–4

3.2 ROLE OF RESPONSE LENGTH IN LLM REASONING

  • Reasoning models can use long chains of thought to improve accuracy, but long responses create an efficiency bottleneck and may reflect perceived task difficulty.Source: §3.2 · p. 4
  • Studies cited in the paper report that incorrect responses are, on average, longer than correct ones, supporting response length as a signal of accuracy.Source: §3.2 · p. 4

4 LENGTH-AWARE DYNAMIC SAMPLING FOR POLICY OPTIMIZATION

LSPO filters RLVR training prompts using response length, retaining short and long responses while excluding the middle. It applies this filtering dynamically before an underlying RLVR algorithm updates the model.

4.1 LENGTH-AWARE FILTERING

  • LSPO keeps the shortest and longest responses, treating intermediate-length prompts as less informative for targeted training.Short responses indicate confidence, while longer ones suggest uncertainty and may offer opportunities to improve reasoning efficiency.Source: §4.1 · p. 4

4.2 LENGTH-AWARE DYNAMIC SAMPLING

  • LSPO caps retained responses with Lmax because retaining the most extreme lengths can degrade performance, and dynamically recalculates thresholds to retain a fixed percentage after accuracy filtering.This dynamic thresholding increases sampling time per step.Source: §4.2 · p. 5
  • LSPO needs enough prompts after accuracy filtering; the authors report that batch sizes of at least 256 work well in practice.A larger Llow or smaller Lhigh admits more prompts into training.Source: §4.2 · p. 5, §4.1 · p. 5
  • LSPO-trained models are stronger than their base algorithms across three challenging math benchmarks, using two models and different base algorithms.Source: Table 1 · p. 6
  • Each iteration removes prompts with uniformly zero or perfect accuracy, recalculates response-length percentiles, retains prompts from the shortest and longest ranges, and repeats collection until the training batch is sufficient.An arbitrary RLVR algorithm, such as GRPO or DAPO, then updates the model on the selected data.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

The experiments evaluate LSPO across models, datasets, and RL algorithms, then test its main results and alternative filtering designs. LSPO improves performance on nearly every tested benchmark, while ablations examine length-based and accuracy-based filtering.

5.1 EXPERIMENT SETUP

  • Experiments train Qwen-2.5-Math-7B and Qwen3-4B-Base for 24 hours, reporting average test accuracy from the last two available checkpoints on three benchmarks.LSPO is additionally evaluated on Llama-3.2-4B-Instruct; the benchmarks are AIME-25, Olympiad, and Minerva-Math, and the base algorithms are GRPO, DAPO, and GSPO.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • With LSPO, trained models perform better on nearly every benchmark tested.The authors report this as a robust result, even without additional optimization of training speed.Source: §5.2 · p. 7
  • Under the reported setting, LSPO’s rollout cost rises by roughly 60%, but reaches the same performance in less total training time.The comparison is between GSPO+LSPO and GSPO alone; the authors note RL training often uses about 80% of total runtime for the final 2% improvement.Source: §5.2 · p. 7

5.3 ABLATION STUDIES

  • Training on both response-length extremes has complementary benefits, whereas intermediate-length prompts alone produce the worst performance.Primarily training on shorter responses yields higher critic scores but does not generalize those benefits to test sets; excluding the shortest responses slightly reduces training and test performance.Source: §5.3 · p. 8
  • Value-based filtering performs comparably to percentile filtering but costs more; relative thresholds can require nearly 20 resampling rounds versus typically fewer than 10.Relative value-based filtering consequently takes longer, allows fewer training steps within the same budget, and produces worse evaluation performance than LSPO.Source: §5.3 · p. 8
  • Accuracy-based filtering adds almost no cost but provides no notable benefit over no additional filtering, making it less preferable as more retained samples increase rollout time.The authors identify predicting accuracy in advance as a possible direction for future exploration.Source: §5.3 · pp. 8–9

6 DISCUSSION

The discussion qualifies LSPO’s applicability and outlines possible ways to improve efficiency and filtering. Its benefits are not theoretically guaranteed for strongly length-dependent losses or fundamentally different training dynamics.

6.1 LIMITATIONS

  • For strongly length-dependent loss functions, LSPO’s filtering advantage is not theoretically guaranteed, though experiments with DAPO’s overlong buffer penalty empirically alleviate the concern.The paper applies that penalty across all loss functions in its experiments.Source: §6.1 · p. 9
  • If future post-training imposes fixed response lengths, such as exactly 4K tokens per prompt, LSPO may be unable to filter meaningfully.The authors describe LSPO as tailored to the dominant class of current RLVR algorithms.Source: §6.1 · p. 9

6.2 FUTURE DIRECTIONS

  • Predicting response lengths from recent samples could filter prompts before full rollouts, potentially reducing rollout cost.The paper also suggests generating a first sentence with a predicted length, then selectively continuing qualifying prompts.Source: §6.2 · p. 9
  • An adaptive filtering threshold could improve both training speed and model capability compared with the fixed percentile threshold used in LSPO.The authors present this as a future research direction.Source: §6.2 · p. 9
  • Entropy or self-confidence are possible alternative filtering signals, but whether they can match length-based gains remains open.Source: §6.2 · p. 9

7 CONCLUSION

LSPO computes average response length during sampling to select a subset of training data for LLM reasoning. Evaluation across multiple models and datasets demonstrates effectiveness, and the paper presents the work as a step toward dynamic sampling criteria beyond gradient magnitude alone.

7 CONCLUSION

  • LSPO computes average response length during sampling and dynamically selects a subset of data for training.Source: §7 · p. 10
  • Evaluations across multiple models and datasets demonstrate LSPO’s effectiveness, and the paper reports dynamic sampling can improve RLVR effectiveness as well as efficiency.Source: §7 · p. 10

A EXPERIMENT DETAILS

The appendix details the training setup, datasets, evaluation benchmarks, and prompting procedure used in the experiments.

A EXPERIMENT DETAILS

  • Experiments use 4 Nvidia H100 GPUs, a 1e−6 learning rate, temperature 1.0, and 8 sampled responses per question.The prompt asks models to reason step by step and return the final answer in a box.Source: A EXPERIMENT DETAILS · p. 13
  • Training uses DAPO-17K and MATH, with math verification for checking answer correctness.DAPO-17K contains 17K datapoints with integer answers; MATH has 7.5K training samples and more varied answer formats.Source: A EXPERIMENT DETAILS · p. 13
  • Evaluation uses AIME25, Minerva-Math, and the text portion of Olympiad-bench, checking answers with math verification.The text portion of Olympiad-bench contains 674 datapoints; AIME25 contains 30 problems and Minerva-Math 272 datapoints.Source: A EXPERIMENT DETAILS · p. 13

B.1 LLAMA-3.2-4B RESULTS

On Llama-3.2-4B-Instruct, LSPO paired with DAPO improves the training curve over DAPO alone, with almost no additional rollout cost.

B.1 LLAMA-3.2-4B RESULTS

  • LSPO with DAPO provides benefits near convergence and surpasses DAPO alone in peak performance.Figure 3 compares both approaches on Llama-3.2-4B-Instruct trained on MATH.Source: Figure 3 · p. 14, §B.1 · p. 14
  • Llama-3.2-4B-Instruct was trained on MATH and evaluated on Math-500 for 20 epochs.Source: §B.1 · p. 14
  • For Llama, LSPO adds almost no rollout cost because DAPO's accuracy-based filtering already absorbs most overhead.Source: §B.1 · p. 14
Loading 2510.01459v1…