PaperlayerView exact revision on arXiv

Source-linked AI summary

Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR

Fanfan Liu, Youyang Yin, Peng Shi, Siqi Yang, Zhixiong Zeng, Haibo Qiu

arXiv:2602.05261v1cs.CL

TL;DR

RLVR training can produce substantially different response-length dynamics because GRPO and GSPO contain length biases, including GSPO’s response length collapse. The paper analyzes these objectives and introduces LUSPO, which applies length-aware sequence optimization; across text-only and multimodal tasks, LUSPO improves stability and performance over existing methods.

  • Problem

    GRPO and GSPO embed response-length preferences in their objectives, but the causes of their differing length dynamics and GSPO’s collapse require analysis.

  • Method

    LUSPO analyzes GRPO and GSPO and scales each GSPO sequence’s loss by its own length to neutralize response-level length bias.

  • Results

    LUSPO improves training stability and performance across text-only and multimodal tasks, consistently outperforming GRPO and GSPO on evaluated benchmarks.

  • Takeaways & Limitations

    LUSPO provides a length-unbiased sequence optimization strategy that alleviates GSPO’s response length collapse while supporting reasoning-task performance.

Abstract

from arXiv · show

Recent applications of Reinforcement Learning with Verifiable Rewards (RLVR) to Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant success in enhancing reasoning capabilities for complex tasks. During RLVR training, an increase in response length is often regarded as a key factor contributing to the growth of reasoning ability. However, the patterns of change in response length vary significantly across different RLVR algorithms during the training process. To provide a fundamental explanation for these variations, this paper conducts an in-depth analysis of the components of mainstream RLVR algorithms. We present a theoretical analysis of the factors influencing response length and validate our theory through extensive experimentation. Building upon these theoretical findings, we propose the Length-Unbiased Sequence Policy Optimization (LUSPO) algorithm. Specifically, we rectify the length bias inherent in Group Sequence Policy Optimization (GSPO), rendering its loss function unbiased with respect to response length and thereby resolving the issue of response length collapse. We conduct extensive experiments across mathematical reasoning benchmarks and multimodal reasoning scenarios, where LUSPO consistently achieves superior performance. Empirical results demonstrate that LUSPO represents a novel, state-of-the-art optimization strategy compared to existing methods such as GRPO and GSPO.

1 Introduction

RLVR algorithms differ in how their objectives shape response length: GRPO and GSPO exhibit distinct length biases, while GSPO can collapse toward shorter responses. The paper analyzes these biases and proposes LUSPO, which adjusts GSPO and improves training and reasoning performance across text-only and multimodal settings.

  • Length bias: GRPO’s token averaging creates length bias: positive advantages favor shorter correct responses, while negative advantages penalize longer incorrect responses less.The resulting updates can push correct answers toward brevity and incorrect answers toward longer outputs.
  • Length bias: GSPO further amplifies response-level length bias through sequence-level clipping and Clip-Higher, which disproportionately removes negative-sample tokens.This imbalance makes positive-sample gradients dominate and encourages progressively shorter responses.
  • Observed dynamics: Under identical experimental conditions, GRPO lengthens responses during training, whereas GSPO produces response length collapse.The comparison uses Qwen2.5-VL-7B-Instruct and varies only the loss function.
  • LUSPO: LUSPO removes GSPO’s length bias by scaling each sequence’s loss by its own length.The modification is intended to alleviate GSPO’s degraded dense-model performance while preserving robust MoE training dynamics.
  • Empirical validation: Experiments across dense, MoE, text-only, and vision-language models show stable training and improvements over GRPO and GSPO on diverse benchmarks.The reported evaluations include AIME24, AIME25, MathVista, and MathVision.

2 Related Work

Related work centers on RLVR methods for improving reasoning and stabilizing reinforcement learning, especially for increasingly important MoE architectures. GRPO faces token-level variance, while GSPO addresses stability through sequence-level optimization.

  • RLVR methods: GRPO eliminates a value model by computing each response’s relative advantage within a group for the same query.The method is presented as a representative RLVR strategy following DeepSeek-R1’s success.
  • MoE motivation: MoE architectures are increasingly explored for efficient training, expert routing, and sparse activation in general-purpose models.The passage frames selecting an appropriate RLVR algorithm as relevant to this architectural trend.
  • Stability challenge: Token-level importance ratios in GRPO and its extensions can have high variance, especially in MoE models and with longer responses, increasing the likelihood of unstable updates.Routing diversity and response length are identified as factors amplifying token-level fluctuations.
  • Sequence-level methods: GSPO replaces token-level importance ratios with sequence-level ratios and applies sequence-level clipping, reward assignment, and optimization.The passage reports improvements over GRPO in training stability, efficiency, and overall performance, particularly for large MoE models.

3 Preliminaries

The preliminaries formulate autoregressive policies and group-based RL objectives, then distinguish GRPO’s token-level weighting from GSPO’s sequence-level importance weighting. Sequence-level weighting is presented as compatible with sequence-level rewards and clipping.

  • Notation: The paper models an autoregressive language model parameterized by θ as a policy πθ, with |y| denoting the number of tokens in response y.A verifier assigns reward r(x, y) to each query-response pair.
  • GRPO: GRPO samples a response group for each query, computes rewards, and optimizes an objective using token-level importance ratios and group-based advantages.All tokens in a response share its common advantage, while G denotes the group size.
  • Importance weighting: GRPO’s token-level importance weighting is contrasted with GSPO’s sequence-level importance weight, which measures how a sampled response differs between the behavior and current policies.The sequence-level measure is described as compatible with sequence-level rewards and as a basis for clipping.
  • GSPO: GSPO adopts a sequence-level optimization objective using a group-based advantage and an importance ratio defined from sequence likelihood.This formulation follows the sequence-level policy comparison described in the preliminaries.

4 Algorithm

Response length in RLVR is shaped both by reward-driven effects and by length preferences embedded in optimization losses. The analysis identifies length bias in GRPO and GSPO and introduces LUSPO to remove GSPO’s trajectory-length dependence.

  • Response Length Variation: Response length changes reflect both reward-driven effects and preferences embedded in the loss function.The paper focuses primarily on loss-function preferences exemplified by GRPO and GSPO.
  • Response Length Bias: Long-response tokens receive smaller GRPO loss contributions because token averages divide by trajectory length.Short responses therefore assign greater weight to each token.
  • Response Length Bias: Correct short responses receive higher token weights, whereas incorrect samples favor longer outputs; high step accuracy leads to shorter responses and low accuracy to longer ones.This establishes a direct link between sample correctness, step accuracy, and response-length preference.
  • Response Length Bias in GSPO: GSPO retains GRPO’s length bias, while sequence-level clipping and Clip-Higher increase clipping and let positive-sample gradients dominate, shortening responses.Sequence-level clipping clips more tokens than token-level clipping, and Clip-Higher clips more negative than positive samples.
  • Length-Unbiased Sequence Policy Optimization: LUSPO scales each sequence’s loss by its own length, eliminating the length-dependent bias present in GSPO for each trajectory.The gradient comparison supports the claim that the modification removes trajectory-level length dependence.
  • Length-Unbiased Sequence Policy Optimization: LUSPO alleviates GSPO’s response-length collapse and promotes more balanced, stable learning across dense and Mixture-of-Experts architectures.The method is intended to prevent longer sequences from being unfairly penalized during training.

5 Experiment

The experiments evaluate LUSPO across dense and Mixture-of-Experts architectures and across text-only and vision-language models. The setup uses specified Qwen backbones, hardware, optimization settings, and scientific-problem datasets.

  • Experimental Scope: Experiments cover dense and Mixture-of-Experts architectures as well as text-only and vision-language models.This design assesses the method across model architectures and modalities.
  • Models: Qwen2.5-7B-Base represents dense text-only models, Qwen3-30B-A3B-Instruct represents text-only MoE models, and Qwen2.5-VL-7B-Instruct supports multimodal evaluation.These are the specified model backbones for the respective settings.
  • Training Setup: Qwen2.5-7B-Base and Qwen2.5-VL-7B-Instruct were trained on 8 Nvidia H800 GPUs, while Qwen3-30B-A3B-Instruct used 4 x 8 Nvidia H800 GPUs.Training primarily used the verl framework.
  • Hyperparameters: Training used AdamW with a fixed learning rate of 1 × 10−6, 20 rollout warm-up steps, batches of 128 prompts, and 8 responses per prompt.Mini-batch size was 16, with maximum generation lengths of 32,768 tokens for text-only and 4,096 tokens for VL models.
  • Datasets and Evaluation: Training used DAPO-MATH-17K for text-only models and ViRL39K for the vision-language model, with evaluations focused primarily on mathematical and logical tasks.Both datasets focus on scientific-related problems.

Reward

The training reward combines accuracy, format adherence, and response length, with accuracy and format rewards explicitly defined. The overlong reward uses maximum generation length, a buffer, and response length.

  • The training reward consists of accuracy, format, and response-length components.
  • Accuracy reward is 0 or 1 depending on whether the answer is correct.
  • Format reward is 0 or 0.5 depending on whether the required prompt format is followed.
  • The overlong reward is defined using maximum generation length, a length buffer, and response length.The buffer is 512 for Qwen2.5-VL-7B-Instruct and 4096 for the two text-only models.

5.2 Main Results

LUSPO improves over GSPO and GRPO across text-only and multimodal evaluations. On multimodal benchmarks, it consistently outperforms both baselines, with especially large gains on WeMath and LogicVista.

  • Both dense and MoE models demonstrate significant improvements with LUSPO over GSPO on text-only benchmarks.
  • LUSPO consistently outperforms GRPO and GSPO across multimodal benchmarks using Qwen2.5-VL-7B-Instruct.The benchmarks include MathVista-mini, MathVision, MathVerse, DynaMath, WeMath, and LogicVista.
  • 5.1% and 6.0% improvements over GSPO occur on WeMath and LogicVista, respectively.
  • GSPO achieves lower average scores than GRPO on several multimodal benchmarks.The passage attributes this to GSPO amplifying length bias from averaging token gradient contributions within a trajectory.

5.3 Training Dynamics

LUSPO increases response length more rapidly and avoids the response-length collapse observed with GSPO, particularly in the vision-language model. On validation data, LUSPO responses are nearly 1.5 times longer.

  • At the same number of training steps, LUSPO increases response length more rapidly than GSPO.
  • In the VL model, GSPO causes pronounced response-length collapse, whereas LUSPO maintains stable and sufficiently long responses.The passage links this stability to better use of multimodal information and sustained reasoning ability.
  • LUSPO’s average validation response length is nearly 1.5 times that of GSPO.

Accuracy Reward

Accuracy reward is used to monitor reinforcement-learning progress, while validation performance assesses generalization. During training, LUSPO achieves higher rewards and improved validation performance than GSPO.

  • Accuracy reward is recognized as a fundamental metric for monitoring reinforcement-learning progress and effectiveness.
  • At the same number of training steps, LUSPO consistently outperforms GSPO in accuracy reward for both dense and MoE models.
  • LUSPO eliminates GSPO’s inherent length bias and produces longer responses that provide a larger exploration space.The passage states that this makes solving more complex problems easier.
  • LUSPO achieves higher training rewards and significantly improved validation performance than GSPO.The validation metric is avg@32 on AIME24, evaluated every 10 steps for the specified text-only models.

5.4 Ablation Study

On ViRL39k and DAPO-MATH-17k, LUSPO consistently outperforms GSPO across five multimodal benchmarks, while maintaining higher response lengths during training.

  • The ablation compares GSPO and LUSPO on ViRL39k and DAPO-MATH-17k using multimodal benchmarks.
  • LUSPO consistently outperforms GSPO across all five benchmarks evaluated on ViRL39k and DAPO-MATH-17k.The comparison uses multimodal benchmarks, with an extra blank image added to DAPO-MATH-17k questions for vision-language training.
  • LUSPO produces consistently higher response lengths than GSPO during training.

6 Conclusion

The paper analyzes response-length bias in GRPO and GSPO and introduces LUSPO as a length-aware sequence-level optimization method. Across text-only and multimodal tasks, the method improves training stability and performance.

  • The paper analyzes the causes of response-length bias in the objective functions of GRPO and GSPO.
  • LUSPO applies a length-aware adjustment to sequence-level optimization to address GSPO’s response-length bias.
  • LUSPO improves training stability and performance in both text-only and multimodal tasks.
  • Extensive experiments across model types and diverse benchmarks demonstrate the robustness and effectiveness of LUSPO.
Loading 2602.05261v1…