PaperlayerView exact revision on arXiv

Source-linked AI summary

Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex

Yun Qu, Qi Wang, Yixiu Mao, Heming Zou, Yuhang Jiang, Yingyue Li, Wutong Xu, Lizhou Cai, Weijie Liu, Clive Bai, Kai Yang, Yangkun Chen, Saiyong Yang, Xiangyang Ji

arXiv:2605.06139v2cs.LGcs.AI

TL;DR

Existing group-based RLVR policy gradients implicitly approximate target projections, leaving the properties of explicit projection unclear. LPO makes projection explicit on the response simplex and consistently improves training performance while preserving stability and response diversity.

  • Problem

    Group-based RLVR methods implicitly approximate target projections, raising the question of what properties emerge when target construction and projection are made explicit.

  • Method

    LPO constructs an explicit target on the sampled-response simplex and projects the policy toward it through exact divergence minimization.

  • Results

    Across diverse reasoning tasks and LLM backbones, LPO achieves higher expected Pass@1 and Pass@k accuracy during training, with stable optimization and preserved response diversity.

  • Takeaways & Limitations

    Explicit target-projection provides a unified geometric framework for group-based RLVR and a flexible design space for stable, diverse optimization methods.

  • Takeaways & Limitations

    The formulation primarily focuses on sequence-level projection within outcome reward settings; step-level projections and broader divergences remain future work.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existing recipes, group-based policy gradient is prevalent, which samples a group of responses per prompt and updates the policy via group-relative advantage signals. This work reveals that these optimization strategies share a common geometric structure: each implicitly defines a target distribution on the response simplex and projects toward it via first-order approximation. Building on this insight, we propose Listwise Policy Optimization (LPO) to explicitly conduct the target-projection, which demystifies the implicit target by restricting the proximal RL objective to the response simplex, and then projects the policy via exact divergence minimization. This framework provides (i) monotonic improvement on the listwise objective with bounded, zero-sum, and self-correcting projection gradients, and (ii) flexibility in divergence selection with distinct structural properties through the decoupled projection step. On diverse reasoning tasks and LLM backbones, LPO consistently improves training performance over typical policy gradient baselines under matched targets, while intrinsically preserving optimization stability and response diversity.

1 Introduction

The paper presents group-based RLVR as implicit target-projection on the finite response simplex and introduces Listwise Policy Optimization (LPO) to make target construction and divergence projection explicit. Theory and evaluations show improvement guarantees, flexible divergence choices, and stable optimization across diverse reasoning tasks and LLM backbones.

  • Group-based policy gradients as implicit target-projections: Group-based policy gradient methods implicitly perform approximate target-projections by constructing reward-weighted listwise distributions over sampled responses on a simplex.This unified geometric perspective reframes advantage-based group RL algorithms through their underlying optimization mechanism.
  • Listwise Policy Optimization: LPO explicitly projects toward a target distribution on the sampled-response simplex by combining closed-form target construction with divergence minimization.The proximal RL objective is constrained to sampled responses, yielding a controllable-temperature target, while policy projection is decoupled through divergence selection.
  • Listwise Policy Optimization: LPO’s decoupled framework provides improvement guarantees and characterizes structural properties of projections while permitting diverse divergence choices.A novel forward KL variant is reported as exceptionally competitive.
  • Evaluations: Across logic, mathematics, programming, and multimodal reasoning tasks with diverse LLM backbones, LPO achieves higher expected Pass@1 and Pass@k accuracy during training than matched-target baselines.The comparison uses matched implicit target constructions.
  • Evaluations: LPO intrinsically induces highly stable optimization and preserves response diversity during training.These properties are presented as outcomes of explicit target-projection and flexible divergence design.

2 Preliminaries

RLVR is framed as a post-training paradigm using reward-based optimization for LLM reasoning, with preliminaries defining its KL-regularized objective and common group-based policy-gradient setup. The paper focuses primarily on sparse, rule-based rewards without an explicit reference penalty.

  • RLVR formulation: RLVR incentivizes LLM reasoning capabilities as a critical post-training paradigm.The preliminaries denote x as a prompt and y as a generated response.
  • RLVR formulation: The standard RLVR formulation uses a reward function R(x, y) and reference policy πref in a KL-regularized objective.The reference-constraint strength is controlled by β.
  • RLVR formulation: The analysis primarily considers rule-based outcome rewards that are binary or sparse, with R ∈[0, 1] and β = 0.This removes the explicit reference penalty from the considered setting.
  • Group-based policy gradient: Group-based policy gradient, represented by GRPO, samples K responses per prompt and converts their rewards into centered and scaled group-relative advantages.A behavior policy, typically the pre-update snapshot πθold, generates responses with rewards forming vectors R and A.
  • Group-based policy gradient: The policy is typically updated by maximizing a clipped surrogate objective based on the group-relative advantages.The preliminaries identify clipping and group-based normalization as standard components of the update.

3 Group-based Policy Gradient as Implicit Target-Projection

Group-based policy gradients can be understood as implicitly projecting the policy toward advantage-induced target distributions on the response simplex. At the on-policy point, this projection is exactly reverse-KL minimization, while off-policy drift introduces bounded approximation error and normalization mainly controls target sharpness.

  • Response-simplex representation: The listwise distribution Pθ represents the policy’s relative preferences over K sampled responses and lies on the response simplex ΔK−1.At the on-policy point, Pθ becomes the uniform distribution 1/K.
  • Implicit target-projection: At the on-policy point, standard group-based policy gradients equal the negative gradient of reverse KL toward w∗ = softmax(A), for zero-mean advantages A.Thus, the advantage design defines the implicit target distribution on the response simplex.
  • Implicit target-projection: The reverse-KL equivalence is exact on-policy, while off-policy approximation error scales as O(δ̄ · (1 + ∥A∥∞)/K).Here, δ̄ measures the maximum relative drift between πθ and the sampling policy πb.
  • Normalization-induced targets: For advantages Ak = (Rk − µ)/τ, shift-invariance removes centering, yielding the target w∗ = softmax(R/τ), where τ controls sharpness while preserving reward ordering.Different normalization schemes therefore mainly change target sharpness rather than response ranking.
  • From approximation to exact projection: Because w∗ and Pθ both lie on the finite response simplex, exact projection can minimize alternative divergences such as Forward KL.This motivates decoupling projection from the policy-gradient approximation.

4 Listwise Policy Optimization

Listwise Policy Optimization (LPO) makes group-based RLVR’s implicit target explicit by separating target construction from divergence-based projection on the response simplex. Its finite-simplex formulation supports tractable optimization, monotonic improvement guarantees, and flexible forward- or reverse-KL projections with stability and diversity benefits.

  • Target-Projection Framework: LPO decouples each iteration into constructing a target distribution w* and projecting the policy toward it by minimizing a chosen divergence on the response simplex.The target derives from a proximal objective on the simplex, while projection uses D(w*∥Pθ).
  • Target Construction: The listwise Gibbs target reweights the baseline toward high-reward responses, approaching the best response as τ → 0 and the baseline as τ → ∞.Under on-policy sampling, the baseline is uniform and w* = softmax(R/τ), recovering existing implicit targets with τ as an explicit trust-region parameter.
  • Target Construction: As K → ∞, the finite-simplex objective recovers KL-regularized RL, while finite response lists make the target tractable and explicit.The limiting solution is w* ∝ πt(y) exp(R(y)/τ), avoiding the intractable partition function of the full policy space.
  • Guarantees: Exact or sufficiently accurate projection yields monotonic listwise-reward improvement, and perfect projection strictly improves reward whenever Pt ≠ w*.The framework also provides an idealized full-policy-space convergence result toward the reward-maximizing policy.
  • Divergence Choices: Forward- and reverse-KL projections provide distinct structures: forward KL uses bounded, zero-sum, self-correcting gradients, while reverse KL recovers the proximal objective and includes an implicit entropy bonus.The zero-sum property supports variance reduction, and the forward-KL mode-coverage result supplies a log-barrier against mode collapse.

5 Main Empirical Results

Across reasoning tasks and diverse LLM backbones, LPO consistently improves training performance over matched-temperature group-based policy-gradient baselines. Its empirical advantages coincide with greater response diversity, more stable optimization, and robustness across projection and sampling settings.

  • Performance gains: LPO surpasses matched group-based policy-gradient baselines in nearly all evaluated settings for both Pass@1 and Pass@k.For Pass@1, LPOfwd and LPOrev each win in 13/15 settings; for Pass@k, they win in 15/15 and 11/15 settings, respectively.
  • Projection divergence effects: LPOfwd outperforms LPOrev in 13/15 Pass@k scenarios, consistent with forward-KL mode coverage preserving diverse valid reasoning paths.The decoupled projection step allows divergence choice to shape structural properties independently of the target temperature.
  • Robustness across temperature parameterizations: LPO retains stable performance gains across all tested temperature designs, although no single implicit temperature strategy dominates every benchmark.This indicates that exact listwise projection provides benefits largely orthogonal to the temperature heuristic.
  • Training dynamics: LPO maintains higher response entropy, lower and more stable gradient norms, and generally longer responses than group-based policy-gradient methods.These dynamics are associated with preserved exploration, controlled optimization, and more extensive reasoning chains.
  • Listwise ablation: Removing the coupled listwise mechanism causes a severe performance drop, whereas group-based PG and LPO retain stabilizing control-variate behavior.The pointwise variant lacks competitive coupling across responses, producing unstable optimization despite keeping the target unchanged.
  • Scaling and generalization: Across group sizes K ∈{2, 4, 8, 16, 32} and Qwen, DeepSeek, Mistral, and Llama backbones, LPO remains competitive or consistently improves performance.Advantages are especially pronounced with smaller groups; LPOrev is stronger on Pass@1, while LPOfwd scales exceptionally well on Pass@64.

6 Conclusion … B Proofs

The paper presents LPO as an explicit target-projection method on the finite response simplex, unifying group-based RLVR through a geometric view that improves stability and supports monotonic improvement. The appendices provide proofs, related-work context, implementation and experimental details, while future work extends projection granularity and divergence choices.

  • 6 Conclusion: LPO explicitly projects policies toward targets on the response simplex, unifying existing group-based RLVR methods as approximate target-projection procedures.The framework directly optimizes on the simplex, improving optimization stability and yielding monotonic performance improvements.
  • 6 Conclusion: The conclusion identifies sequence-level projection within outcome-reward settings as the current scope and proposes step-level projections and broader divergences as future directions.These extensions aim to further develop the decoupled target-projection framework.
  • Appendix Overview: The appendix overview organizes supplementary proofs, conceptual discussions, and experimental details supporting the main text.It introduces appendices covering related works, proofs, discussions, implementation, experiments, and data examples.
  • B Proofs: Appendix B provides detailed mathematical derivations for all theoretical claims.This appendix is identified as the proofs section supporting the paper’s theoretical results.
  • Appendix Overview: Appendix C expands the framework’s conceptual and practical scope by unifying group-based RLVR algorithms, comparing listwise projection with pointwise and preference optimization, and exploring future extensions.These discussions clarify how the proposed formulation relates to neighboring optimization paradigms.
  • Appendix Overview: Appendices D–F document the experimental setup, extended empirical findings, and representative data examples used across the evaluated reasoning tasks.The extended results include scalability validation, on-policy optimization, training dynamics, and generalization across diverse LLM families.
  • A Related Works: The related-work discussion situates RLVR within the shift from PPO with learned value models toward critic-free, group-based policy-gradient methods for reasoning tasks.It also connects the framework to RL-as-inference and listwise formulations in alignment and learning-to-rank research.
  • A Related Works: LLM responses form a finite response simplex with exact normalization, enabling LPO’s listwise projection to couple all responses through shared normalization.This contrasts with pointwise projections typically used in continuous-action RL-as-inference methods.

B.1 KL Gradient Derivations

This section derives forward- and reverse-KL gradients on the response simplex using behavior-relative logits. The reverse-KL gradient further simplifies into a baseline-subtracted form governed by the expected logit gap under the current policy.

  • Shared setup: Both KL derivations use behavior-relative logits s_θ,k = log π_θ(y_k|x) − log π_b(y_k|x), whose parameter gradient equals ∇_θ log π_θ(y_k|x) because π_b is frozen.The frozen behavior policy removes its contribution when differentiating the logits.
  • Forward KL: The forward-KL derivation substitutes log P_θ,k = s_θ,k − log ∑_K into the gradient and simplifies the resulting summation using the logit-gradient identity.The entropy H(w*) is constant with respect to θ in the intermediate expression.
  • Reverse KL: The reverse-KL derivation differentiates with respect to each logit using the softmax Jacobian ∂P_θ,k/∂s_θ,j = P_θ,k(δ_kj − P_θ,j) and the product rule.This produces the per-logit coefficients used in the reverse-KL gradient.
  • Reverse KL: Writing P_θ,k and w*_k with partition functions makes their log-ratio equal to the logit gap d_k = s_θ,k − ϕ_k minus the k-independent constant c_s.The constant is c_s = log Z_s − log Z_ϕ.
  • Reverse KL: The reverse-KL gradient therefore has a baseline-subtracted form, with the baseline given by the expected logit gap under the current policy P_θ.The partition-function constant cancels from the expected-gap expression.

B.2 Proof of Proposition 1

Proposition 1 shows that, at the on-policy point, group-based policy gradients equal reverse-KL descent toward the softmax advantage target. Off-policy, this equivalence degrades linearly with sampling drift, motivating exact listwise projection.

  • On-policy equivalence: At πθ = πb, the group-based policy gradient equals the negative gradient of reverse KL toward w* = softmax(A).The proof evaluates both gradients through their per-logit coefficients and finds them identical at the on-policy point.
  • On-policy equivalence: On-policy logit offsets vanish, producing the uniform list distribution Pθ,k = 1/K and logit gaps dk = −Ak.These substitutions yield the coefficient match with the standard policy gradient.
  • Centering assumption: The zero-mean advantage condition is without loss of generality because softmax shift-invariance absorbs any uniform prompt-level reward baseline.This includes baselines such as ReMax’s greedy baseline, which are mathematically nullified by the local simplex’s zero-sum structure.
  • Off-policy approximation error: Off-policy, the gradient discrepancy vanishes on-policy and grows linearly with the importance-ratio drift δ̄ in the local regime δ̄ < 1/2.The analysis bounds the parameter-space discrepancy using Gmax and concludes that the approximation holds only near the sampling distribution.
  • Connection to group-based policy gradients: A first-order expansion decomposes the exact reverse-KL projection into pointwise advantage fitting, intrinsic entropy regularization, and listwise normalization.Thus, unclipped group-based policy gradients retain only the first-order advantage-fitting component, whereas LPO preserves the coupled listwise terms.

B.3 Proof of Theorem 1 … C.3 Existing Group-based RLVR as Implicit Target-Projection

The paper formalizes LPO as explicit target projection on the response simplex, proving unique Gibbs targets, reverse-KL equivalence, performance guarantees, and structural gradient properties. It also situates existing group-based RLVR methods within this implicit target-projection framework while clarifying scope, extensions, and design choices.

  • B.3 Proof of Theorem 1: Theorem 1 establishes that the listwise objective has a unique maximizer, the Gibbs target w* on the response simplex.Uniqueness follows because the reward term is linear and D_KL(w∥P_t) is strictly convex when P_t,k > 0.
  • B.4 Proximal Objective as Reverse KL: Proposition 3 rewrites the proximal objective as reverse-KL minimization toward w*, making the target-projection interpretation explicit.Specifically, J^(P_θ) = −τD_KL(P_θ∥w*) + τ log Z^, so maximizing the objective is equivalent to minimizing D_KL(P_θ∥w*).
  • B.5 Proof of Theorem 2: Theorem 2 bounds performance loss after projection using the projection error TV(P_t+1, w*) ≤ ϵ_proj under bounded rewards |R_k| ≤ R_max.The proof connects reward improvement to symmetric KL divergence and bounds reward error through total variation and Pinsker’s inequality.
  • B.6 Proof of Proposition 2: Exact full-space proximal updates produce π_t(y) ∝ π_0(y) exp(tR(y)/τ) and converge in expected reward to max_y R(y).This idealized result motivates LPO because the full sequence-space partition function is intractable for autoregressive LLMs.
  • B.7 Proof of Corollary 1: Corollary 1 shows that forward-KL projection gradients have bounded, zero-sum, and vanishing coefficients, supporting stable self-correcting updates.The coefficient bounds also control the gradient norm through the triangle inequality.
  • B.8 Proof of Corollary 2: Corollary 2 derives a rigorously corrected lower bound on each policy probability when the projection remains close to the target.The proof applies data processing to a binary event space and exponentiates the resulting inequality.
  • C.1 Contribution Clarification: The framework’s core contribution is unifying group-based policy-gradient methods as implicit Gibbs-target construction with approximate reverse-KL projection, while decoupling target choice from projection divergence.The analysis applies to K ≥ 2 group-based settings; forward and reverse KL are design choices, and experiments isolate projection mechanisms with paired temperatures.
  • C.2 Extensions and Future Directions: Future directions include step-level and off-policy projection, but current LPO requires K ≥ 2, while alternative divergences offer a distinct design axis.Single-sample relaxations are exploratory and may preserve boundedness while sacrificing zero-sum gradients.

C.4 Listwise vs. Pointwise Projection … D.2 Models

The paper frames LPO as normalized target projection on the response simplex, yielding coupled, zero-sum, bounded, and self-correcting updates while extending naturally from pairwise preference optimization to general divergences. Experiments span multiple reasoning tasks, reward designs, datasets, and eight released model families, with broader implications for efficient and capable LLM post-training.

  • C.4 Listwise vs. Pointwise Projection: LPO projects the reward-weighted Gibbs target on a normalized response simplex, unlike pointwise projection, producing coupled two-sided updates that suppress over-allocated and boost under-allocated responses.Its forward-KL coefficient is c_k = Pθ,k − w∗_k, whereas pointwise updates treat responses independently and are one-sided.
  • C.4 Listwise vs. Pointwise Projection: LPO’s listwise coefficients sum to zero, satisfy ∑k |c_k| ≤2, and vanish as Pθ approaches w∗, providing variance control, bounded steps, and self-correcting convergence.Pointwise projection instead has ∑k cpoint_k = −1 and lacks the same relative scaling and termination behavior.
  • C.5 Connection to DPO and Preference Optimization: When K = 2, LPO becomes a pairwise objective related to DPO, sharing a sigmoid structure but differing in online training, trust-region targets, and explicit divergence projection.LPO extends from pairwise preferences at K = 2 to listwise optimization at K > 2 and the population-level RL-as-inference limit as K →∞.
  • C.5 Connection to DPO and Preference Optimization: Unlike ranked-data methods such as LiPO, LPO uses absolute rewards for online RLVR target projection without ranking-model assumptions.The distinction is between Plackett–Luce-style preference optimization and Gibbs target construction from reward signals.
  • C.6 Extension to General Divergences: The projection framework applies to any differentiable divergence on the probability simplex, including general f-divergences such as Jensen–Shannon divergence.The zero-sum coefficient identity follows from the softmax parameterization, while boundedness and mode-seeking behavior depend on the divergence choice; KL is used as a stable default.
  • C.7 Entropy Regularization and Reverse KL Diversity: Reverse-KL LPO is equivalent to a maximum-entropy RL objective, while adding γH(πθ) mixes the target as ˜w∗ = softmax(R/(τ + γ)) in the on-policy setting.The entropy bonus is redundant when τ is a controllable hyperparameter.

D.3 Training Details · E Extended Experimental Results · E.1 Scalability Validation

The experiments use matched group-based training configurations across four reasoning scenarios and assess LPO’s training dynamics, scalability, and extensibility. LPO is evaluated against representative policy-gradient baselines and on a large Polaris benchmark with Qwen3-14B-Base.

  • D.3 Training Details: Training compares LPO with GRPO, Dr.GRPO, and MaxRL using K = 8 sampled responses per prompt across four reasoning scenarios.Generation uses temperature 1.0, top p = 1.0, top k = −1.0, and no KL penalty.
  • D.3 Training Details: Math and PRIME-Code use batch sizes 256/128 with 4096-token responses, while Countdown and Geometry use 128/64 with 1024-token responses.The configurations perform two gradient updates per iteration.
  • D.3 Training Details: Optimization uses Adam at learning rate 1e−6, β = (0.9, 0.999), weight decay 0.1, and clipping parameter ϵ = 0.2 across tasks.Token-level log-density ratios are additionally clipped to [log(1−ϵ), log(1+ϵ)].
  • D.3 Training Details: All experiments run on 8 NVIDIA H20 GPUs.
  • E Extended Experimental Results: Training-dynamics analyses compare LPO variants with Dr.GRPO and MaxRL using response entropy, gradient norms, and response lengths.Figure 9 covers Dr.GRPO, while Figure 10 covers MaxRL.
  • E.1 Scalability Validation: On approximately 53k Polaris reasoning problems with Qwen3-14B-Base, LPO-fwd reaches GRPO’s peak performance at 70 steps versus 200 training steps for GRPO.The experiment compares both LPO variants with GRPO and reports improvements in Pass@1 and an additional metric whose value is truncated in the supplied passage.
  • E.1 Scalability Validation: Additional scalability analyses examine LPO generalization across Qwen, DeepSeek, Mistral, and Llama backbones on Countdown, plus a fully on-policy regime with one gradient update per iteration.

E.2 Extended Training Dynamics … E.5 Evaluation Results

Across supplementary experiments, LPO shows stable optimization advantages, generalizes across LLM families, matches GRPO under fully on-policy conditions in its reverse variant, and improves exploration with its forward variant. Final evaluations extend the comparison to mathematics benchmarks and out-of-distribution reasoning tasks.

  • E.2 Extended Training Dynamics: LPO maintains higher response entropy, lower and more stable gradient norms, and longer response chains than Dr.GRPO and MaxRL.These extended dynamics corroborate the structural advantages of listwise projection across the compared baselines.
  • E.3 Generalization across LLM Families: Across Qwen, DeepSeek, Mistral, and Llama families, LPO consistently improves Countdown performance over PG, with especially stable gains under Pass@64.The evaluation includes base, distilled, and instruction-tuned models.
  • E.4 Fully On-Policy Optimization: Under fully on-policy training with batch and optimization mini-batch sizes of 256, LPOrev produces curves practically indistinguishable from standard GRPO.This setup performs exactly one gradient update per iteration and supports the theoretical connection between the objectives.
  • E.4 Fully On-Policy Optimization: LPOfwd shows higher sample efficiency during early training and achieves superior Pass@k accuracy under the identical fully on-policy setup.This result distinguishes LPOfwd through exploration rather than equivalence to standard GRPO.
  • E.5 Evaluation Results: Table 3 reports final mathematics-benchmark evaluations using standard k configurations, with Pass@1 and Pass@k averaged across benchmarks.Base denotes the backbone without RLVR; bold and underlined values mark the best and second-best results for each policy gradient baseline.
  • E.5 Evaluation Results: Table 4 compares LPO with counterpart policy-gradient baselines on OOD reasoning tasks after all methods are trained on MATH using Qwen3-8B-Base.The tasks are MMLU-Pro, ARC-c, and GPQA-diamond.
  • E.5 Evaluation Results: Specific LPO variants can improve the overall average across the reported OOD reasoning evaluations.The supplied passage indicates this improvement without specifying the affected variants or numerical values.

F Data Examples

This section documents the prompt templates and provides representative examples from the paper’s reasoning tasks. The examples specify required reasoning and answer formats, alongside ground-truth outputs where available.

  • MATH and Geometry3k use official verl prompt templates, while Countdown follows Pan et al. (2025) and PRIME code uses Cui et al. (2025).
  • A geometry prompt asks for the center-coordinate sum of a circle defined by diameter endpoints and requires the final answer in \boxed{}, with ground truth 0.
  • A Countdown prompt asks users to form 35 from [2, 54, 17], using each number once and returning work in <think> and <answer> tags.
  • PRIME code examples require structured action-based reasoning, including assessment, verification, simplification, and synthesis, before producing the requested output.
  • Another example requires internal reasoning in <think> tags and the final answer in \boxed{}, with ground-truth answer 4.
Loading 2605.06139v2…