PaperlayerView exact revision on arXiv

Source-linked AI summary

SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning

Chenyi Li, Yuan Zhang, Bo Wang, Guoqing Ma, Wei Tang, Haoyang Huang, Nan Duan

arXiv:2602.01062v1cs.AI

TL;DR

RLVR can improve LLM reasoning while narrowing the range of generated solutions. SetPO adds a leave-one-out, kernel-based diversity signal to policy optimization, and the paper reports consistent improvements over strong baselines across model scales and benchmarks.

  • Problem

    RLVR reasoning gains can coincide with reduced outcome diversity, while token-level entropy and Pass@K do not directly capture semantic diversity across trajectories or guarantee Pass@1 gains.

    Source: §1 · pp. 1–2

  • Method

    SetPO measures semantic similarity across complete trajectories, assigns leave-one-out marginal diversity credits, and adds them as a weighted advantage-shaping term.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    SetPO consistently outperforms strong baselines across mathematical reasoning and combinatorial tasks and a wide range of model sizes.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The analysis supports using rarer trajectories as higher-contribution signals for increasing set-level diversity during policy optimization.

    Source: §1 · p. 2, §6 · p. 8

    The theoretical analysis assumes a measurable, symmetric kernel bounded between zero and one and a continuously differentiable, bounded-derivative, non-increasing shaping function.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards has shown notable effectiveness in enhancing large language models (LLMs) reasoning performance, especially in mathematics tasks. However, such improvements often come with reduced outcome diversity, where the model concentrates probability mass on a narrow set of solutions. Motivated by diminishing-returns principles, we introduce a set level diversity objective defined over sampled trajectories using kernelized similarity. Our approach derives a leave-one-out marginal contribution for each sampled trajectory and integrates this objective as a plug-in advantage shaping term for policy optimization. We further investigate the contribution of a single trajectory to language model diversity within a distribution perturbation framework. This analysis theoretically confirms a monotonicity property, proving that rarer trajectories yield consistently higher marginal contributions to the global diversity. Extensive experiments across a range of model scales demonstrate the effectiveness of our proposed algorithm, consistently outperforming strong baselines in both Pass@1 and Pass@K across various benchmarks.

1. Introduction

RLVR can improve reasoning while concentrating outputs on fewer solutions, and token-level entropy or Pass@K objectives may not capture semantic diversity or preserve Pass@1. SetPO adds a leave-one-out diversity signal to policy optimization; the paper reports consistent gains over strong baselines across model scales.

1. Introduction

  • RLVR performance gains have been associated with reduced policy entropy and trajectories recoverable from the base model under larger sampling budgets.These observations motivate preserving outcome diversity during post-training.Source: §1 · p. 1
  • Token-level entropy can favor longer responses and misses semantic variation across complete outputs, while Pass@K gains may not improve Pass@1.The section frames these as limitations of existing diversity proxies and repeated-sampling objectives.Source: §1 · pp. 1–2
  • SetPO assigns each rollout a leave-one-out marginal diversity credit and adds it as a weighted advantage-shaping term in policy optimization.Trajectories covering distinct semantic modes receive higher credits than redundant trajectories.Source: §1 · p. 2
  • In a toy multimodal bandit, SetPO keeps trajectories distributed across multiple clusters and covering correct modes at late training steps, unlike GRPO’s collapse to one cluster.The toy environment has twelve semantic modes, four of which yield positive reward.Source: §1 · p. 2, Figure 1 · p. 2
  • SetPO is reported to consistently outperform strong baselines across reasoning benchmarks and model scales from 1.5B to 32B.The introduction highlights the broad comparison, without providing metric values here.Source: §1 · p. 2

2. Related Works

Related work improves group-based policy optimization and encourages exploration using entropy or trajectory-level measures. SetPO instead evaluates diversity holistically at the sampled-set level using leave-one-out contributions.

Reinforcement Learning for LLM Reasoning

Diversity in Reinforcement Learning

3. Diversity-Aware Reward Calculation

The section defines semantic diversity over complete trajectories and estimates each sample’s contribution by removing it from the sampled set. SetPO plugs those contributions into group-based policy optimization as an advantage-shaping term.

3.1. Preliminaries: Group-based Policy Optimization

  • GRPO samples a group of G outputs per prompt and assigns each trajectory a reward used to form group-relative advantages without a learned value function.The group-based setup supplies the framework into which the diversity term is added.Source: §3.1 · p. 3

3.2. Diversity of a Language Model

  • SetPO measures complete-output semantic similarity with a bounded symmetric kernel over embeddings, then aggregates kernelized local mass using a non-increasing shaping function.Lower local mass corresponds to a sparser, more novel semantic neighborhood; the chosen shaping function is g(x) = −log(1 + x).Source: §3.2 · pp. 3–4

3.3. Leave-one-out Margin for a Single Trajectory

  • For each trajectory, the leave-one-out margin compares the diversity estimate for the full rollout group with the estimate after that trajectory is removed.A larger margin indicates a greater diversity gain from including the trajectory, while a small or negative margin suggests redundancy; the paper notes the computation is negligible for typical group sizes.Source: §3.3 · p. 4

3.4. Set-Level Policy Optimization Methods

  • SetPO adds each sample’s diversity score, weighted by λ, to its base advantage and substitutes the augmented advantage into the group-policy objective.The diversity plug-in modifies only the advantage term and is compatible with group-based objectives such as DAPO and GSPO.Source: §3.4 · p. 4

4. Theoretical Analysis

The theoretical analysis treats diversity changes under distribution perturbations and establishes that low-mass trajectories receive greater influence. It also characterizes the leave-one-out score as an exact, anti-redundant marginal contribution under the stated assumptions.

4.1. Population view

  • Under the kernel and shaping-function assumptions, a single-trajectory perturbation’s influence combines intrinsic novelty from local mass with an interaction penalty for similarity to existing trajectories.The analysis states that the objective follows diminishing returns and favors trajectories in low-mass regions.Source: §4.1 · pp. 4–5

4.2. Leave-one-out as marginal contribution

  • The leave-one-out score is the exact diversity decrease on removing a trajectory, and it decreases as that trajectory becomes more similar to any other set member.The paper further states that uniformly less-similar points rank higher and that the score aligns with increasing the diversity objective under the empirical distribution.Source: §4.2 · pp. 5–6

5. Numerical Experiments

The experiments evaluate SetPO across mathematical reasoning benchmarks, the Countdown task, and models up to 32B parameters. SetPO-augmented methods improve reported performance, while diversity evaluations also show broader output coverage.

5.1. General Mathematical Reasoning

  • On the Olympiad benchmark, SetPO+GRPO and SetPO+GSPO improve performance and reduce run-to-run variability.Figure 2 shows variance across runs as shaded regions.Source: Figure 2 · p. 6, §5.1 · p. 7
  • SetPO+DAPO reaches 53.8% average Pass@1, while SetPO+GRPO raises average accuracy from 47.2% to 52.8% (+5.6) and AIME24 accuracy from 15.7% to 21.6%.For 1.5B models, SetPO+DAPO achieves 46.7% average accuracy.Source: §5.1 · pp. 6–7

5.2. Countdown Evaluation

  • On Countdown, SetPO consistently exceeds GRPO across Pass@K settings, including 3- and 7-rollout configurations, temperatures of 0.5 and 1.0, and varying KL penalties.The reported advantage widens as k increases.Source: Figure 3 · p. 7, §5.2 · pp. 7–8

5.1. General Mathematical Reasoning

  • SetPO-enhanced methods receive higher diversity scores than baselines across 1.5B and 7B models, using a five-point assessment of generated answers.The AIME24 figure scores per-problem diversity from 1 to 5 using 16 Gemini-2.5-Flash-generated solutions.Source: §5.1 · p. 7, Figure 4 · p. 7

5.3. Scaling Up to 32B Models

  • On Qwen2.5-32B, SetPO+GRPO outperforms GRPO across all four mathematical benchmarks in Pass@1 and Pass@k, with almost 5 percentage-point Pass@1 gains on AIME24 and AIME25.The performance gap persists as sampling budget k increases.Source: Figure 5 · p. 8, §5.3 · p. 8

5.2. Countdown Evaluation

  • In the 7-rollout Countdown setting, SetPO+GRPO raises diversity width from 351 to 415 and average mode from 2.3 to 3.5.Diversity width counts problems with at least two different outputs; average mode is the mean number of distinct answers per question.Source: Table 3 · p. 8, §5.2 · p. 8

6. Conclusion

SetPO rewards trajectories by their leave-one-out contribution to rollout-set diversity, and the paper reports higher accuracy and maintained solution diversity across model sizes and task types.

6. Conclusion

  • SetPO consistently outperforms strong baselines, achieving higher accuracy while maintaining solution diversity across model sizes, mathematical reasoning, and combinatorial tasks.It estimates semantic density from trajectory similarity and incorporates each trajectory’s leave-one-out diversity credit into group-based policy optimization.Source: §6 · p. 8

Impact Statement

  • The paper frames its work as advancing reinforcement learning and states that it has no societal consequences requiring specific highlighting.This is the stated position in the impact statement.Source: Impact Statement · p. 9

A. Brief Introduction to DAPO and GSPO

  • For DAPO, the supplied passages describe a GRPO-style algorithm with clipped optimization and dynamic sampling; GSPO uses sequence-level importance ratios and clipping to align optimization with sequence-level rewards.The GSPO design is described as reducing variance and improving stability, especially for long responses and large-scale training.Source: A. Brief Introduction to DAPO and GSPO · p. 11

B. Our Algorithm

C.1. Preliminaries and Definitions

  • The quantity estimates the semantic density of distribution P around trajectory y; the canonical shape function used in this work is g(x) = −log(1 + x), which is smooth and strictly decreasing.The assumptions require a measurable, symmetric kernel bounded between 0 and 1, and a continuous, strictly decreasing shaping function with an L-Lipschitz derivative.Source: §C.1 · p. 12

C.2. Infinitesimal Analysis: The Influence Function

The influence analysis studies how adding a trajectory changes the diversity functional. It identifies rarity as a source of higher influence, with a local monotonicity result under stated conditions.

C.2. Infinitesimal Analysis: The Influence Function

  • The diversity influence function characterizes the first-order change in diversity when a trajectory is added to the distribution.The analysis considers the mixture perturbation P_ε = (1 − ε)P + εδ_τ as ε approaches zero.Source: §C.2 · p. 13
  • The trajectory-specific interaction term cooperates with intrinsic novelty to penalize redundancy when the shaping function is strictly decreasing.The reference-distribution baseline Ψ(P) is independent of the trajectory, so the uncentered score can suffice for ranking or gradient-based reweighting.Source: §C.2 · pp. 13–14
  • Under the local curvature and smoothness conditions, the influence ranking aligns with rarity when the remainder and kernel-tail terms are sufficiently small.For g(m) = −log(1 + m), the paper also reports bounded gradients on [0, 1], avoiding gradient blow-up in low-mass regions.Source: §C.2 · p. 15

C.3. Finite Perturbation Analysis and Error Bounds

The finite-perturbation analysis bounds diversity change and its approximation error. It finds that rarer trajectories have higher certified gain under the stated bound.

C.3. Finite Perturbation Analysis and Error Bounds

  • The finite change in diversity is bounded using a main signal and an error radius, with the error controlled by local redundancy and global average similarity.The passage defines κ(P) as the distribution’s global average similarity.Source: §C.3 · pp. 15–16
  • The analysis states that reducing local redundancy both increases the novelty signal and compresses the error radius, giving rarer trajectories a superior projected-gain trade-off.The certified lower bound is defined as J(τ, ε) = M(τ, ε) − E(τ, ε).Source: §C.3 · p. 16
  • The certified lower bound on diversity gain is strictly decreasing in a trajectory’s local mass for positive perturbation size.Thus, within this bound, lower local mass corresponds to higher certified gain.Source: §C.3 · p. 16

D. Discrete Analysis for Leave-One-Out Contribution si

The discrete analysis studies how removing one trajectory changes diversity scores. It establishes similarity-based monotonicity and ordering results for leave-one-out contributions.

D. Discrete Analysis for Leave-One-Out Contribution si

  • Theorem D.3 establishes strict monotonicity of a trajectory’s leave-one-out contribution with respect to its pairwise similarity to another trajectory.The proof notes that the reduced-set score excluding that trajectory does not depend on the relevant pairwise similarity.Source: D. Discrete Analysis for Leave-One-Out… · p. 17
  • When one trajectory is strictly less similar to every other point than another, its leave-one-out contribution is higher.This is the rare–redundant ordering under pointwise dominance.Source: D. Discrete Analysis for Leave-One-Out… · pp. 17–18
  • A positive weighted similarity advantage can yield a higher leave-one-out contribution even when pointwise dominance does not hold.The result provides a weighted characterization of the gap between the two contributions.Source: D. Discrete Analysis for Leave-One-Out… · p. 18
  • Removing one trajectory makes a neighboring trajectory denser when the removed point acts as its diluter; the neighbor score decreases under removal when their similarity is below its local mass.Source: D. Discrete Analysis for Leave-One-Out… · p. 18

E.1. Training Settings

Training settings define a semantic-similarity kernel for measuring answer overlap, while evaluation uses an unbiased Pass@k estimator with a fixed generation budget of 64.

E.1. Training Settings

  • SetPO training uses a cosine-similarity kernel over answer embeddings, clamped to [0, 1], with separate hyperparameter tables for GRPO, GSPO, DAPO, and larger-model or Countdown settings.The kernel compares output answers through their semantic embeddings.Source: §E.1 · p. 19, Table 4 · p. 19, Table 5 · p. 19

E.2. Evaluation Settings

  • Pass@k is estimated from a larger sample pool using the probability that a random subset of size k contains a correct solution, reducing variance relative to generating only k samples.The estimator is described as unbiased.Source: §E.2 · p. 19
  • Evaluation fixes the generation budget at n = 64 and tests k ∈ {1, 2, 4, 8, 16, 32, 64}; the estimate equals empirical accuracy when n = k and is 1 when n − c < k.Source: §E.2 · p. 21

F.1. Complete Comparison on Mathematical Benchmark

Across mathematical benchmarks, SetPO-augmented baselines show stable performance improvements and reduced variance, with gains persisting at larger sampling budgets. The paper also reports low training-time overhead and examples of varied solution strategies.

F.1. Complete Comparison on Mathematical Benchmark

  • SetPO consistently improves performance and reduces variance across sampling budgets for GRPO, GSPO, and DAPO, with results also reported for larger models.The comparisons use five random seeds; the figures show means and variance across runs.Source: §F.1 · p. 21, Figure 6 · p. 22, Figure 7 · p. 23

F.2. Reaching Higher Performance when K is sufficiently high

  • SetPO+GRPO substantially outperforms the base model on AIME24 across Pass@1 through Pass@256, including at K = 256.The comparison is presented as evidence that gains persist under high-pass evaluation.Source: §F.2 · p. 21, Figure 8 · p. 23

F.3. Timing Cost on Adding SetPO to Original Algorithms

  • Adding SetPO to GRPO, GSPO, or DAPO increases end-to-end runtime by less than 10%, and the paper says implementation improvements could reduce this overhead.Source: §F.3 · p. 21, Figure 9 · p. 24

G.1. Case Study: Diversity in Mathematical Reasoning Solutions

  • For an olympiad-style number theory problem, SetPO+GRPO produces distinct algebraic and constructive solution strategies that reach the same answer.Source: §G.1 · p. 21

Solution I

  • Solution I expresses a two-digit base-b number as n = ab + c, rearranges the b-beautiful condition, and reports 211 as the least qualifying base.Source: Solution I · p. 25

Solution II

  • Solution II searches over square roots for two-digit perfect squares, then iterates over bases and reports 211 as the least qualifying base.Source: Solution II · p. 25

G.2. Case Study: Diversity in Countdown Task Solutions

In the Countdown example, SetPO+GRPO generates multiple valid expressions for the target 63 using different arithmetic strategies and structural arrangements. The appendix also supplies task prompts and a strategy-based diversity scoring rubric.

G.2. Case Study: Diversity in Countdown Task Solutions

  • For target 63 from operands {54, 24, 21, 3}, SetPO+GRPO generates seven valid expressions with variation in both algebraic strategy and structure.Source: §G.2 · pp. 25–26
  • The examples include difference amplification using 3 × 3 + 54 and division-based decomposition into partial sums, alongside commutative rearrangements.The passage interprets the variation as evidence of diverse valid constructions.Source: §G.2 · p. 26

H. Prompts Used in this Paper

Prompt: AIME Solution Diversity Evaluator

EVALUATION CRITERIA

OUTPUT FORMAT

  • The evaluator prompt requires the score first in double brackets, followed by a brief reason.Source: OUTPUT FORMAT · p. 27

Related papers

Loading 2602.01062v1…