PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploration-Preserving Policy Optimization

Hangzhan jin, Mohammad Hamdaqa, Doina Precup

arXiv:2610.04011v1cs.AI

TL;DR

Group-relative objectives can make aggregate credit follow sampled mode frequency, while repeated-sampling utility depends on finding correct responses across samples. ExPPO redistributes credit using prompt-relative surprisal and pass rate; experiments report improved in-distribution coverage, competitive out-of-distribution generalization, and greater diversity among verified-correct responses.

  • Problem

    Group-relative objectives give equally rewarded responses equal advantages, making aggregate mode credit follow sample frequency, while repeated-sampling utility depends on finding correct solutions across multiple responses.

    Source: §1 · p. 1

  • Method

    ExPPO shapes group-relative advantages using bounded prompt-relative response surprisal and pass-rate gating, then applies shared absolute-advantage-mass normalization.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    Across three backbones, experiments show improved in-distribution coverage with competitive out-of-distribution generalization, sustained policy entropy, and increased diversity among verified-correct responses.

    Source: §7 · p. 9

  • Takeaways & Limitations

    ExPPO reuses existing rollout statistics and integrates into GRPO and DAPO with negligible computational overhead.

    Source: §7 · p. 9

    Evaluation covers three backbones, matched 900-step reasoning runs, and a four-method Phi-3.5-mini comparison with three 300-step training runs per method; larger models, broader benchmarks, and longer schedules remain uncharacterized.

    Source: B SCOPE AND LIMITATIONS · p. 14

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards improves reasoning, while the allocation of learning signal shapes which solutions remain accessible under repeated sampling. Group-relative objectives assign equal advantages to equally rewarded responses, making aggregate credit proportional to sampled mode frequency. We introduce Exploration-Preserving Policy Optimization (ExPPO), a lightweight advantage-shaping rule that redistributes credit using prompt-relative, length-normalized response surprisal and prompt pass rate. ExPPO combines bounded shaping with shared normalization to preserve verifier polarity and approximately maintain each prompt group's total absolute sequence-advantage mass. Our analysis characterizes response-level credit allocation alongside sampled mode updates, deriving local conditions for gains in entropy and correct-mode discovery. Experiments show improved in-domain and out-of-domain reasoning coverage, higher aggregate response accuracy, and strong coverage at large sampling budgets. A controlled multi-answer evaluation further demonstrates increased correct-mode yield and gains in diversity among verified-correct responses. Code is available at https://github.com/jinhangzhan/ExPPO

1 INTRODUCTION

The introduction argues that group-relative credit can make learning signal follow sampled mode frequency, limiting which responses remain accessible under repeated sampling. It introduces ExPPO as a prompt-relative redistribution rule and previews its analysis and evaluation of accuracy, coverage, and correct-mode diversity.

Motivation

  • Equal-reward responses receive equal sequence advantages, making aggregate mode credit follow sample frequency and giving rare correct modes fewer updates in finite groups.The introduction also notes that RLVR can improve pass@1 while reducing pass@K or answer diversity.Source: §1 · p. 1

Related work

  • The introduction positions prompt-relative control as a way to allocate each prompt group's learning signal across distinct responses.It contrasts that focus with approaches addressing entropy, collection-level utility, or rarity-aware credit.Source: §1 · p. 1

Method

  • ExPPO uses prompt-relative, length-normalized rollout surprisal, bounded shaping, pass-rate gating, and shared absolute-advantage-mass normalization.The rule preserves verifier polarity and assigns higher signed credit to higher-surprisal responses within each equal-advantage class.Source: §1 · p. 2

Analysis and evaluation

  • The paper characterizes sampled mode-level credit allocation and response-level allocation, with conditional local results for entropy and correct-mode discovery.Its experiments measure response accuracy, large-K coverage, entropy dynamics, and diversity among verifier-checked answer modes.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries describe group-based sampling from a frozen rollout policy, verifier-assigned sequence rewards, and the group-relative policy objective. They establish that each response's sequence advantage is shared across its tokens.

Reinforcement learning with verifiable rewards

  • For each prompt, a frozen rollout policy independently samples G responses, a verifier assigns sequence rewards, and the current policy learns from them.A response is represented as tokens with a realized length and token context.Source: §2 · p. 2

Group-relative policy optimization

  • GRPO assigns a detached sequence advantage to each response, and the same advantage applies to every token in that response.The text introduces prompt/group sampling and token-averaging notation for the objectives.Source: §2 · p. 2
  • The GRPO objective uses a clipped surrogate and reference-policy KL term; GRPO-H adds an entropy term.The passage identifies the importance ratio, clipped surrogate, reference-policy KL, and Shannon entropy.Source: §2 · p. 2

3 FROM REWARD CREDIT TO SUPPORT ALLOCATION

This section links equal per-response reward credit to multiplicity-weighted mode updates and distinguishes global policy entropy from diversity among correct responses. It argues that correct-mode diversity supplies information not captured by global entropy alone.

Mode-level credit

  • Equal per-response credit makes a correct mode's aggregate positive coefficient proportional to its sampled multiplicity.The mode-level update converts count differences between correct modes into relative log-odds shifts, and rare-correct trajectories also enter fewer finite-group updates.Source: §3 · p. 2

Entropy and correct-mode diversity

  • Global entropy can grow through failed continuations because entropy bonuses favor unlikely tokens regardless of response correctness.The entropy chain rule separates uncertainty over success, diversity among correct responses, and diversity among failures.Source: §3 · p. 3
  • Two policies can share global entropy and total correct mass while distributing probability differently across valid modes, so correct-mode diversity complements global entropy.The example contrasts uneven and even allocation across two correct modes while holding success probability at 0.5.Source: §3 · p. 3

4 EXPLORATION-PRESERVING POLICY OPTIMIZATION

ExPPO shapes verifier advantages using prompt-relative response surprisal and sampled pass rate, then applies shared normalization. The rule bounds shaping to preserve verifier polarity and is designed to keep prompt-group credit comparable to the base rule.

Prompt-relative response surprisal

  • ExPPO scores each response using length-normalized rollout-policy surprisal standardized within its prompt group and bounded by tanh.The score indicates whether a response's surprisal is above or below the group mean; length normalization controls length effects and group standardization gives a prompt-relative scale.Source: §4 · p. 3

Pass-rate-gated advantage shaping

  • The pass-rate gate sets the strength and direction of surprisal shaping, with bounded corrections preserving each advantage's sign.Low pass rates emphasize rare alternatives; high pass rates reduce repeated reinforcement and strengthen penalties on common errors.Source: §4 · p. 3

4 EXPLORATION-PRESERVING POLICY OPTIMIZATION

  • Table 1 summarizes response-level credit allocation before shared normalization, with rarity defined relative to the prompt.The gate emphasizes rare responses in low-success groups and common responses in high-success groups; bounded shaping preserves verifier sign.Source: Table 1 · p. 3

Shared normalization and integration

  • Shared absolute-mass normalization approximately matches the base prompt group's total absolute sequence-advantage mass while preserving response ordering.The method substitutes the shaped advantage into GRPO and retains DAPO's clipping, dynamic sampling, token aggregation, and overlong-response treatment while reusing rollout log-probabilities.Source: §4 · pp. 3–4

5 ANALYSIS

The analysis connects prompt-relative surprisal shaping to response-level credit allocation and local changes in entropy and correct-mode discovery. Its guarantees preserve verifier polarity, while exploration gains depend on the modeled setting and alignment conditions.

Bounded polarity-preserving redistribution

  • For 0≤λ≤1, shaping preserves base verifier-advantage polarity and bounds each response multiplier to [1−λ,1+λ].For λ<1, every nonzero shaped advantage strictly retains its base sign; relative multiplier spread is bounded by κ(λ)=(1+λ)/(1−λ).Source: §5 · p. 4

Surprisal ordering and response-space odds

  • Among equal-credit responses, EXPPO gives higher-surprisal responses greater shaped credit and shifts local odds toward them.The ordering holds for a mixed group with nonzero base advantage and λ>0; the odds result is within the stated KL-local family.Source: §5 · pp. 4–5

Surprisal alignment, entropy, and matched-KL utility

  • The entropy analysis separates the reward-advantage direction from EXPPO’s additional contribution, which aligns with length-normalized surprisal.The paper distinguishes full-response surprisal in its response-entropy characterization from logged next-token entropy as a complementary training diagnostic.Source: §5 · p. 5

Correct-mode discovery

  • Positive residual alignment raises local matched-KL expected correct-mode yield for sufficiently small positive λ; under binary rewards, correctness utility has zero residual alignment.For K≥2, the correct-mode utility weights low-mass correct modes more heavily, introducing variation across correct modes.Source: §5 · p. 5

6 EXPERIMENTS

Experiments assess response accuracy, repeated-sample coverage, and correct-mode diversity across three backbones. ExPPO improves coverage across ID and OOD evaluations, with additional gains in large-sample and code-reasoning settings.

6.1 EXPERIMENTAL SETUP

  • The evaluation uses eight benchmarks, four ID and four OOD, with response accuracy and pass@k coverage averaged across each suite.Training uses DAPO-Math-17k prompts; core comparisons share a 900-step schedule and evaluation at step 900.Source: §6.1 · pp. 5–6

6.2 PERFORMANCE DURING TRAINING

  • ExPPO finishes highest across all six model–suite trajectories and is the only core method to exceed base-model OOD coverage on every backbone during training.GRPO and GRPO-H show OOD forgetting on Qwen2.5-7B and Phi-3.5-mini.Source: §6.2 · p. 6

6.3 ENTROPY DYNAMICS AND LARGE-SAMPLE COVERAGE

  • ExPPO retains higher entropy than GRPO on three models alongside stronger coverage; at the largest sampling budget, it matches or exceeds other trained core methods on both smaller backbones and three Qwen2.5-7B tasks.Qwen2.5-7B initialization remains competitive at the largest budgets.Source: §6.3 · p. 6, Figure 3 · p. 7

6.4 CONTROLLED MULTI-ANSWER EVALUATION

  • On synthetic coin-count prompts, ExPPO increases observed and effective correct-mode yields over GRPO and GRPO-H across backbones, with broader conditional diversity on Qwen2.5-7B.Qwen3-0.6B gains diversity at near-saturated accuracy; REPO-R has a similar diversity point estimate and lower yields.Source: §6.4 · p. 7

6.5 COMPARISON WITH THE CLOSEST BASELINES

  • ExPPO leads both ID and OOD coverage means and six task means, tying F-GRPO on AIME25; at k=128, Phi-3.5-mini macro coverage is 51.03 versus 44.34 for GRPO-UR, 40.30 for REPO-R, and 12.17 for F-GRPO.The closest-baseline comparison also reports three-seed Phi-3.5-mini results separately.Source: §6.5 · p. 8, Figure 4 · p. 9
  • Across three independent Phi-3.5-mini training seeds, ExPPO leads ID coverage and response accuracy; REPO-R closely matches OOD coverage with less variation.GRPO-UR leads OOD accuracy but has lower, more variable coverage; F-GRPO has the lowest suite means and greatest coverage variation.Source: §6.5 · p. 8, Table 4 · p. 9

6.6 BEYOND MATHEMATICS: CODE REASONING

  • On CodeReasoner-derived Python execution reasoning, ExPPO leads both code metrics and OOD means, with OOD response accuracy 8.92 points above GRPO-H.The comparison uses Phi-3.5-mini and extends the reported accuracy–coverage benefit beyond mathematical training.Source: §6.6 · p. 9, Table 5 · p. 9

7 CONCLUSION

ExPPO redistributes response-level credit to support exploration, with experiments reporting broader coverage and correct-response diversity. The related work distinguishes this prompt-relative allocation from entropy-control, set-level, and other rarity-aware approaches.

7 CONCLUSION

  • ExPPO supports exploration through bounded prompt-relative surprisal, pass-rate gating, and shared advantage-mass normalization, with improved coverage and correct-response diversity across evaluations.The conclusion also reports competitive out-of-distribution generalization and negligible computational overhead.Source: §7 · p. 9

A RELATED WORK

  • Prior RLVR studies report higher pass@1 alongside lower large-K coverage, narrower answer support, and greater dependence on behaviors already present in the base policy.ExPPO instead examines credit allocation across sampled responses within a prompt.Source: A RELATED WORK · p. 14
  • Entropy-control and set-level methods address exploration through policy uncertainty or collective response utility, whereas ExPPO reallocates response-level credit while approximately preserving prompt-group absolute sequence-advantage mass.Source: A RELATED WORK · p. 14
  • Rarity-aware methods target underrepresented responses or correct-support structure; ExPPO uses prompt-relative standardized surprisal, pass-rate gating, bounded corrections, and shared mass normalization.The controlled task measures correct-mode yield and conditional diversity among correct pairs, with comparisons to F-GRPO, GRPO-UR, and REPO-R.Source: A RELATED WORK · p. 14

B SCOPE AND LIMITATIONS

The evaluation covers three backbones, matched reasoning runs, and a four-method Phi-3.5-mini comparison. Larger models, broader benchmarks, and longer schedules remain outside the evaluation scope.

B SCOPE AND LIMITATIONS

  • The evaluation covers three backbones, matched 900-step reasoning runs, and a four-method Phi-3.5-mini comparison.Source: B SCOPE AND LIMITATIONS · p. 14
  • The Phi-3.5-mini comparison uses three training runs per method at 300 steps under limited computational resources.Source: B SCOPE AND LIMITATIONS · p. 14
  • Larger models, broader benchmark suites, and longer schedules would further characterize scaling with model capacity and optimization time.Source: B SCOPE AND LIMITATIONS · p. 14

C BINARY-REWARD GRPO AND MODE-LEVEL CREDIT

With binary rewards, equal-reward GRPO responses receive equal sequence advantages, so sampled mode multiplicity shapes aggregate credit and relative mode updates. The appendix also relates these updates to policy objectives and entropy-based exploration.

C BINARY-REWARD GRPO AND MODE-LEVEL CREDIT

  • A correct mode appearing nm times receives aggregate positive GRPO coefficient proportional to nm, so a larger count yields a larger relative log-odds update than a less-sampled correct mode.The mixed-group coefficient includes the sampled pass rate through (1 − ρq)/ρq.Source: C BINARY-REWARD GRPO AND MODE-LEVEL… · pp. 14–15
  • In the shared logit geometry, shaping adds an allocation contribution that can shift relative odds toward a rare mode compared with the shared-scale base update.The sampled logit step is an exponential tilt, and setting h = e^A evaluates the response-level allocation in that family.Source: C BINARY-REWARD GRPO AND MODE-LEVEL… · p. 15
  • At the old policy, response-level shaping changes the prompt gradient through a combination of the original advantage gradient and the shaping term, retaining parameter sharing through response score vectors.Response averaging uses weights 1/(GTi); token-mean aggregation instead uses the reciprocal of total generated tokens in the loss batch.Source: C BINARY-REWARD GRPO AND MODE-LEVEL… · p. 15

D.1 GRPO AND GRPO-H OBJECTIVE DETAILS

  • The GRPO objectives use sequence advantages with token importance ratios and a reference-policy KL penalty; response averaging and token-mean aggregation are alternative conventions.The implementation uses token-mean aggregation, while response-level shaping applies with either convention.Source: §D.1 · pp. 15–16

D.2 ENTROPY BONUSES IN MUJOCO AND THEIR CONNECTION TO RLVR

  • Entropy bonuses can promote uncertainty independently of verifier credit, including tokens in failing continuations; their task effects depend on initialization, dynamics, and coefficient scale.Trajectory-level entropy can credit earlier actions for reaching high-entropy states, while direct entropy acts at sampled states.Source: §D.2 · p. 16

E EXPERIMENTAL DETAILS

The experimental details specify shared training configurations, checkpoints, evaluation measures, and closest-method baselines. ExPPO and GRPO occupy the same end-to-end compute band, while the other profiled methods differ in speed but remain closely matched in peak memory.

E EXPERIMENTAL DETAILS

  • Training uses shared checkpoints per model and a fixed 17,398-prompt dataset, with 16 rollouts per prompt and a 900-step schedule for the main reasoning runs.The Phi-3.5-mini coefficient, component, and four-method comparisons use checkpoint 300; ExPPO uses λ = 0.2.Source: E EXPERIMENTAL DETAILS · p. 17
  • Standard evaluation uses temperature 0.7, top-p 0.95, and a 10,240-token cap; the closest-method Qwen2.5-7B comparison evaluates one step-900 checkpoint per method with three decoding seeds.Source: E EXPERIMENTAL DETAILS · p. 17
  • Evaluation reports avg@k response accuracy across eight benchmarks and separates in-domain from out-of-domain files; large-sample coverage counts prompts with at least one verified success in each ordered response prefix.The maximum displayed budget is 256 for AIME24/25 and 128 for Minerva/OlympiadBench.Source: E EXPERIMENTAL DETAILS · pp. 17–18, Table 6 · p. 18

E.1 MEASURED TRAINING COST

  • Training-cost profiles run for 100 steps, with throughput and per-GPU peak memory measured over steps 11–100 and estimated GPU-hours scaled to the full eight-GPU job.GRPO-H uses an unchunked entropy-loss profile without activation checkpointing.Source: Table 7 · p. 18
  • ExPPO and GRPO occupy the same end-to-end compute band; GRPO-UR and F-GRPO are faster, GRPO-H takes longer, and all six methods have closely matched peak memory.Source: §E.1 · p. 18

F OPTIMIZATION IMPLEMENTATION

ExPPO reshapes response-level advantages using existing rollout quantities and can be integrated into both GRPO and DAPO. The notation distinguishes realized group statistics from population response quantities used in analysis.

F OPTIMIZATION IMPLEMENTATION

  • ExPPO transfers to GRPO and DAPO by substituting its shaped sequence advantages while leaving each base objective’s other mechanisms intact.DAPO retains asymmetric clipping, dynamic sampling, token-level aggregation, and overlong-response shaping.Source: F OPTIMIZATION IMPLEMENTATION · pp. 18–19
  • For one prompt group, the algorithm computes base advantages and success indicators, derives surprisal and pass-rate gates, shapes and normalizes advantages, then optimizes the base loss.The normalized sequence advantage is broadcast across response tokens.Source: F OPTIMIZATION IMPLEMENTATION · p. 19

G NOTATION AND PROBABILITY CONVENTIONS

  • The analysis distinguishes observed group pass rate ρq from population success probability pq, and freezes realized group statistics when varying a response distribution.The local analysis uses the finite response space induced by the generation length limit.Source: G NOTATION AND PROBABILITY CONVENTIONS · pp. 19–20
  • The method’s main quantities are length-normalized surprisal si, bounded relative surprisal ui, gate gi, correction Bi, and normalized advantage eAi.The notation table identifies Bi as |Ai|giui and eAi as the normalized shaped advantage.Source: Table 8 · p. 19, G NOTATION AND PROBABILITY CONVENTIONS · p. 20

H PROOFS

The proofs characterize how ExPPO reallocates response credit and how that allocation changes local objectives. The guarantees depend on alignment and admissible shaping conditions rather than asserting universal gains.

H PROOFS

  • For λ < 1, shaping changes each nonzero advantage by a bounded multiplier in [1 − λ, 1 + λ], while shared nonnegative normalization preserves sign and ordering.The within-group largest-to-smallest multiplier ratio is at most (1 + λ)/(1 − λ).Source: H PROOFS · p. 21
  • Within an equal-advantage class, the shaping score strictly increases with relative surprisal, and shared normalization preserves that response ordering.The score is increasing on both sides of zero and meets continuously at zero.Source: H PROOFS · p. 22
  • For correct-mode discovery, the local tilt slope is Cov(VK, h); for correct-pair diversity, it is Cov(VD, h), with the latter measuring redistribution among correct modes.For K ≥ 2, discovery utility varies across correct modes when their masses differ.Source: H PROOFS · p. 22
  • Positive residual alignment between utility and the correction increases the leading matched-KL utility gain under the stated positive-base-alignment and finite-λ conditions.For each admissible λ, the result gives a positive gain over the base update at sufficiently small KL budgets.Source: H PROOFS · p. 23
  • When Cov(S, B) is positive, the first-order entropy direction is nonnegative above the stated threshold, and overlapping intervals can satisfy both entropy and target-utility conditions.These are conditional local results, not unconditional entropy gains.Source: H PROOFS · p. 23

J.1 COEFFICIENT SENSITIVITY AND COMPONENT ABLATIONS

Across ablations, baselines, adaptation settings, and probes, ExPPO generally improves coverage, with tradeoffs and scope boundaries in some comparisons. Controlled evaluations also examine correct-mode diversity and normalized credit allocation.

J.1 COEFFICIENT SENSITIVITY AND COMPONENT ABLATIONS

  • Every swept ExPPO coefficient beats matched GRPO coverage, but ID coverage peaks at λ = 0.2 and OOD coverage at λ = 0.75.Full ExPPO also has the highest ID and OOD means among component variants; positive-only shaping favors ID, while negative-only favors OOD.Source: §J.1 · p. 24, Table 9 · p. 24

J.2 CLOSEST BASELINES AT THE PHI-3.5-MINI ABLATION BUDGET

  • 46.00 overall coverage makes ExPPO the highest-coverage method, ahead of REPO-R (44.14), GRPO-UR (42.21), and F-GRPO (28.39).ExPPO leads on five benchmarks; REPO-R leads on Minerva, GPQA, and TruthfulQA.Source: §J.2 · p. 25, Table 10 · p. 25

J.3 INDEPENDENT-RUN STABILITY ON PHI-3.5-MINI

  • ExPPO raises mean coverage on all eight tasks and both domain averages across independent Phi-3.5-mini runs, with little ID-coverage variation.GRPO is more sensitive to the training trajectory, while response-accuracy results show the same macro pattern.Source: §J.3 · p. 27, Table 13 · p. 27

J.4 CODE-REASONING ADAPTATION AND OOD GENERALIZATION

  • After CodeReasoner-derived Python execution-reasoning adaptation, ExPPO finishes highest on both code metrics and every OOD column.Its OOD response accuracy is 8.92 points above GRPO-H.Source: §J.4 · p. 28, Table 14 · p. 28

J.5 PER-BENCHMARK CHECKPOINT TRAJECTORIES

  • ExPPO finishes strongest on most per-task checkpoint panels, while GRPO-H loses mathematical coverage on Qwen3-0.6B and Phi-3.5-mini and is unstable on several OOD tasks.Figures 5 and 6 disaggregate the ID and OOD coverage trajectories by task.Source: Figure 5 · p. 28, §J.5 · p. 29, Figure 6 · p. 29

J.6 DAPO-BASED EXPPO

  • Under DAPO, ExPPO raises both domain macros above DAPO and initialization, leading DAPO on seven of eight tasks.The largest recovery is on Safety, where DAPO loses most of its initial accuracy and ExPPO preserves it.Source: §J.6 · p. 30, Table 15 · p. 30

J.7 CONTROLLED MULTI-ANSWER TASK

  • On the controlled multi-answer task, ExPPO raises accuracy and both correct-mode yields over GRPO-UR.The evaluation uses 64 fixed prompts and K = 128 samples per prompt; each distinct verified count vector is a semantic answer mode.Source: §J.7 · pp. 30–32, Table 18 · p. 32

J.8 COMMON-CHECKPOINT CONTINUATION AND NORMALIZED CREDIT ALLOCATION

  • In shared-checkpoint continuation, ExPPO has the highest post-update point estimates for accuracy and diversity and the smallest observed diversity decrease, but paired diversity intervals span zero.The category audit reports an approximately −0.001 change in absolute-advantage mass after normalization, at the reported precision.Source: §J.8 · p. 33, Table 19 · p. 33, Table 20 · p. 33

Related papers

Loading 2610.04011v1…