PaperlayerView exact revision on arXiv

Source-linked AI summary

Parameter Exploration for RLVR via Variational Learning

Vatsal Venkatkrishna, Nico Daheim, Iryna Gurevych

arXiv:2608.09805v1cs.LGcs.AIcs.CL

TL;DR

RLVR can stall when self-generated rollouts lack diverse, informative reward groups. This paper introduces parameter-space exploration through Perturbed Parameter Policy Optimization (3PO), finding that multiple sampled models consistently improve downstream performance over GRPO and action-space baselines.

  • Problem

    RLVR methods can stall when homogeneous rollout groups receive identical rewards, while action-space exploration cannot change token order and limits trajectory diversity.

  • Method

    3PO samples perturbed model weights from a learned posterior during rollouts, using strategies that vary parameter samples and rollout grouping to control exploration.

  • Results

    Multiple model samples consistently outperform GRPO and action-space baselines, with C3PO achieving the best average performance across both model families.

  • Takeaways & Limitations

    Parameter-space exploration provides an additional lever that improves RLVR performance while rescuing more zero-advantage groups and reducing malformed or incorrect rollouts.

  • Takeaways & Limitations

    M3PO and C3PO have approximately 1.5× higher wall-clock times than GRPO despite equivalent computational requirements in pseudocode.

Abstract

from arXiv · show

Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significantly impact downstream performance. Many existing methods control exploration in the action-space, for example, using temperature scaling. However, these methods cannot reorder tokens but only influence the variance in the output distribution. This limits exploration and can lead to divergence or stalled training. Here, we investigate parameter-space exploration, where rollouts are generated by sampling different policies from a posterior that may each explore different rollouts. Sampling less or more diverse policies is then a complementary control lever over exploration. We introduce a family of methods called Perturbed Parameter Policy Optimization (3PO) which use different sampling strategies and different rollout grouping for reward estimation. Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks show that these approaches consistently improve average downstream performance over standard GRPO at a near-identical FLOPs cost. Moreover, using multiple parameter samples consistently produces fewer zero-advantage groups and malformed or incorrect rollouts during training than GRPO and action-space baselines. Overall, our work presents evidence that parameter-space exploration can improve reinforcement learning for LLMs.

1 Introduction

The introduction motivates parameter-space exploration for RLVR because temperature scaling cannot reorder tokens and may cause degeneration, then presents 3PO as learned-noise policy sampling and reports validation across reasoning and code tasks.

  • Motivation: Temperature scaling controls token-distribution variance but preserves relative token order, limiting the trajectories that can be explored.It uniformly changes vocabulary-token probabilities and can make erroneous tokens likely at high temperatures, causing degeneration.
  • Motivation: Parameter-space exploration perturbs weights, directly altering logits and potentially reordering token distributions.It can seek high-quality trajectories from policies near the current policy, mitigating action-space noise that samples unlikely or low-quality outputs.
  • 3PO: 3PO samples models from an approximate posterior during RLVR, updating it during training to adaptively learn noise injection.Each training step adds learned noise to model parameters, with one or more samples controlling the amount of exploration.
  • Evaluation: 3PO is evaluated on OLMo-3-1025-7B and Qwen2.5-Math-7B across mathematical reasoning and code generation tasks.The introduction reports improved downstream performance and faster convergence without more rollouts than AdamW-based training.
  • Results: C3PO achieves the best average performance across both model families.The introduction identifies C3PO as the chunked noising approach.

2 Background on RLVR and parameter exploration

RLVR trains LLMs from self-generated outputs scored by verifiable rewards, while GRPO derives learning signals from grouped rollout advantages. Because uniform-reward groups can stall learning and action-space exploration cannot change token rankings, the paper motivates parameter-space exploration using variational learning and controllable parameter noise.

  • RLVR: RLVR assigns rewards to self-generated sequences, typically using a verifiable ground-truth solution to score the final answer.The reward can be binary, such as r_i ∈ {0, 1}.
  • GRPO: GRPO samples G rollouts per problem and computes advantages from each reward relative to the group mean and standard deviation.The shorthand r = (r_1, …, r_G) denotes the group rewards.
  • Exploration challenge: Zero-advantage groups stall learning when all rollouts receive the same reward, making it difficult to balance failed and successful attempts.This reflects the exploration-exploitation tradeoff in sustaining a useful learning signal.
  • Exploration challenge: Action-space methods reweight or regularize token probabilities without changing policy parameters, whereas parameter exploration perturbs parameters as a complementary exploration axis.Temperature scaling is rank-preserving and requires careful tuning, while parameter-space exploration is compatible with prior action-space approaches.
  • Variational parameter exploration: Variational learning models a distribution over neural-network parameters, and IVON controls exploration by varying parameter noise through its effective sample size λ.Large λ reduces noise, while small λ increases parameter noise and prediction diversity.

3 Perturbed Parameter Policy Optimization

Perturbed Parameter Policy Optimization (3PO) adds parameter-space exploration through noisy policies, complementing existing action-space exploration methods. It studies three strategies that differ in how perturbations are sampled and how rollout groups are formed for advantage estimation.

  • Overview: 3PO introduces parameter-space exploration with noisy policies and remains compatible with temperature control, clip-higher, and entropy regularization.The method treats parameter-space exploration as an additional exploration axis rather than replacing existing approaches.
  • B3PO: B3PO samples one weight perturbation per batch and reuses it across prompts and rollout groups.This batched-noising design synchronizes the perturbation with the rollout engine, analogous to step-level temperature control in weight space.
  • M3PO: M3PO samples M perturbations, computes the GRPO loss for each, and averages gradients before updating the posterior once.Multiple Monte Carlo samples address cases where a single perturbation performs poorly or produces homogeneous rollouts.
  • C3PO: C3PO samples N weight perturbations with groups of size G/N, then computes advantages over the full group of G responses.Unlike M3PO, C3PO makes the advantage estimate reflect rollouts from N distinct weight samples.
  • Training stability: Seq-MIS was important for stabilizing training and ensuring convergence because gradient-estimation policies differ from the N rollout generators.Caching and replaying noise is an alternative, but it requires more memory.

4 Experiments

Experiments on OLMo-3-1025-7B and Qwen2.5-Math-7B compare 3PO with GRPO and action-space exploration baselines across mathematical reasoning and code generation. Parameter-space exploration generally improves performance and sustains learning by rescuing more zero-advantage groups while producing fewer malformed or incorrect rollouts.

  • Experimental setup: Experiments use OLMo-3-1025-7B and Qwen2.5-Math-7B with warm-start SFT followed by RLVR, primarily training with GRPO on DAPO-MATH-17k.Evaluation covers AIME 2024–26, AMC 2023, MATH-500, and Minerva using unbiased Pass@1 across K = 8 rollouts.
  • Mathematical reasoning: All 3PO variants outperform action-space exploration baselines on average, with gains concentrated on harder benchmarks.The compared action-space baselines are EntReg, Polaris, and KLCov, with effective group size fixed to G = 16 rollouts per batch.
  • Mathematical reasoning: Multiple model samples outperform single-model methods on average, while M3PO improves over B3PO but remains limited by reduced group size under equal compute.B3PO can improve individual benchmarks without meaningfully improving average performance beyond GRPO; relaxing equal-compute constraints further improves results.
  • Exploration analysis: 3PO rescues more zero-advantage groups than baselines, and M3PO and C3PO continue rescuing groups later in training.The evaluation defines a rescued group as one where the method produces at least one correct rollout while GRPO produces none; action-space methods fail to rescue more groups than they lose.
  • Exploration analysis: Action-space baselines produce more malformed or incorrect rollouts than GRPO: Polaris yields malformed outputs through temperature flattening, while EntReg obtains diversity mainly through incorrect rollouts.Parameter-space trajectories are sampled from plausible LLM policies, and IVON jointly learns the distribution over policies to improve trajectory quality.
  • Code generation: C3PO outperforms other methods on code-generation reward and accuracy, reaching an LCBv6 score of 15.17.The 3PO methods show a larger reward gap over GRPO on code generation than on mathematical reasoning, where the SFT model scores 9.5% on LCBv6 versus 35.02% on math benchmarks.

5 Further analysis

Further analysis finds λ = 109 to be a good default, while exploration diversity and unconstrained variance reduction improve results; learned SFT Hessians provide no clear advantage.

  • 5.1 ESS scaling: λ = 109 is a good default, whereas λ = 108 always hurts performance and collapses C3PO training.λ controls weight-space noise; λ = 1010 produces method-dependent behavior and can reduce the noise added during optimization.
  • 5.2 MC samples: Under equal compute, increasing M does not improve performance because variance reduction is roughly cancelled by reduced group diversity.M3PO uses M independently sampled weight settings while shrinking each group to G=16/M, holding total rollouts fixed.
  • 5.2 MC samples: Without equal-compute correction, M3PO steadily and substantially outperforms the corrected run and B3PO with G = 16.This indicates that reduced gradient-update variance benefits M3PO when compute is unconstrained.
  • 5.3 Chunks: All C3PO runs with N ≥2 outperform the single-perturbation baseline, with N = 2 −4 identified as a good strategy.Larger N improves earlier training steps but yields smaller gains in final scores; N =1 reduces to B3PO.
  • 5.4 Hessian initialization: The learned SFT Hessian provides no clear advantage over initializing h to a constant h0, with <8% of values outside [0.9h0, 1.1h0].The two Olmo3 C3PO configurations track each other closely, and the loaded Hessian is nearly isotropic at the end of SFT.

6 Conclusion

The paper introduces 3PO, a family of parameter-space exploration strategies for RLVR that samples weights from a learned posterior during rollouts and outperforms action-space baselines.

  • 6 Conclusion: 3PO samples weights from a learned posterior at rollout time, providing an additional lever for exploration in RLVR.The method is presented as a family of parameter-space exploration strategies.
  • 6 Conclusion: Drawing multiple model samples per gradient step consistently outperforms action-space baselines across the experiments.This establishes the central empirical advantage of parameter-space exploration in the reported RLVR experiments.
  • 6 Conclusion: 3PO rescues more zero-advantage groups and produces fewer malformed or incorrect rollouts than action-space baselines.These outcomes indicate improved rollout quality and fewer uninformative reward-estimation groups during training.

7 Limitations

The study is limited by compute and implementation constraints, preventing exploration of larger-model scaling and expanded test-time sampling. It also leaves more directed posterior estimators and alternative sampling strategies for future work.

  • Compute and implementation constraints: Compute limitations prevented testing whether 3PO’s benefits grow with model scale or whether larger models tolerate lower λ before training collapse.The authors identify larger-model scaling as an unexplored direction that could permit more exploration.
  • Compute and implementation constraints: Modern RL code stacks use a single model, making variational-learning changes difficult to implement.This implementation constraint further limited the study.
  • Unexplored extensions: Limited resources prevented scaling the number of MC samples and rollouts for test-time scaling.The study therefore does not evaluate expanded test-time sampling budgets.
  • Unexplored extensions: Non-diagonal posterior estimators and sampling strategies beyond Monte-Carlo sampling remain promising directions for more directed exploration.These alternatives were not explored in the study, which used IVON and Monte-Carlo sampling.

A Hyperparameter Settings · B Reward and entropy curves

The paper specifies on-policy RLVR and warm-start SFT settings, including sampling, optimization, reward, and clipping choices. Across both model families, the proposed methods converge faster than GRPO, with model-dependent entropy behavior.

  • A Hyperparameter Settings: Experiments use verl for training, vLLM for rollouts and evaluation, and one gradient update per prompt batch in an on-policy setup.Unless otherwise stated, evaluation uses τ = 0.6, top-p = 0.95, top-k = 50, K = 8, and an unbiased pass@k estimator.
  • A Hyperparameter Settings: Polaris raises sampling temperature from 1.0 to 1.4 when entropy reaches 85% of its original value.Thresholds of 75% and 50% either switched too late for recovery or never triggered; γ = 5e−3 completely collapsed EntReg training.
  • A Hyperparameter Settings: The ≈1.5× wall-clock overhead arises from systems costs, including unfused IVON kernels and inefficient multi-model sampling in vLLM.The passage attributes the latter to performing K inference operations and notes that more efficient sampling could reduce the overhead.
  • A Hyperparameter Settings: The warm-start SFT phase uses IVON with learning rate 50.0, weight decay 10−8, λ = 1010, β1 = 0.9, β2 = 0.9999, h0 = 0.001, and clipping radius 0.001.Training filters the Llama-Nemotron Post-Training Dataset to DeepSeek-R1 responses up to 4096 tokens and applies cosine decay with 10% linear warmup.
  • A Hyperparameter Settings: The RLVR phase uses token-level GRPO with PPO clip bounds of 0.2 and a binary correctness reward of +1/0.The supplied passage also specifies batch size and optimizer settings in the continuation of the RLVR configuration.
  • A Hyperparameter Settings: RLVR training uses 32 prompts with 16 rollouts each, one gradient step per rollout batch, and a constant learning-rate schedule with 6% linear warmup.AdamW uses learning rate 10−6 and weight decay 0.1; IVON uses learning rate 1.0 and λ = 109, while C3PO masks weights outside [0.5 −2.0].
  • B Reward and entropy curves: Across both model families, the proposed methods consistently converge faster than GRPO; for Olmo3, C3PO’s entropy declines rapidly.For Qwen2.5-Math, all methods maintain largely similar entropy profiles, while Olmo3’s decline suggests concentration on high-reward regions.

C Impact of scaling ESS (λ)

The weight-noise scale λ strongly affects stability and performance, with model- and method-specific sensitivity. λ = 109 is a good default for Olmo3, whereas Qwen2.5-Math C3PO favors larger values.

  • Olmo3: λ = 109 yields consistent gains for all three methods on Olmo3, while λ = 108 is generally unstable and C3PO completely collapses.Higher λ values generally outperform λ = 108 on Olmo3.
  • Qwen2.5-Math: Qwen2.5-Math is less robust to small λ than Olmo3, with C3PO showing especially erratic reward and entropy curves at low λ.C3PO samples a fresh model many more times during rollout generation, increasing its sensitivity to λ.
  • Qwen2.5-Math: For Qwen2.5-Math C3PO, the default λ = 109 is too noisy, while λ = 5 × 109 or 1010 would likely work best.B3PO behaves similarly across Olmo3 and Qwen2.5-Math.

D Ablating the number of Monte Carlo samples in M3PO · E Scaling the chunk size in C3PO

M3PO’s Monte Carlo sample count trades variance reduction against group diversity and compute, with fixed rollout budgets largely canceling the performance benefit. In C3PO, chunk size N > 1 sharply increases group diversity, while N = 2 captures almost all gains at marginally lower reward than larger chunks.

  • D Ablating the number of Monte Carlo samples in M3PO: Increasing M reduces gradient variance but proportionally increases compute, requiring smaller G to preserve the rollout budget and thereby reducing group diversity.Grouped-advantage methods such as GRPO rely on group diversity.
  • D Ablating the number of Monte Carlo samples in M3PO: All 3PO methods converge faster than vanilla GRPO, while entropy profiles remain largely similar for Qwen2.5-Math.Olmo3-C3PO’s entropy declines rapidly.
  • D Ablating the number of Monte Carlo samples in M3PO: Adding too much noise with small λ hurts performance, whereas large λ values can make sampled models too similar.The λ scaling curves cover all 3PO methods.
  • D Ablating the number of Monte Carlo samples in M3PO: Qwen2.5-Math is especially sensitive to small λ, where overexploration causes oscillating curves, making λ tuning important.These effects are shown in the Qwen2.5-Math λ-scaling analysis.
  • D Ablating the number of Monte Carlo samples in M3PO: Scaling M without shrinking the rollout group may improve performance, but it incurs proportional computational cost.Additional compute is required to realize the variance-reduction benefit.
  • D Ablating the number of Monte Carlo samples in M3PO: With fixed total rollouts, varying M ∈{1, 2, 4, 8} and shrinking G proportionally roughly cancels variance-reduction gains through lost group diversity.This trend holds on Qwen2.5-Math and mirrors the Olmo3 analysis.
  • E Scaling the chunk size in C3PO: Larger N yields marginally higher reward, but N = 2 captures almost all performance gains in C3PO.The reward and entropy trends mirror the pass@1 analysis.
  • E Scaling the chunk size in C3PO: For C3PO, N > 1 sharply increases group diversity, enabling continued discovery of high-reward regions of the solution space.The entropy trend matches Figure 6.

F Effects of a learned noise prior on other algorithms

Initializing the Hessian with a learned noise prior did not improve any evaluated 3PO method, and it even slowed B3PO convergence. The findings indicate that 3PO can be applied to off-the-shelf checkpoints without an IVON-based SFT calibration phase.

  • Learned noise prior: A learned Hessian prior did not improve performance for any evaluated 3PO method.The prior was expected to stabilize learning relative to initialization from scratch, but the experiments did not show that behavior.
  • Learned noise prior: For B3PO, the learned prior slowed convergence.
  • Learned noise prior: The results imply that 3PO can use off-the-shelf checkpoints without an IVON-based SFT phase to calibrate the noise distribution.

G Multi-seed robustness … K Algorithms

Across robustness checks and algorithmic ablations, parameter-space exploration remains effective across seeds, while learned noise, Seq-MIS correction, and joint action–parameter exploration are important for stability. The algorithms implement batched or chunked perturbations with GRPO-based advantage updates.

  • G Multi-seed robustness: C3PO improves over GRPO by an average of +0.89 points across six mathematical-reasoning benchmarks, positive on all three seeds, with paired-t p = 0.027.AIME’24 improves by +1.8 points (p = 0.023), with gains concentrated on harder benchmarks; only GRPO and C3PO were compared because repeated runs were computationally expensive.
  • H Isotropic initialization vs. learned noise: Matched-magnitude isotropic noise attains average accuracy 42.12, below GRPO’s 42.99 and full C3PO, eventually converging to a similar plateau.Its reward rises faster than GRPO early, whereas full C3PO plateaus above GRPO late in training.
  • I Ablating the Seq-MIS correction for C3PO: Removing Seq-MIS correction leaves C3PO’s training reward essentially flat, unlike the steadily rising reward of corrected runs.The correction addresses the training–inference mismatch caused by generating a rollout group from N distinct perturbed models while using the GRPO ratio.
  • J A Limiting case of C3PO: Greedily decoding each rollout from a freshly sampled model is very unstable, producing wildly oscillating rewards and worse performance than temperature-sampled C3PO.This limiting case uses N = G and temperature τ = 0.0; the most stable run at λ = 10^10 still performed markedly worse.
  • K Algorithms: B3PO and M3PO sample weight perturbations once per gradient step, generate all G rollouts from one rollout model, and accumulate losses over M Monte Carlo samples.M = 1 for B3PO; the posterior is initialized and updated with IVON around GRPO losses.
  • K Algorithms: The chunked method gathers G rollouts per prompt across N independent weight draws, with each perturbation generating G/N rollouts before computing buffered GRPO advantages.The accumulated response buffer is used for the GRPO objective with Seq-MIS correction, followed by an IVON posterior update.
Loading 2608.09805v1…