PaperlayerView exact revision on arXiv

Source-linked AI summary

No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping

Thanh-Long V. Le, Myeongho Jeon, Kim Vu, Viet Lai, Eunho Yang

arXiv:2509.21880v3cs.CLcs.AIcs.LG

TL;DR

RLVR methods can lose optimization signal on zero-variance prompts, where all sampled responses receive the same reward. RL-ZVP directly uses correctness and token entropy to shape advantages, improving performance over GRPO across six math benchmarks while exposing zero-variance prompts as useful training data.

  • Problem

    GRPO’s normalized advantages vanish when all responses for a prompt receive identical rewards, leaving zero-variance prompts unused for policy optimization.

  • Method

    RL-ZVP assigns advantage direction from response correctness and scales its magnitude with token-level entropy for zero-variance prompts.

  • Results

    RL-ZVP outperforms GRPO across six math reasoning benchmarks, with average gains of 4.00 points in Acc@8 and 4.28 points in Pass@8.

  • Takeaways & Limitations

    Zero-variance prompts can provide valuable learning signals for RLVR rather than being discarded or filtered out.

  • Takeaways & Limitations

    Experiments are limited to models up to 8B parameters and verifiable tasks with binary rewards; scaling and graded or ambiguous feedback remain open challenges.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) is a powerful framework for improving the reasoning abilities of Large Language Models (LLMs). However, current methods such as GRPO rely only on problems where the model responses to the same input differ in correctness, while ignoring those where all responses receive the same reward -- so-called zero-variance prompts. In this work, we argue that such prompts are not useless but can, in fact, provide meaningful feedback for policy optimization. To this end, we introduce Reinforcement Learning with Zero-Variance Prompts (RL-ZVP), a novel algorithm that extract learning signals from zero-variance prompts. RL-ZVP directly rewards correctness and penalizes errors even without contrasting responses, modulating feedback with token-level characteristics to preserve informative, nuanced signals. Across six math reasoning benchmarks, RL-ZVP achieves significant improvements of up to 8.61 points in accuracy and 7.77 points in pass rate over GRPO, while consistently outperforming other baselines that filter out zero-variance prompts. These results highlight the untapped potential of learning from zero-variance prompts in RLVR. The project page is available at https://bltnynk.github.io/publications/rl-zvp/.

1 INTRODUCTION

RLVR methods such as GRPO lose learning signals when all rollouts for a prompt receive the same reward. RL-ZVP instead uses zero-variance prompts through correctness-aligned, entropy-guided advantages and improves math reasoning performance.

  • Motivation and approach: GRPO’s normalized advantages collapse to zero when every response in a group receives the same reward, preventing those prompts from contributing to optimization.GRPO therefore relies on groups containing both correct and incorrect responses for nonzero contrasting feedback.
  • Motivation and approach: RL-ZVP extracts learning signals from zero-variance prompts instead of discarding groups whose rollouts are all correct or all incorrect.The method challenges filtering practices that exclude these prompts from RL training.
  • Method: RL-ZVP aligns advantage direction with response correctness and sets advantage magnitude using each sampled token’s entropy.This provides token-level modulation while preserving direct rewards for correct responses and penalties for incorrect responses.
  • Results: RL-ZVP improves over GRPO by 4.00 points in Acc@8 and 4.28 points in Pass@8 on average across six math benchmarks and two model scales.It achieves gains of up to 8.66 points in Acc@8 and 7.77 points in Pass@8.

2 PRELIMINARY: GROUP RELATIVE POLICY OPTIMIZATION

GRPO samples response groups, scores them against ground truth, normalizes rewards into advantages, and broadcasts each response-level advantage to its tokens. Its objective then uses importance-sampling ratios to optimize the policy over the dataset.

  • GRPO formulation: GRPO samples G responses for each question, compares them with the ground-truth answer, and produces final rewards for the group.The responses are sampled from the previous training policy.
  • GRPO formulation: GRPO normalizes group rewards to compute advantages instead of using a costly value model like PPO.These normalized advantages provide the response-level learning signal.
  • GRPO formulation: Each response-level advantage is broadcast identically to every token in that response.The formulation sets the token-level advantages equal across all positions.
  • GRPO formulation: The final GRPO objective uses importance-sampling ratios to account for the off-policy distribution of the training batch.The dataset-level objective aggregates the per-prompt policy objectives.

3 REINFORCEMENT LEARNING WITH ZERO-VARIANCE PROMPTS

RL-ZVP addresses GRPO’s zero-variance failure by assigning correctness- and entropy-informed token-level advantages, preserving GRPO elsewhere while extracting learning signals from prompts whose responses share one reward.

  • 3.1 MOTIVATION: ADVANTAGE VANISHING IN ZERO-VARIANCE PROMPTS: 30% to 99% of training batches can consist of zero-variance prompts, making them a substantial potential source of learning signal.This pattern was observed across two model-and-dataset settings and varies during training.
  • 3.1 MOTIVATION: ADVANTAGE VANISHING IN ZERO-VARIANCE PROMPTS: When every response in a group receives the same reward, GRPO advantages become zero and the objective discards the prompt’s learning signal.GRPO therefore provides no effective update for these prompts under its standard advantage formulation.
  • 3.1 MOTIVATION: ADVANTAGE VANISHING IN ZERO-VARIANCE PROMPTS: The approach challenges methods that filter zero-variance prompts by treating them as usable policy-optimization feedback rather than discarded rollout cases.Prior filtering practices motivated the paper’s central design choice, while the method’s direction and magnitude are specified separately.
  • 3.2 ADVANTAGE FORMULATION: RL-ZVP uses positive advantages for all-correct prompts and negative advantages for all-incorrect prompts, rewarding correctness and penalizing errors without requiring contrasting responses.It classifies zero-variance prompts by response correctness and uses the advantage sign to set update direction.
  • 3.2 ADVANTAGE FORMULATION: Token entropy scales zero-variance updates so higher-entropy tokens receive stronger positive reinforcement, while higher-entropy tokens in negative prompts receive smaller penalties.The formulation uses token-level entropy to vary update magnitude rather than treating every token identically.
  • 3.3 FINAL OBJECTIVE: RL-ZVP applies the shaped advantages through a training loop that verifies sampled responses, branches on zero variance, and updates the policy with the resulting objective.Non-zero-variance mini-batches use the standard GRPO advantages, whereas zero-variance mini-batches use the RL-ZVP formulation.
  • 3.3 FINAL OBJECTIVE: RL-ZVP introduces no additional loss term; it reformulates only GRPO’s zero-variance case while retaining the original objective otherwise.The method is therefore presented as a generalization of GRPO rather than a separate optimization objective.
  • 3.3 FINAL OBJECTIVE: The method generalizes GRPO exactly on non-zero-variance prompts but produces a non-trivial update instead of zero for zero-variance prompts.Entropy is detached during implementation so it acts as a scalar factor rather than a differentiable parameter.

4 EXPERIMENTS

Across two model scales and six math benchmarks, RL-ZVP is evaluated against GRPO and prompt-filtering baselines, showing stronger performance, stability, and reasoning behavior while ablations examine its components and scaling.

  • 4.2 RESULTS & ANALYSIS: RL-ZVP improves over GRPO by up to 8.61 points in accuracy and 7.77 points in pass rate across six math benchmarks.It achieves average gains of 2.84 Acc@8 and 4.62 Pass@8 points at small scale, and 5.15 Acc@8 and 3.95 Pass@8 points at large scale.
  • 4.2 RESULTS & ANALYSIS: RL-ZVP outperforms prompt-filtering baselines under equal rollout budgets and remains stronger despite their 3×–5× rollout advantage.On the large scale under equal rollout budgets, gains average 19.20 Acc@8 and 13.56 Pass@8 points over the filtering baselines.
  • 4.2 RESULTS & ANALYSIS: RL-ZVP produces more stable training dynamics, avoiding GRPO’s early large-scale performance collapse and maintaining smoother improvement across scales.On the 8B scale, GRPO drops sharply during the first 200 gradient steps, whereas RL-ZVP improves smoothly.
  • 4.2 RESULTS & ANALYSIS: RL-ZVP steadily reduces entropy while increasing response length, supporting longer and more confident reasoning across model scales.At small scale, response length grows substantially during training; at large scale, entropy stabilizes after an initial decline rather than collapsing.
  • 4.2 RESULTS & ANALYSIS: RL-ZVP-trained models exhibit more systematic reasoning strategies than GRPO-trained models, replacing shallow trial-and-error with structured exploration.Examples include checking solutions, exploiting symmetry, factoring directly, and using the Law of Cosines.
  • 4.3 ABLATION STUDY: Full RL-ZVP performs best in the component ablation, while α = 0.10 or 0.20 provides the strongest scaling balance.Performance drops sharply at α = 0.05 and α = 0.30, showing that both under-scaling and over-scaling can impair training.

5 RELATED WORK

RLVR methods often discard zero-variance prompts, either implicitly or through explicit filtering. Related work also explores prompt selection and token-level reward shaping to address these limitations.

  • RLVR methods discard zero-variance prompts either implicitly or explicitly, wasting computation and reducing data efficiency.
  • Online Prompt Selection: DAPO oversamples after rollout until batches contain only non-zero-variance prompts, while GRESO predicts and filters likely zero-variance prompts before rollout.
  • Fine-grained Reward Shaping: Recent RLVR methods broadcast sample-level advantages across response tokens, motivating token-level reward shaping based on token roles.

6 CLOSING REMARKS

RL-ZVP extracts learning signals from zero-variance prompts by rewarding correctness, penalizing errors, and scaling updates with token entropy. The authors report improvements over GRPO and other baselines while identifying broader extensions as future work.

  • RL-ZVP directly rewards correct responses and penalizes incorrect ones without requiring contrasting answers within the same group.
  • RL-ZVP scales updates by token entropy to elicit stronger reasoning behaviors in the policy model.
  • The experiments report significant improvements over GRPO and other strong baselines, while positioning zero-variance prompts as exploitable learning resources.
  • Limitations & Future Directions: Experiments are limited to models up to 8B parameters and verifiable tasks with binary rewards, leaving larger models and graded or ambiguous feedback for future study.

A FULL IMPLEMENTATION DETAILS

The implementation uses a shared verl-based pipeline with scale-specific hardware and evaluation schedules. Experiments follow mostly standard hyperparameters, with additional baseline-specific settings and documented configurations.

  • All methods are implemented on the same verl pipeline, using 4×RTX 3090 GPUs for Qwen3-1.7B-Base and 8×A100 GPUs for Qwen3-8B-Base.
  • Evaluation uses best results for some small-scale benchmarks and final checkpoints for others, with a different schedule enabled by greater large-scale training efficiency.
  • The common setup uses clip-higher with ϵhigh = 0.28 and ϵlow = 0.2, token-level loss, and no KL-divergence term.
  • Baseline-specific configurations include GRESO’s probabilistic filtering parameters and omission of DAPO’s soft overlong punishment for GRPO-DS.
  • Table 5 provides the full hyperparameter settings used in the experiments.

B.1 ALTERNATIVE TRAINING DATA

The alternative-data experiment trains on a random 20k-prompt NuminaMath-CoT subset and compares RL-ZVP with GRPO. RL-ZVP consistently outperforms GRPO across the six benchmarks and shows lower entropy with longer responses during training.

  • The alternative experiment uses a random 20k-prompt NuminaMath-CoT subset and compares RL-ZVP with GRPO under the main experimental setup.
  • Table 6 reports the six-benchmark evaluation results for NuminaMath-CoT training, with corresponding accuracy and entropy dynamics shown in Figure 6.
  • RL-ZVP consistently outperforms GRPO across all six benchmarks when trained on the alternative NuminaMath-CoT dataset.
  • RL-ZVP’s training dynamics show lower entropy and longer response lengths than GRPO throughout training.

B.2 OUT-OF-DOMAIN EVALUATION

RL-ZVP was evaluated beyond math reasoning and beyond the Qwen model family. It maintained higher average performance than GRPO and other baselines across most tested settings.

  • The out-of-domain evaluation covered coding, general knowledge, scientific reasoning, and logical reasoning benchmarks.
  • The cross-family evaluation compared RL-ZVP with GRPO and prompt-filtering baselines using the same experimental setup.
  • RL-ZVP achieved higher average performance overall than GRPO and other baselines on Llama-3.2-3B-Instruct across most benchmarks.

C MORE ABLATION STUDIES

Additional ablations show that RL-ZVP depends on a suitable scaling factor and an asymmetric advantage design. Too-small or too-large scaling factors weaken optimization, while enforcing symmetry harms performance.

  • C.1 SCALING FACTOR ABLATION: Setting α too small reduced gains over GRPO, whereas overly large α values destabilized optimization.
  • C.2 COMPARISON WITH A SYMMETRIC VARIANT OF RL-ZVP: RL-ZVP uses an intentionally asymmetric advantage formulation for zero-variance prompts rather than an exact opposite penalty for negative prompts.
  • C.2 COMPARISON WITH A SYMMETRIC VARIANT OF RL-ZVP: The symmetric RL-ZVP variant underperformed RL-ZVP across all benchmarks and fell below GRPO.

D FULL TRAINING DYNAMICS OF THE MAIN EXPERIMENTS

Full training dynamics show RL-ZVP outperforming prompt-filtering baselines while producing longer responses with more stable entropy. This advantage persists despite baselines receiving more rollouts under equal gradient-step comparisons.

  • D FULL TRAINING DYNAMICS OF THE MAIN EXPERIMENTS: At small scale, GRPO-DS and GRESO exceeded GRPO but remained below RL-ZVP, while their rising entropy suggested entropy explosion.
  • D FULL TRAINING DYNAMICS OF THE MAIN EXPERIMENTS: At larger scale, prompt-filtering baselines lagged significantly behind RL-ZVP and showed less response-length growth.
  • D FULL TRAINING DYNAMICS OF THE MAIN EXPERIMENTS: RL-ZVP consistently and substantially outperformed prompt-filtering baselines despite those baselines receiving more rollouts under equal gradient steps.
  • D FULL TRAINING DYNAMICS OF THE MAIN EXPERIMENTS: The zero-variance prompt ratio was approximately 30% per batch on NuminaMath-CoT versus nearly 99% on DAPO-Math-17k at training start.

F QUALITIVATE EXAMPLES

Qualitative examples compare GRPO and RL-ZVP responses on astronomy, geometry, and polygon problems. The examples include intermediate reasoning and final answers, including cases where responses reach different numerical conclusions.

  • The examples include GRPO and RL-ZVP responses from Qwen3-1.7B-Base and Qwen3-8B-Base on multiple reasoning problems.
  • For the star-magnitude problem, the displayed final answers include approximately 20.39 and 87.954.
  • For the rectangular-box problem, a trial assignment is rejected because it gives ab + bc + ca = 47 rather than 27, while another response reports 673.
  • The polygon example reports a final value of m + n + p + q = 385 after presenting several attempted solution approaches.
Loading 2509.21880v3…