PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR

Tommy Sha, Skylar Zhai, Siqi Zhao

arXiv:2609.09075v1cs.LGcs.AI

TL;DR

RLVR prompt selection can waste rollouts on silent groups whose uniformly rewarded outcomes produce no reward-advantage signal, while history-based difficulty estimates are unavailable before initial target-policy rollouts. ThinkPrior uses a verifier-scored external anchor to initialize a Beta difficulty prior, then updates it from training outcomes without changing the loss or optimizer. Across sixteen seeds, it more than halves early silent groups and cuts wasted rollouts through step 30, with no detected final-accuracy difference; on the 250-prompt pool, the fixed-budget effect is a reallocation rather than a net saving.

  • Problem

    Cold-start prompt selection lacks per-prompt difficulty information before target-policy rollout history exists, despite silent groups consuming compute without reward-advantage signal.

  • Method

    ThinkPrior uses one verifier-scored external-anchor pass to initialize a per-prompt Beta difficulty posterior, selects by expected learnability, and updates from target-policy outcomes without changing the loss or optimizer.

  • Results

    Across sixteen seeds, silent@10 falls from 23.8% to 10.6% and wasted rollouts through step 30 by 19%, with no detected accuracy difference.

  • Takeaways & Limitations

    The evidence supports a cold-start benefit from the prior, while the measured ThinkPrior+DAPO composition shows a net generation reduction.

  • Takeaways & Limitations

    The original collapse endpoint remains unresolved: revised post hoc counts are exploratory and still not statistically significant, so they do not establish a stability result or causal attribution.

Abstract

from arXiv · show

In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from training outcomes, changing neither the loss nor the optimizer. On Qwen2.5-Math-7B across sixteen seeds, ThinkPrior more than halves early silent groups and cuts wasted rollouts through step 30 by nearly a fifth, while we detect no difference in final accuracy. On this 250-prompt pool the fixed-budget result is a reallocation rather than a net saving. The measured ThinkPrior+DAPO composition reduces generated rollouts by 10.6% while both arms retain the same 3840-rollout update budget. The prior requires no target-policy rollout before the first selection, but the posterior thereafter uses target-policy outcomes.

Introduction

RLVR’s group-relative reward-advantage signal disappears on uniformly rewarded groups, making prompt difficulty a costly selection problem at cold start. ThinkPrior uses an external verifier-scored anchor to initialize difficulty before target-policy rollouts, reducing early waste without changing training loss or optimizer.

  • Motivation: Silent groups provide no reward-advantage gradient because every rollout receives the same group-relative advantage.The studied KL-free objective therefore pays generation and verification costs without producing reward-advantage signal.
  • Motivation: History-based selectors cannot distinguish an unseen prompt at step 0 without target-policy rollout history or another signal.This creates the cold-start prompt-selection problem and associated early rollout waste.
  • Method: ThinkPrior initializes a per-prompt Beta difficulty posterior from one verifier-scored external-anchor pass before the first target-policy selection.Training then selects by expected learnability and updates the posterior using real target-policy outcomes.
  • Method: The zero-rollout prior changes neither the loss nor optimizer and can layer onto existing prompt-selection mechanisms.After training starts, target-policy outcomes update the posterior.
  • Results: 23.8% to 10.6%: silent@10 falls across sixteen seeds, while wasted rollouts through step 30 decrease by 19% with no detected accuracy difference.The reported early reductions measure cold-start efficiency rather than a final-accuracy improvement.
  • Results: ThinkPrior+DAPO shows a net generation reduction, while the fixed-budget result on the 250-prompt pool is a reallocation rather than a net saving.The composition retains the same update rollout budget while reducing generated rollouts by the measured comparison.

Related Work

Prior work explains uniformly rewarded groups as a consequence of group-relative baselines and offers several data-selection strategies. ThinkPrior differs by predicting verifier pass rate with an externally obtained, policy-independent signal whose cost is paid before training rollouts.

  • RLVR and group-relative objectives: Uniformly rewarded groups yield zero advantage under group-based baselines, including GRPO, RLOO, and related normalization or importance-ratio variants.This phenomenon is established across several group-relative objective variants.
  • Difficulty estimation: Item response theory provides a natural difficulty-scoring candidate, but its population-difficulty target differs from ThinkPrior’s policy pass-rate target.The paper evaluates a 2PL bank despite that difference.
  • Data selection: Curricula, learning-progress methods, and prioritization by learning potential establish a broader tradition of data selection for training.These approaches are mainly discussed in pretraining and instruction-tuning contexts.
  • Closest precedent: RHO-LOSS is the nearest precedent in cost structure, using one small reference model to prioritize examples that are learnable but not yet learned.RLVR instead selects for pass-rate conditions determining whether compute produces a nonzero reward-advantage term.

Method

ThinkPrior targets prompts likely to produce reward-advantage signal by combining an external-anchor Beta initialization with posterior expected learnability. It preserves the GRPO loss and optimizer while using target-policy outcomes for subsequent updates.

  • Silent groups: If C ∈{0, G}, all group-relative advantages are zero, so the KL-free reward-advantage contribution vanishes.The silent-group probability is s(p)=p^G+(1−p)^G, while learnability is U(p)=1−s(p).
  • Learnability: At G=8, learnability is 0.992 at p=0.5 versus 0.337 at p=0.05 or 0.95.Expected advantage mass is (G−1)U(p), so prompts near intermediate pass rates provide more expected signal-bearing yield.
  • Cold start: Uninformative Beta initialization makes posterior expected learnability constant across prompts, leaving first-step selection to tie-breaking.Prompt-dependent initialization is required to separate candidates before target-policy outcomes are observed.
  • External-anchor prior: ThinkPrior uses a verifier-scored external anchor to initialize each prompt’s Beta posterior before the first target-policy rollout.The anchor pass rate estimates prompt difficulty, while the online posterior later incorporates selected prompts’ training outcomes.
  • Selection rule: Training ranks prompts by posterior expected learnability, preferring better-known prompts at equal posterior means rather than maximizing uncertainty.The score is the posterior mean of learnability, not learnability evaluated at the posterior mean; finite posterior concentration creates a dispersion penalty.
  • Training integration: The prior changes only selection and posterior updates, leaving the loss and optimizer untouched while spending the same B·G rollouts per step as the online baseline.Its anchor pass is an additional cost: on the 250-prompt pool, k=16 requires 4000 small-model generations, and the comparison excludes verification, training, and orchestration overhead.

Experimental Setup

Experiments train Qwen2.5-Math-7B with GRPO and LoRA on a 250-problem MATH pool, comparing prompt-selection rules under a shared trainer. Evaluation covers final accuracy and cold-start rollout waste, with additional settings testing anchor and model-scale dependence.

  • Training configuration: The main setting trains Qwen2.5-Math-7B with GRPO and LoRA for 60 steps using B=8 prompts and G=8 rollouts per prompt.Each fixed-budget step therefore contains 64 rollouts.
  • Data and evaluation: The 250-problem training pool spans all five MATH levels and seven subjects, with evaluation on MATH500.The setup also evaluates dependence on a math-specialized backbone and a 1.5B policy scale.
  • Prior construction: The anchor is Qwen2.5-3B-Instruct, using 16 verifier-scored samples per prompt, with κ=4 throughout.A length-prior ablation instead uses Qwen3-0.6B chain-of-thought length.
  • Baselines: All arms share one trainer and differ only in prompt-selection rule, while online-NP uses the same Beta posterior and top-B rule with α=β=1.This design isolates the selection rule and makes online-NP the closest no-prior ablation.
  • Metrics: Reported metrics are final MATH500 and Level-5 accuracy, silent@10, and cumulative waste@30.Silent@10 covers the first 10 update steps; waste@30 counts rollouts in silent candidate groups through step 30.

Results

ThinkPrior consistently reduces early silent-group waste relative to no-prior or online-only selection, while accuracy remains unresolved or similar across the reported comparisons. Its composition with DAPO also lowers generated rollouts at the same update budget, but several comparisons limit causal attribution.

  • Main comparison: 0.125–0.167 versus 0.204–0.425: verifier-scored offline-prior arms have lower silent@10 than arms using online signals alone.Accuracy does not separate the families; reported MATH500 arm means span 0.533–0.605 with priors and 0.525–0.605 without.
  • Composition with DAPO: 65.4%: ThinkPrior+DAPO cuts DAPO waste@30 from 1565 to 541 at the same observed mean MATH500 accuracy of 0.605.The composition changes the candidate pool and therefore demonstrates compatibility with different selection machinery, not isolation of the initialization alone.
  • Robustness and scope: The related-configuration cells do not establish separate causal contributions because the no-prior and frozen comparisons change more than one factor at a time.Descriptively, the combined configuration reaches silent@10 0.138, compared with 0.246 for the frozen prior and 0.204 for the online learner without a prior.
  • Sixteen-seed reestimate: 19%: ThinkPrior reduces cumulative wasted rollouts through step 30, while MATH500 accuracy changes by only +0.68 points with an interval spanning zero.The fixed-budget arms both generate and use 3840 rollouts; the prior reallocates 63 rollouts away from early silent groups rather than reducing the update budget.
  • Robustness and scope: 16.6%: on the 1200-prompt pool, ThinkPrior discards 5085 rollouts on average versus 6099, although later waste and silence ranges overlap.On the smaller pool, the fixed-budget accounting is a reallocation, and the margin closes by step 40 without an identified mechanism.

Analysis

ThinkPrior’s selection results do not show an accuracy difference, but they do show lower early silence and wasted computation. The evidence remains bounded by unresolved collapse behavior and limited scope.

  • Analysis: A 1.5B anchor is sufficient for silent@10 near 0.10–0.11, whereas length leaves silent@10 at 0.217.The anchor-size result is inseparable from seed noise on accuracy.
  • Analysis: The original collapse endpoint is unresolved: the KL-free comparison gives 2/6 for online-NP versus 0/6 for ThinkPrior, with p=0.455.A post hoc relative-drop threshold changes the count to 4/6 versus 0/6, p=0.061, still not significant.
  • Analysis: The scope varies pool size and horizon directly, while the evidence claims no accuracy effect rather than parity.At n=16, neither accuracy difference is resolved, and one model family at pilot scale bounds the rest.

Conclusion

ThinkPrior supplies a zero-rollout difficulty prior for cold-start prompt selection under the studied KL-free objective. The evidence supports an early cold-start benefit, but bounds the fixed-budget result and broader claims.

  • Conclusion: ThinkPrior uses an external-anchor initialization to supply a zero-rollout difficulty prior, halving early silent groups without a detected accuracy difference.The supported claim is a cold-start benefit, not pool exhaustion, component-level causality, or a general stability effect.
  • Conclusion: On the 250-prompt pool, the fixed-budget result is a reallocation rather than a net saving.Only the measured ThinkPrior+DAPO composition shows a net generation reduction; code, complete data, and trajectories are not currently public.

Appendix A: Proofs

The proofs characterize silence, expected learnability, and advantage mass under binary verifier rewards. They establish why prompt-dependent initialization is necessary and why concentration is preferred at equal posterior means.

  • Proposition 1: A group is silent exactly when all rewards agree, with probability p^G + (1−p)^G; non-silence has probability U(p).For binary rewards, silence occurs when C∈{0,G}.
  • Proposition 1: U is strictly concave, symmetric about p=1/2, and uniquely maximized at U(1/2)=1−2^(1−G).It vanishes exactly at p∈{0,1}.
  • Proposition 1: Every non-silent group carries the same total squared advantage, so expected squared-advantage mass is proportional to U(p).This quantity is not the gradient norm and licenses no convergence or accuracy claim.
  • Proposition 2: A prompt-independent Beta initialization makes Uβ constant across prompts, so step-0 selection reduces to tie-breaking.Separating prompts at step 0 therefore requires an initialization that depends on the prompt.
  • Proposition 3: At equal posterior mean, Uβ increases with concentration m and approaches U(µ) as the posterior concentrates.The rule prefers better-known difficulty estimates, opposite to uncertainty sampling.

Appendix B: Extended analysis

The extended analysis examines where difficulty signals work, how pool composition limits headroom, and how deterministic selection can exclude misestimated prompts. It also narrows interpretation of ranking comparisons.

  • Where the length signal comes from: The length prior’s signal is concentrated in anchor truncation, with AUC 0.665 versus 0.915 for the verified probe.Among prompts the anchor finished, length carries no further information about difficulty; the comparison is not a clean verification isolation.
  • Pool composition and available headroom: Only 76 of 250 prompts reach U(p)>0.8, capping selection headroom on this spread-sampled pool.A 1200-prompt pool reproduced the early reduction, while the compute study remains pilot-scale.
  • Ordering quality inside the band: Inside the learnable band, the bank’s global ranking advantage narrows from 0.14 to 0.07.This is consistent with the interpretation that verified offline information matters more than the sharpest global ranker.
  • How an underestimated prompt can stay unselected: The anchor assigns φ̂=0 to 99 prompts, including 8 in the learnable band, while φ̂=1 covers 19 prompts, including 13 in-band.Both extremes drive Uβ to its floor, and no diversity or exploration repair is implemented.
  • Pool size limits the effect here: By step 30, the trace touches 69 of 76 band prompts, but deterministic top-B selection does not establish complete coverage.The data do not distinguish delayed use from persistent exclusion for the remaining prompts.

Appendix C: Limitations

The appendix limits the evidence on accuracy, generality, and pool representativeness. It does not establish parity or broad-model conclusions.

  • Accuracy scope: Accuracy effects remain unresolved because eleven three-seed arms cannot resolve accuracy, and intervals include training variance only.A single prior realization is shared by every seed.
  • Evaluation scope: The evaluation covers one model family at pilot scale.
  • Pool scope: The main prompt pool is not a uniform draw from MATH.

Appendix D: Behaviour at 300 steps

The 300-step appendix distinguishes a prespecified endpoint from exploratory trajectory analyses. The longer-horizon evidence remains descriptive and does not establish a robust difference.

  • Original endpoint: At the original endpoint, online-NP has 2/6 runs below 0.05 versus ThinkPrior’s 0/6, with Fisher two-sided p=0.455.This is the primary KL-free result and does not support a difference.
  • Exploratory analysis: The exploratory relative-drop rule changes the KL-free count to 4/6 versus 0/6, but remains non-significant at Fisher two-sided p=0.061.The threshold was chosen post hoc and favors ThinkPrior.
  • KL qualification: The KL comparison does not isolate the KL term or a pool mechanism, and β>0 invalidates the silent-group premise used by Proposition 1.
  • Claim boundaries: Pooling KL arms yields 6/10 versus 0/10 with p=0.011 under the exploratory threshold, but this post hoc analysis is not used for claims.Twenty runs are too few to estimate an event rate precisely.
  • Larger pool: The rerun uses 1200 prompts, 4.8× the original pool, with the same spread-sampling and anchor protocol.The enlarged-pool appendix tests whether the descriptive pattern changes with pool size.
  • Larger pool: At step 200 in the enlarged pool, ThinkPrior discards 5085 of 12800 rollouts versus 6099 for online-NP, a 16.6% reduction.Per-seed quantities overlap, so the endpoint fails the prespecified separation rule.

Appendix F: Training and evaluation details

The appendix fixes the training, sampling, objective, and baseline details across arms, while comparing a length-based prior with other selection approaches.

  • Training and evaluation: LoRA uses r=32, α=64, learning rate 3×10^-5, temperature 0.9, top-p 1.0, and 512 new tokens.Prompts are truncated to 384 tokens and evaluation is greedy.
  • Training objective: All arms use the same simplified KL-free GRPO objective, hardware class, gradient clipping, and one inner iteration.No arm benefits from a different optimizer.
  • Sampling accounting: Each update uses exactly B non-silent groups, while excess groups from a final refill round count toward generation but not the update.
  • Length prior: The length prior uses one greedy Qwen3-0.6B pass, token distance between think delimiters, and a two-parameter logistic fit.It is the cheapest tested prior and the only one that does not verify an answer.
Loading 2609.09075v1…