PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifier-Induced Support Reshaping in On-Policy Optimization

Shaohang Wei, Zikun Su, Feifan Song, Wen Luo, Wei Li, Guangyue Peng, Houfeng Wang

arXiv:2608.00220v1cs.LGcs.CL

TL;DR

The paper asks whether RLVR preserves successful behaviors needed for later objectives, not merely current-task performance. It measures cross-task support through repeated verifier-scored sampling and bidirectional training, finding that verifier optimization can reshape future rewardable support and limit joint capability.

  • Problem

    Evidence of RLVR’s effect on future-task discoverability remains limited, despite the need to preserve later learnability during successive post-training stages.

  • Method

    Across two model families, the study uses repeated verifier-scored sampling and bidirectional Math/IF-RLVR training, including sequential opposite-verifier training.

  • Results

    Math-RLVR polarizes instruction-following support, while IF-RLVR reduces math searchability and constrains subsequent Math-RLVR; joint gains combine only partly.

  • Takeaways & Limitations

    Endpoint improvements under on-policy optimization do not guarantee preserved future trainability or joint capability across verifiers.

  • Takeaways & Limitations

    Neither tested training order provides a simple sequential solution: each trades adaptation on one objective against retention or support for the other.

Abstract

from arXiv · show

We show that on-policy reinforcement learning with verifiable rewards (RLVR) can improve the current objective while making successful behaviors for later objectives too rare to sample and reinforce. We call this verifier-induced support reshaping and define effective rewardable support as successful trajectories reachable within a fixed rollout budget. Across two model families, we study this effect through repeated verifier-scored sampling and bidirectional training on mathematical reasoning and constrained instruction following, including sequential training with the opposite verifier. Math-RLVR raises average instruction-following success but reduces the number of prompts with any successful response under repeated sampling. On IFEval with Qwen3-8B-Base, pass@1 rises by 6.5 percentage points while best@32 falls by 9.8 percentage points, and the same divergence appears across both models and IF benchmarks. Conversely, IF-RLVR shifts math responses from step-by-step openings toward direct answers, lowers best@k across sampling budgets, and reduces reward variation for later Math-RLVR. Token-distribution analyses and controlled opening interventions show that these changes concentrate in the first few response tokens. RLVR mainly reranks openings already available in the base policy, and the selected opening causally affects math searchability. The tested reference-policy constraints, routing priors, and on-policy distillation preserve cross-task support only partially; MathIF and ReasonIF show that marginal gains translate only partly into responses that are both correct and constraint-following. Therefore, endpoint improvements do not guarantee future trainability or joint capability under on-policy optimization. Code is available at https://github.com/sylvain-wei/verifier-induced-support-reshaping

1. Introduction

The introduction frames continual RLVR as a future-trainability problem: optimizing one verifier can reshape which successful behaviors remain reachable for later objectives. Bidirectional Math/IF experiments attribute this effect largely to response openings and show that endpoint gains only partly preserve joint capability.

  • Motivation: Continual RLVR must preserve successful behaviors for later objectives that remain reachable under a fixed sampling budget.The paper defines this reachable set as effective rewardable support and distinguishes support reshaping from catastrophic forgetting.
  • Main findings: Bidirectional Math/IF experiments show that Math-RLVR raises IF pass@1 but lowers best@32, while IF-RLVR lowers math best@k and constrains later Math-RLVR.Math responses also shift from step-by-step openings toward direct answers after IF-RLVR.
  • Joint capability: MathIF and ReasonIF show that improvements in mathematical reasoning or instruction following translate only partly into responses satisfying both objectives.This extends the support-reshaping concern from separate task outcomes to joint capability.
  • Mechanism: Cross-task changes concentrate in the first few response tokens, and controlled route interventions show that opening choices causally affect math searchability.The contribution frames this as a change in route selection rather than broad erasure of reasoning ability.
  • Support preservation: Reference-policy constraints, routing priors, and on-policy distillation only partially preserve future support or trade it against target-task gains.The paper evaluates these support-preservation strategies during sequential training.

2. Related Work

Prior work established RLVR as a way to improve mathematics, coding, reasoning, and instruction following through verifiable rewards. This paper instead focuses on whether on-policy training preserves rewardable support for later objectives and joint behavior under sequential training and distillation.

  • Verifiable Rewards and Reasoning Post-training: RLVR uses executable or checkable rewards to improve performance on optimized mathematics, coding, and reasoning tasks.Earlier work also developed learned verifiers, self-consistency, reasoning benchmarks, and process supervision for evaluating and improving reasoning outputs.
  • Instruction Following and Reasoning: Verifiable instruction-following benchmarks convert explicit length, format, keyword, and structural requirements into automatically checkable constraints.Recent work shows that these rewards can directly improve instruction following.
  • Instruction Following and Reasoning: This work asks whether optimizing one verifier preserves rewardable support needed by another objective and the joint behavior users need.The question extends beyond task-specific math accuracy or instruction-following pass rate.
  • RLVR Mechanisms, Sequential Post-training, and Support Preservation: Prior studies examine RLVR distribution shifts, token-level behavior changes, new ability learning versus reasoning-pattern reweighting, and on-policy distillation for stabilizing post-training.This paper studies a different limitation: later-objective trajectories may become too rare to sample and reinforce after another verifier shifts the sampling distribution.

3. Experimental Design

The study compares Math-RLVR and IF-RLVR from shared starting policies, then tests whether the first training stage affects later learning. It uses two model families, task-specific verifiers, and repeated-rollout support evaluations spanning separate and joint behaviors.

  • Training Paths: The core comparison evaluates Math-RLVR versus IF-RLVR from the same starting policy, followed by IF→Math and Math→IF sequential training paths.These paths test whether the first RLVR stage affects subsequent learning.
  • Training Paths: Experiments use Qwen3-8B-Base and Qwen2.5-Math-7B, with Math-RLVR on 7.5k MATH examples and IF-RLVR on IFTrain.Math-RLVR uses an exact final-answer verifier, whereas IF-RLVR uses deterministic checkers requiring all explicit response constraints to pass.
  • Evaluation and Support Measures: Support is evaluated on AIME24, AIME25, MATH-500-128, IFEval, and IFBench using k stochastic rollouts per prompt.MathIF and ReasonIF provide a separate endpoint set testing whether correctness and constraint following occur in the same response.

4. Verifier-Induced Cross-Task Support Reshaping

Verifier choice reshapes cross-task effective support: Math-RLVR improves average instruction-following success while reducing repeated-sampling coverage, whereas IF-RLVR makes correct math responses harder to reach. Sequential training exposes complementary tradeoffs, with IF-first reducing later math-training signal and Math-first trading instruction-following gains against math retention.

  • Single-stage cross-task shifts: Math-RLVR raises average instruction-following success but leaves fewer prompts solvable under repeated sampling.The shift polarizes prompt outcomes toward all-correct and all-wrong groups.
  • Single-stage cross-task shifts: IF-RLVR lowers AIME best@k at every tested budget from k=4 to k=32 as response openings shift from deliberative reasoning initiation toward direct answer initiation.Higher direct-answer-initiation share co-occurs with lower AIME best@32, while the evidence remains behavioral before the causal tests in Section 5.
  • Sequential training: After IF-first training, the mixed group falls from 33.6% to 1.6% by step 20, limiting reward variation available to later Math-RLVR.The mixed group consists of prompts with both successful and failed rollouts, which provide within-group reward variation for the later update.
  • Sequential training: Math-first training followed by unconstrained IF-RLVR improves instruction-following performance but sharply reduces the math performance gained in the first stage.Reference-policy constraints only trade adaptation for retention: β=0.12 largely retains math-task pass@1 but yields the smallest instruction-following gains, and no tested coefficient matches unconstrained gains while preserving math.

5. Mechanism: Support Reshaping Concentrates at the Response Opening

RLVR-induced distributional changes are concentrated at the first response token, where Math-RLVR and IF-RLVR favor opposite routes. Controlled opening interventions show that this route entry causally changes downstream math searchability.

  • Distributional localization: The first generated token has the highest mean JS divergence across every model, RLVR branch, and benchmark, with AIME ratios of 9.8×–106.7× versus interior positions.The largest policy change therefore localizes to route entry rather than the full generation trace.
  • Candidate reranking: At high-JS positions, Base and RLVR share 6.88–8.96 top-10 candidates, while 89.3%–99.4% of RLVR top choices already appear in Base’s top-3.At the AIME opening under IF-RLVR, the top choice can instead have median Base ranks of 14.5 and 18 with mean probabilities of 1.9% and 0.4%.
  • Route selection: Across both models, Math-RLVR makes DRI-like “Alright” or “To” the top first token, whereas IF-RLVR makes the DAI-like “Answer” dominant.The two verifiers thus push generation toward opposite response routes at the opening.
  • Causal opening intervention: Forcing Base-side or DRI openings from IF-RLVR checkpoints raises best@32 on both AIME and MATH-500, establishing a causal effect of opening route on math searchability.Reverse interventions are asymmetric: IF-side or DAI openings reduce searchability for Qwen3-8B-Base but are neutral or positive for Qwen2.5-Math-7B.

6. Mitigating Support Reshaping: Levers and Limits

The tested mitigation strategies only delay or partially preserve cross-task support: a one-time DRI prior is eventually overwritten, while OPD outcomes depend on teacher-state selection and do not consistently retain math support.

  • DRI-biased SFT prior: A one-time route prior changes when DAI dominance occurs but not the endpoint: every condition reaches a DAI rate of 1 by step 100.Soft DRI delays the shift longest, retaining AIME24 best@32 = 0.400 and IFEval best@16 = 0.948 at step 70 with DAI rate 0.033.
  • On-policy distillation: Dense OPD from a converged IF-RLVR teacher sharply reduces math performance while increasing shortcut responses.MATH-500-128 mean@16 falls from 0.3433 to 0.0879, and shortcut-response share rises from 4.5% to 18.8%.
  • On-policy distillation: Teacher and student token-preference shifts correlate negatively, showing that dense supervision does not automatically preserve cross-task support.The Spearman correlation is 𝜌= −0.594, with positive student shifts concentrated mainly on short responses.
  • Teacher-state selection: Different OPD teacher states trade off instruction-following transfer against math-support retention rather than optimizing both.T2/40 reaches the highest IFEval mean@16 of 0.814, while T1/20 is the only student improving MATH-500 mean@16, by +0.93 percentage points.
  • Overall limits: Neither mitigation preserves math support throughout the full IF adaptation path, because DRI priors are overwritten and OPD requires teacher selection across multiple objectives.The overall result motivates an independent joint-support test of whether math and instruction-following gains coexist in the same response.

7. Marginal Gains Translate Only Partly into Joint Support

MathIF and ReasonIF require correctness and strict following in the same response, revealing that verifier-aligned marginal gains translate only partly into joint support. Math-RLVR and IF-RLVR improve their target metrics far more than joint success, with non-target metrics sometimes declining.

  • Joint-support results: In both training directions, non-target metrics change far less than verifier-aligned metrics and sometimes decline, limiting observed joint gains.The benchmarks stress-test whether separate marginal improvements occur in the same rollout.
  • Evaluation setup: MathIF constrains the full mathematical response, while ReasonIF constrains the reasoning trace and isolates the final answer.Joint support J = C∧F requires correctness and strict following in the same response.
  • Joint-support results: Math-RLVR raises C by 11.3/5.6 pp, but J rises only 2.4/1.7 pp across MathIF/ReasonIF.F remains nearly unchanged, so correctness gains only partly become same-response joint support.
  • Joint-support results: IF-RLVR raises F by 18.1/15.3 pp, while J rises only 5.5/6.2 pp across MathIF/ReasonIF.C increases only slightly on MathIF and decreases on ReasonIF.

8. Conclusion … A.2. Primary RLVR Training Paths

Verifier choice in on-policy RLVR changes both current performance and future trainability by reshaping effective support, while the experiments and appendices define the training paths, model references, benchmarks, diagnostics, and analyses used to study this effect.

  • 8. Conclusion: Verifier choice shapes current scores and future trainability: Math-RLVR polarizes instruction-following support, while IF-RLVR favors direct answers and reduces math searchability.Controlled interventions identify a causal role for route entry, joint tests show gains combine only partly, and effective support depends on rollout budget.
  • Appendix Contents: The supplementary material contains proofs, prompts, and additional experimental results supporting the paper’s analyses.
  • A. Experimental Setup and Training Recipes: The experimental appendix covers models and base policies, primary RLVR training paths, sequential training, and mitigation recipes.
  • Appendix Contents: The benchmark appendix specifies mathematics, instruction-following, joint-capability, decoding, support metrics, support-group probes, and stability evaluations.
  • Appendix Contents: The analytical appendices classify opening routes, use an independent semantic judge, analyze token-level distributions, test candidate overlap and top-token promotion, and run prefix interventions.
  • Appendix Contents: Extended and stress-test appendices report cross-model robustness, qualitative cases, endpoint and sampling details, metric filters, and instruction-constraint data lineage.
  • A.1. Models and Base Policies: Experiments use Qwen3-8B-Base and Qwen2.5-Math-7B, with each pre-training policy designated Base and used as the reference policy for distributional comparisons.Training was conducted primarily on a node with eight NVIDIA H20 GPUs, each with 96 GB of GPU memory.
  • A.2. Primary RLVR Training Paths: Math-RLVR trains on MATH with exact-match rewards for extracted final boxed answers, whereas IF-RLVR uses IFTrain’s 29 non-overlapping constraint types and a rule-based instruction verifier.The MATH training split contains 7.5k examples and spans varied difficulty and problem types.

A.3. Sequential Training and Mitigation Recipes … B.4. Decoding Parameters and Support Metrics

The paper evaluates sequential verifier training, routing-prior and distillation mitigations, and benchmark-specific support using standardized and repeated stochastic decoding. Its diagnostics span mathematical, instruction-following, and joint-capability tasks, distinguishing single-sample success from reachable rewardable trajectories.

  • A.3. Sequential Training and Mitigation Recipes: Sequential experiments continue on-policy training from converged IF-RLVR or Math-RLVR endpoints, with reference-KL values β∈{0, 0.04, 0.08, 0.12} testing retention–adaptation trade-offs.Math→IF includes an unconstrained sequential baseline.
  • A.3.1. Routing-Prior Cold-Start Conditions: Four routing-prior cold starts receive SFT followed by identical 100-step IF-RLVR training, while On-Policy Distillation matches Base students to teacher rollout distributions at selected checkpoints.The cold starts differ in opening composition and SFT configuration; distillation uses a top-K reverse KL objective.
  • B. Benchmark Suite and Decoding Protocols: The methodology section documents benchmark compositions, decoding configurations, and diagnostic metric formulations before applying these protocols across the evaluation suite.This provides the stated organizational scope for the benchmark and decoding sections.
  • B.1. Mathematics Benchmarks: The benchmark suite combines high-difficulty AIME24 and AIME25 problems with MATH-500-128, a stratified 128-prompt subset spanning difficulty levels 1 through 5.AIME24 and AIME25 each contain 30 problems.
  • B.2. Instruction-Following Benchmarks: Instruction-following evaluation uses IFEval’s 541 prompts and 25 constraint types alongside IFBench’s 58 unseen constraints over 300 held-out WildChat prompts.Official deterministic decoding reports standardized scores, while repeated stochastic decoding uses temperature 0.7 and top-p 0.95.
  • B.3. Joint-Capability Benchmarks: MathIF tests answer correctness and full-response compliance, while ReasonIF separately evaluates constraint following in reasoning traces and final answers.MathIF attaches one to three IFEval-style constraints; ReasonIF applies one constraint per prompt and places the final answer in separate tags.
  • B.4. Decoding Parameters and Support Metrics: Core support diagnostics sample 32 stochastic rollouts per prompt and distinguish expected single-sample pass@1 from best@k reachability within k samples.Pass@1 estimates expected current-policy success probability, whereas best@k asks whether at least one rewardable trajectory remains reachable.

B.5. Support-Group Probes and Standalone Training Trajectories … D. Token-Level Mechanism and Distributional Analyses

The paper tracks how standalone Math-RLVR and IF-RLVR alter math support-group composition, then uses deterministic opening classification and independent semantic judging to analyze verifier-passed behavior and token-level mechanisms.

  • B.5. Support-Group Probes and Standalone Training Trajectories: Checkpoint probes classify prompts as all-wrong, mixed, or all-correct based on whether none, some, or all repeated stochastic rollouts succeed.The probes use a fixed set of math prompts and are unfiltered, unlike dynamic-sampling training logs that exclude prompts without within-group reward variation.
  • B.5. Support-Group Probes and Standalone Training Trajectories: Math-RLVR reduces the all-wrong share from 20.3% to 11.7% while expanding the all-correct share.These support-group measurements connect independently probed checkpoints and do not represent intermediate training steps.
  • B.5. Support-Group Probes and Standalone Training Trajectories: IF-RLVR increases the all-wrong share from 20.3% to 64.1% while the all-correct share nearly vanishes.The standalone trajectories move math support-group composition in the opposite direction from Math-RLVR.
  • C. Opening Route Classification and Behavioral Diagnostics: The behavioral-diagnostics section defines exact pattern-matching rules for the opening-route classifier and an independent semantic judge.These rules support reproducible analysis without relying on subjective interpretation.
  • C.1. Deterministic Opening Route Classification Protocol: The opening-route analysis uses a deterministic, non-parametric, priority-ordered classifier over each rollout’s initial 64-character window.This design captures opening-route decisions while preventing downstream reasoning from triggering opening-mode rules.
  • C.1. Deterministic Opening Route Classification Protocol: The classifier normalizes text, evaluates mutually exclusive rules in priority order P1 →P4, and achieves over 98.5% precision in manual spot-checking across 500 rollouts.Evaluation stops at the first matching rule after stripping leading whitespace, newlines, and Markdown header symbols.
  • C.2. Independent Semantic Judge (DeepSeek-V4-Pro): An independent LLM judge categorizes verifier-passed IF rollouts as contentful, shortcut, or other responses.Contentful responses are genuine and thorough; shortcut responses exploit superficial verifier-passing strategies; other responses fail constraints or have severe content defects.
  • D. Token-Level Mechanism and Distributional Analyses: Token-level analyses compute distributional divergences and apply causal interventions to study the mechanisms underlying the observed behavior.The supplied section description identifies both divergence calculations and causal intervention methodologies as the analysis tools.

D.1. Position-Wise JS Divergence over the Top-𝐾Union … F.1. Endpoints and Sampling

The paper finds that RLVR-induced policy differences are concentrated in a small fraction of token positions, while changing response openings can reshape downstream searchability and joint behavior across model families. Extended analyses reproduce these effects, including instruction-following shortcuts and independent endpoint stress tests.

  • D.1. Position-Wise JS Divergence over the Top-𝐾Union: Median token-level JS ranges from 1.1 × 10−5 to 3.7 × 10−4 across ten model–checkpoint–dataset combinations.Positions above 0.05 comprise 0.4%–13.5%, while those above 0.1 comprise 0.1%–7.7%.
  • D.2. Candidate-Set Overlap and Top-Token Promotion: High-JS analysis compares Base and RLVR candidate reuse across the top-5, top-10, and top-64 sets, recording RLVR top-1 rank and renormalized Base probability.A missing RLVR top-1 token receives Base rank 65 and renormalized probability zero.
  • D.3. Single-Token and Prefix Interventions: Forced first-token and prefix interventions test whether target-policy openings causally control downstream math searchability.The interventions replace the first token or inject DRI/DAI opening prefixes before free sampling resumes.
  • E. Extended Results and Qualitative Cases: Extended results show that Qwen2.5-Math-7B reproduces single-sample IF gains with best-of-32 contractions, while IF-RLVR severely collapses math searchability.These findings provide cross-family robustness validation of the core effects.
  • E.2. Qualitative Case Studies: Surface-compliant shortcut responses front-load required placeholders or keywords, whereas uncollapsed instruction-following rollouts preserve contextual flow and comprehensive fulfillment.The contrast concerns responses that satisfy rule checkers while omitting substantive content versus substantive rollouts.
  • F.1. Endpoints and Sampling: The joint-behavior stress test uses independent Qwen3-8B-Base, Math-RLVR step-720, and IF-RLVR step-720 endpoints rather than earlier checkpoints.It samples 16 rollouts per benchmark, endpoint, and prompt at temperature 0.7, top-p 0.95, and at most 8,192 new tokens.

F.2. Metrics and Main-Table Filters · G. Instruction-Constraint Data Lineage

The evaluation defines correctness, strict constraint satisfaction, and their joint event, using a fixed 290-prompt MathIF main-table filter with exactly 16 rollouts per endpoint. The instruction-constraint datasets share a response-constraint lineage but separate baseline, training, out-of-domain, mathematical-answer, and reasoning-trace roles.

  • F.2. Metrics and Main-Table Filters: MathIF evaluates correctness C, strict constraint satisfaction F, and joint success J = C∧F on the same rollout.For MathIF, F requires all supported constraints to be satisfied.
  • F.2. Metrics and Main-Table Filters: 290 MathIF prompts remain in the main table after filtering for scalar-form gold answers, complete constraint scores, and exactly 16 rollouts per endpoint.The fixed filter excludes answers requiring symbolic equ...
  • G. Instruction-Constraint Data Lineage: The five datasets extend a common response-constraint family toward baseline, training, out-of-domain, mathematical-answer, and reasoning-trace settings.IFEval supplies baseline verifiable types; IFTrain adds training types; IFBench reserves out-of-domain types; MathIF and ReasonIF alter task content and constraint target.
  • G.1. Roles, Sources, and Constraint Targets: 25 IFEval types, 29 IFTrain types, and 58 IFBench types serve distinct roles as baseline evaluation, training coverage, and unseen-constraint evaluation.The inventories are separated so IFBench is not treated as another IFEval split.
  • G.1. Roles, Sources, and Constraint Targets: All 15 MathIF types use IFEval-style response constraints, changing the underlying task rather than the constraint paradigm.ReasonIF instead applies familiar case, format, language, length, punctuation, and ending constraints to reasoning traces.
  • G.2. Complete Verifier-Type Inventory: The IF-RLVR training path uses the 29 IFTrain verifier types, whose public identifiers combine group and constraint columns.The verifier inventory records each public type identifier and the checks it performs, with instance parameters stored in data-file fields.
  • G.2. Complete Verifier-Type Inventory: The public IF-RLVR recipe may additionally mix in 25 IFEval types, while 58 IFBench types remain reserved for unseen-constraint evaluation.These roles apply after the full benchmark sets are loaded and before the Chapter 7 main-table filters are applied.
Loading 2608.00220v1…