PaperlayerView exact revision on arXiv

Source-linked AI summary

Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning

Qiqian Fu

arXiv:2608.21811v1cs.AIcs.LG

TL;DR

Sparse rewards leave most GRPO rollout groups without learning signal in vision-language math reasoning. The paper compares eleven text, distribution, and value priors under matched conditions, then tests whether in-domain slices predict cross-domain transfer. Delivered priors separate from ineffective ones, but one widely used slice reverses transfer rankings while the hardest slice predicts them closely.

  • Problem

    Sparse-reward GRPO training on visual-math problems often has all-wrong rollout groups that contribute no gradient, limiting learning evidence.

  • Method

    The paper trains eleven prior-injection arms spanning hints, on-policy teacher distillation, and value critics under identical conditions with blind judging and paired exact tests.

  • Results

    Delivered priors occupy 30.6–32.9% on DynaMath versus 27.0–29.1% for the remaining five arms, while old-319 anti-correlates with transfer at ρ = −0.74 and sparse-800 predicts it at ρ = +0.89.

  • Takeaways & Limitations

    Prior injection helps when it reaches the policy, hint gains come from exploration, and evaluation slices with chance-level subtasks can reverse apparent method rankings.

  • Takeaways & Limitations

    The pooled in-domain metric is a weighted summary of non-random slices, while sparse-800 shares training distributions and template families; DynaMath provides the load-bearing transfer evidence.

Abstract

from arXiv · show

Reinforcement learning for vision-language math reasoning starves under sparse reward: on a pool of 20,830 visual-math problems where Qwen2-VL-2B answers 3.6% of rollouts correctly, 85-97% of GRPO rollout groups are entirely wrong and contribute zero gradient. We train eleven methods under identical conditions in this regime, each injecting a different prior: text (reference-solution hints), distribution (on-policy distillation from a 7B teacher), and value (a value-pretrained critic with an MSE or HL-Gauss categorical loss). A prior helps exactly when it is delivered: the six arms whose prior effectively reaches the policy separate with no overlap from the remaining five -- the no-prior baseline and four arms whose prior is teacher-capped, gated away, or lost to a mis-parameterized critic -- both on the pooled in-domain metric and on cross-domain transfer (DynaMath). The central finding, however, concerns evaluation: one slice of the in-domain pool -- long used as this project's general-distribution check -- anti-correlates with genuine cross-domain transfer (Spearman rho = -0.74, n = 11 arms, permutation p = 0.011), while the hardest in-domain slice predicts it closely (rho = +0.89, p < 0.001). We attribute the inversion to a near-chance multiple-choice subset that rewards models for not having changed; read through it, the best cross-domain method looked mediocre and the worst looked like the champion. Among the methods, hint-guided exploration -- not UFT's auxiliary loss -- drives hint gains, and replacing the critic's MSE loss with HL-Gauss cross-entropy is worth +14.4 points in-domain. All accuracies are blind-judged, with paired exact tests.

1 Introduction

This paper compares eleven prior-injection strategies for sparse-reward vision-language math RL and finds that delivered priors improve transfer, while evaluation composition can reverse method rankings.

  • Problem: 85–97% of GRPO groups are all-wrong on the 20,830-problem pool, so sparse rewards often provide exactly zero gradient.Qwen2-VL-2B answers 3.6% of rollouts correctly on this pool.
  • Study design: Eleven training arms compare text, distribution, and value priors under identical data, budgets, seeds, judging, and hardware.All accuracies are blind-judged, with pairwise claims tested using McNemar’s exact test.
  • Method findings: 30.6–32.9% versus 27.0–29.1% on DynaMath separates the six arms whose priors reach the policy from five arms whose priors do not.The lower group includes plain GRPO and arms limited by teacher ability, failure gating, or critic mis-parameterization.
  • Method findings: +14.4 pp in-domain accuracy results from replacing the critic’s clipped-MSE loss with HL-Gauss cross-entropy.Hint-guided exploration, rather than UFT’s auxiliary likelihood term, accounts for hint gains; two priors do not compose.
  • Evaluation finding: ρ = −0.74 for old-319 versus ρ = +0.89 for sparse-800 when predicting cross-domain transfer across eleven arms.The paper attributes the inversion to old-319’s near-chance multiple-choice majority, which rewards retaining base-model behavior.
  • Verification: The paper retracts earlier evaluation-based conclusions and presents blind judging, scoring audits, pre-training gates, and reward-hacking analyses as a reusable verification protocol.The protocol caught scoring distortions and tested guessing and formatting explanations for the reported gains.

2 Related Work

Related work frames sparse-reward vision-language math RL through hint-based exploration, teacher-driven distribution priors, value critics, and concerns about benchmark validity and reward-driven gains.

  • Text priors: UFT prefixes rollouts with reference-solution fragments, using hints as exploration scaffolds and optionally adding supervised likelihood on hint tokens.The paper separates hint-guided exploration from the auxiliary likelihood term in its hint arms.
  • Distribution priors: On-policy distillation supplies per-token advantages from a teacher, while replay-based variants reuse teacher trajectories as rollout prefixes.Because reverse-KL imitation cannot exceed the imitated behavior, distillation-only training inherits the teacher’s ceiling.
  • Value priors: Value-based approaches use pretrained critics and decoupled advantage estimation, while categorical return-support classification replaces scalar critic regression.The paper evaluates value-pretraining and critic-loss parameterization as separate design choices.
  • Vision-language math RL: GRPO transfers to vision-language models such as Qwen2-VL, with MathV360K and DynaMath providing visual-math training and evaluation settings.These benchmarks situate the paper’s comparison within vision-language mathematical reasoning.
  • RL reliability: Prior work warns that accuracy rewards, spurious rewards, normalization biases, and privileged information can distort apparent RL progress.These concerns motivate examining both reward mechanisms and the behavior induced by hint-based methods.
  • Evaluation validity: Benchmark surface composition can shift rankings, motivating this paper’s sharper evaluation-validity analysis of a held-out slice that reverses cross-domain ordering.The related literature connects answer format and option order to divergence between apparent measurement and actual ranking drivers.

3 Experimental Setup

The experiment creates a genuinely sparse visual-math regime, then compares eleven prior-injection methods under controlled training and blind evaluation across in-domain and cross-domain datasets.

  • Motivation: Round 4 reruns the comparison after making rewards genuinely sparse and broadens it to text, distribution, and value priors.Earlier rounds found no hint advantage because all-wrong GRPO groups fell from 77% to roughly 30% during training.
  • Evaluation: A strict Answer: X scorer gives 3.5% versus 43.4% under semantic rescoring, so difficulty selection uses semantic scores and reported results use a blind LLM judge.The strict scorer measures format compliance rather than ability; paired comparisons use McNemar’s exact test.
  • Pool construction: The sparse pool contains 21,630 problems solved at most 1-in-8 times, with 20,830 used for training and 800 held out.Within the pool, the base model answers 3.6% of rollouts correctly.
  • Pool construction: 85–97% of opening GRPO groups are all-wrong, while prefixing half a reference solution raises base accuracy from 7.2% to 34.4%.These pre-training checks verify both reward sparsity and hint efficacy.
  • Methods: The eleven arms include plain GRPO, three hint variants, four distillation variants, two critic variants, and one stacked text–distribution prior.All arms train on the same sparse pool under identical batch size, learning rate, seed, and 500-step budget.
  • Evaluation: Evaluation combines two in-domain MathV slices and cross-domain DynaMath, with pooled in-domain scores reported alongside per-slice breakdowns.DynaMath has no overlap with MathV360K, and the two in-domain slices measure different quantities.

4 Results

The eleven arms split cleanly according to whether their injected prior reached the policy, but evaluation validity determines which methods appear strongest. Delivered-prior arms separate from undelivered-prior arms, while sparse-800 predicts DynaMath and old-319 ranks methods backwards.

  • Main comparison: Six delivered-prior arms span 41.6–44.1 on pooled in-domain accuracy, versus 34.9–37.8 for five undelivered-prior arms.The same six-versus-five split appears on sparse-800: 49.4–53.6% versus 38.8–41.0%.
  • Main comparison: The lower five comprise plain GRPO and arms whose priors were teacher-capped, failure-gated, or lost through a mis-parameterized critic.The delivered and undelivered labels are empirical, reflecting whether the prior reached the policy rather than the nominal architecture.
  • Main comparison: DynaMath fully separates the groups: delivered-prior arms occupy 30.6–32.9%, while undelivered-prior arms occupy 27.0–29.1%, with no overlap.Within-group ordering spans at most 2.3 percentage points and is not interpreted under one training seed.
  • The benchmark that ranked backwards: Vapo tops old-319 at 32.6 but ranks last on DynaMath at 27.0, whereas vapohl scores 19.7 on old-319 and ranks second on DynaMath at 31.9.These contrasting placements expose the held-out slice’s reversed ranking of cross-domain performance.
  • The benchmark that ranked backwards: Sparse-800 predicts DynaMath closely at ρ = +0.89, while old-319 anti-correlates at ρ = −0.74; the correlations concern group ordering, not fine-grained arm rankings.The reported permutation values are p < 0.001 for sparse-800 and p = 0.011 for old-319.
  • The benchmark that ranked backwards: Old-319 is dominated by near-chance four-option Geometry3K multiple choice, where scoring rewards retaining base-model behavior rather than changed reasoning outputs.The paper presents this as an interpretation consistent with the slice composition, not a direct decomposition of each arm’s predictions.
  • The benchmark that ranked backwards: The pooled in-domain metric correlates with DynaMath at ρ = +0.76 under its 71.5/28.5 weighting but only +0.40 with equal slice weighting.The paper identifies sparse-800 alone as the cleanest single in-domain number and retracts earlier conclusions based on old-319 as a generalization measure.

5 Method Findings

Across eleven arms, priors improved transfer only when they reached the policy, while hint exploration and HL-Gauss critic training identified the active ingredients. Evaluation-set correlations also showed that sparse-800 predicted DynaMath whereas old-319 reversed the ranking.

  • Hints: +10.4 to +11.4 pp: all three hint arms beat GRPO on sparse-800 under the present sparse-reward regime.The earlier hint result was weaker because all-wrong groups fell from roughly 77% to about 30%, whereas this pool begins at 85–97%.
  • Evaluation: sparse-800 predicted DynaMath with Spearman ρ = +0.89, while old-319 anti-predicted it with ρ = −0.74; vapo topped old-319 but ranked last on DynaMath.The pooled in-domain metric inherited predictive validity from sparse-800.
  • Hints: Hint-guided exploration, not UFT’s auxiliary log-likelihood term, accounts for hint gains: hintonly, uft001, and uft002 were statistically indistinguishable on sparse-800.Their sparse-800 accuracies were 51.4, 52.0, and 52.4, respectively, with all pairwise comparisons nonsignificant.
  • Value prior: +14.4 pp: replacing vapo’s clipped-MSE critic loss with HL-Gauss cross-entropy moved sparse-800 from 39.2 to 53.6 and DynaMath from 27.0 to 31.9.The change also raised explained variance from 0.59 to 0.66 and removed a cold-start pathology.
  • Distribution prior: Annealed global distillation beat GRPO by +8.6 pp on sparse-800, whereas targeted persistent distillation remained at GRPO level on every set.saf posted the best DynaMath-hard score, 18.3%, while srpo’s gated, non-decaying signal did not improve performance.
  • Distribution prior: Pure distillation converged to GRPO-level performance because reverse-KL imitation inherits the 7B teacher’s 27.9% ceiling, unlike reward-driven hint arms reaching roughly 52% on sparse-800.reopd tracked opd everywhere, consistent with its multi-turn replay mechanism being unnecessary in this single-turn setting.

6 Verification Discipline

The study treated verification as a deliverable by auditing scoring, gating mechanisms before training, and testing reward-hacking explanations. These checks exposed parser-induced ranking inversions and operational failures before or during experimentation.

  • Scoring audit: A scoring audit found that the heuristic extractor penalized arms differently, with effects up to 30 pp that could invert rankings.Only blind-judged numbers appear in the paper after corrected estimates matched subsequent blind-judge results within about 1 pp.
  • Scoring audit: The apparent 6.9% uft001 score on old-319 was entirely a parser bug affecting all 141 multiple-choice rows it touched.Distillation arms were under-scored by 19–30 pp because their terse answers did not match extraction patterns.
  • Hacking analysis: Four reward-hacking hypotheses were tested against the hint arms, with their verdicts summarized in Table 6.The analysis was prompted by the hint arms’ large sparse-800 gains.
  • Pre-training gates: Pre-training gates checked tokenizer identity, critic reachability, ReOPD sampling geometry, hint unlock, and projection behavior before training began.Framework patches were environment-gated, self-tested, and bit-identical to stock behavior when gating variables were unset.
  • Runtime checks: Runtime gates caught a 2,062-token hint prompt failure, too-short teacher answers requiring a stored lead-in, and an incorrect categorical-head submodule replacement.These failures were detected during execution rather than only in post-mortem analysis.

7 Limitations

The conclusions are bounded by single-seed training, non-random pooled in-domain weighting, distribution overlap in sparse-800, and asymmetric supervision across method families. Several comparisons are also limited by underpowered splits and judge variance.

  • Experimental scope: All eleven arms used seed 42, while three-seed measurements of the same GRPO recipe showed seed-to-seed standard deviation of roughly 2.4–2.7 pp.The authors identify multi-seed replication of the group-level result as immediate future work.
  • Statistical resolution: Differences under about 5 pp, including the ordering within the delivered-prior group, should not be ranked because scoring-pipeline noise spans that range.The headline between-group gaps of +8 to +14 pp in-domain exceed this noise.
  • Evaluation scope: The pooled in-domain metric is a weighted summary, not unbiased accuracy, because its 71.5/28.5 slice weighting reflects available sizes rather than a sampling design.The authors identify sparse-800 alone as the cleanest single in-domain number.
  • Evaluation scope: old-319 should not be interpreted as a generalization measure; it is retained only for comparability with rounds 1–3.Its composition caused the project’s earlier conclusions to be retracted explicitly.
  • Distribution scope: sparse-800 shares training distributions and template families, with roughly 3 pp of template-duplicate inflation for the strongest arm.DynaMath is the load-bearing transfer evidence because it shares no items with MathV360K.
  • Interpretation scope: The positive result that sparse-800 predicts transfer is specific to this setting, whereas the negative lesson about drifted held-out slices is presented as general.A same-distribution held-out set could instead favor arms that overfit training data.
  • Statistical resolution: DynaMath-hard is underpowered at n = 104: saf’s 18.3% margin over base reaches only p = 0.09, so no per-arm conclusions are drawn from that split alone.The full DynaMath evaluation remains the relevant cross-domain comparison.
  • Supervision scope: Distillation arms receive no ground-truth answers, whereas reward-driven arms do, so comparisons across families involve asymmetric supervision.The authors treat this as a property of the methods rather than an oversight.

8 Conclusion

The paper concludes that prior injection matters in sparse-reward vision-language math only when the prior is delivered, but its most durable result concerns evaluation validity. A drifted held-out slice can reverse method rankings, motivating multi-arm cross-domain checks and stronger verification.

  • Method findings: A delivered prior—not prior injection by design alone—separates successful arms from no-prior, teacher-capped, gated, and mis-parameterized alternatives.The active ingredients are hint-guided exploration, annealed global teacher-KL, and a well-parameterized value critic.
  • Method findings: Removing UFT’s auxiliary objective preserves hint gains, while changing value-loss parameterization by one line adds +14.4 pp in-domain and crosses the arm boundary.These comparisons narrow the mechanisms responsible for the observed family-level gains.
  • Evaluation validity: The old-319 generalization check ranked eleven arms in reverse order of genuine cross-domain transfer because its near-chance multiple-choice majority rewarded unchanged guessing behavior.The paper presents this composition-driven failure mode as relevant to projects using fixed held-out slices.
  • Verification and next steps: Multi-arm correlation checks, blind judging, scoring audits, pre-training gates, and hacking analyses are proposed as safeguards for keeping evaluation numbers attached to reality.Immediate next steps are multi-seed replication and per-item decomposition of the inverted set.
Loading 2608.21811v1…