PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPO

Sofia Torres, Gabriel Almeida, Carter Adams, Camila Rocha

arXiv:2610.06861v1cs.LGcs.CL

TL;DR

Guidance-augmented RL methods for LLM reasoning have reported gains, but lacked convergence, bias, and weighting theory. GA-GRPO models guidance as a stochastic operator and analyzes its bias and optimal weight; across nine benchmarks, its optimal-weight variant matches or surpasses several baselines with fewer GPU-hours.

  • Problem

    Guidance-augmented RL methods lack convergence guarantees, bias bounds, and principled guidance-weighting rules, despite reported empirical gains.

    Source: §1 · p. 2

  • Method

    GA-GRPO models external guidance as a stochastic operator that rewrites the question distribution and bounds estimator bias by guidance divergence.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    31% fewer GPU-hours; optimal-weight GA-GRPO matches or surpasses TAPO, LUFFY, ExPO, and vanilla GRPO across nine math and OOD benchmarks.

    Source: §1 · p. 3, §7 · p. 13

  • Takeaways & Limitations

    The derived optimal guidance weight decays with training time, formalizing the practice of annealing guidance strength.

    Source: §1 · p. 2, §4.4 · p. 8

    The bounded-divergence guarantee assumes the guidance operator's divergence remains bounded along the optimization trajectory.

    Source: §4.1 · pp. 5–6

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become the dominant paradigm for eliciting multi-step reasoning in large language models, and a recent wave of methods (LUFFY, ExPO, PAPO, TAPO) further augments RL with \emph{external guidance} - expert traces, self-explanations, or retrieved thought patterns. Although each method reports empirical gains, none provides convergence rates, bias bounds, or an optimal weighting rule for the guidance signal. We close this gap with \emph{Guidance-Augmented GRPO} (GA-GRPO), a unified theoretical framework that casts external guidance as a stochastic guidance operator G re-writing the question distribution, and analyses the resulting policy-gradient estimator as a biased on-policy estimator whose bias is bounded by the total-variation guidance divergence delta\_G between the guidance-augmented sampling distribution and the policy's own distribution. The framework subsumes vanilla GRPO, LUFFY, ExPO, PAPO, and TAPO as special cases obtained by particular choices of G. Under smoothness and bounded-divergence assumptions we prove that GA-GRPO converges at rate O(1/sqrt(T)) to an O(delta sqrt(T))-neighbourhood of the GRPO stationary point, derive the closed-form MSE-optimal guidance weight lambda-star(T, delta, sigma\_0 squared) = sigma\_0 squared / (sigma\_0 squared + R\_max squared delta squared T), and prove a matching minimax lower bound showing the Omega(delta squared T) bias term is unavoidable. Experiments on Qwen2.5-Math-7B-Base across nine math and OOD benchmarks confirm that optimal-weight GA-GRPO matches or surpasses TAPO, LUFFY, ExPO, and vanilla GRPO while requiring 31\% fewer GPU-hours, and eight analysis experiments validate each theoretical prediction.

1 Introduction

External-guidance methods have reported empirical gains, but their convergence, estimator bias, and guidance weighting lacked a unified theoretical account. GA-GRPO models guidance as a stochastic operator and provides theoretical guarantees and experiments supporting its use.

1 Introduction

  • Prior guidance methods reported benchmark gains, but did not provide convergence rates, bias bounds, or principled weighting rules, leaving practitioners to tune guidance expensively.The methods include LUFFY's expert traces, ExPO's answer-conditioned self-explanations, PAPO's perception-error penalties, and TAPO's retrieved thought patterns.Source: §1 · pp. 1–2
  • GA-GRPO treats external guidance as a stochastic operator that rewrites the question distribution, bounding policy-gradient bias by the divergence from the policy's distribution.The framework includes vanilla GRPO, LUFFY, ExPO, and TAPO as special cases.Source: §1 · p. 2
  • The theory predicts guidance helps when δ²T is small relative to σ0², hurts in late training as bias accumulates and variance shrinks, and should be downweighted over time.These predictions were confirmed across nine benchmarks, including MATH-500, AIME 2024, GSM8K, GPQA-Diamond, ARC-C, and MMLU-Pro.Source: §1 · p. 2
  • Under smoothness and bounded-divergence assumptions, GA-GRPO converges at O(1/sqrt(T)) to an O(δ√T)-neighbourhood of the GRPO stationary point.A matching minimax lower bound establishes that the Ω(δ²T) bias term is unavoidable.Source: §1 · p. 2
  • The framework derives a closed-form MSE-optimal guidance weight that decays with training time.The weight depends on training step, guidance divergence, and baseline estimator variance.Source: §1 · p. 2
  • 31% fewer GPU-hours; optimal-weight GA-GRPO matches or surpasses TAPO, LUFFY, ExPO, and vanilla GRPO across nine math and OOD benchmarks.Eight analysis experiments are also reported to validate the theoretical predictions.Source: §1 · p. 3

2 Related Work

Related work spans RL and GRPO for reasoning, empirical methods that inject external guidance, and theoretical tools for policy-gradient stability and bias-variance analysis. The paper positions GA-GRPO as a unified theoretical account of guidance-augmented RL for reasoning.

2 Related Work

  • GRPO uses group-relative advantages and verifiable rewards for reasoning [DeepSeek-AI, 2025], following RLHF with PPO [Ouyang et al., 2022; Schulman et al., 2017] and reward-model-free preference optimization [Rafailov et al., 2023].Chain-of-thought prompting [Wei et al., 2022] and process reward modelling [Lightman et al., 2023] are presented as orthogonal sources of intermediate supervision.Source: §2 · p. 3
  • LUFFY mixes expert traces with on-policy rollouts [Yan et al., 2025], ExPO bootstraps from answer-conditioned self-explanations [Zhou et al., 2025], TAPO retrieves MCTS-mined thought patterns, and PAPO targets multimodal perception errors [Wang et al., 2025].These guidance methods are described as empirical; GA-GRPO provides a framework that subsumes them as special cases.Source: §2 · p. 3
  • Prior theory covers policy-gradient convergence [Zhang et al., 2019], trust-region stability [Schulman et al., 2015], off-policy estimators [Huang & Jiang, 2020; Papini et al., 2024; Daley et al., 2023], and score-function bias-variance trade-offs [Farquhar et al., 2019].Related work also includes historical-trajectory natural policy gradients [Lin et al., 2024].Source: §2 · pp. 3–4
  • Adjacent work studies structured external signals in vision-language learning [Zhou et al., 2024a;b], medical LLMs and agents [Zhou et al., 2025a; Zhou et al., 2025b; Zhou et al., 2026a], and long-reasoning memory architectures [Wang et al., 2024; Wang et al., 2025].The paper connects these paradigms to balancing external signals with internal capabilities.Source: §2 · p. 4

3 Guidance-Augmented GRPO

Section 3 formulates GRPO for autoregressive reasoning and generalizes it with a stochastic operator that rewrites questions before rollout sampling. It defines guidance divergence and combines on-policy and guided estimators, recovering existing methods through particular choices of the operator and weighting.

3.1 Problem Formulation

  • Questions yield autoregressive token rollouts with terminal verifiable rewards; GRPO uses group-relative advantages from G rollouts per question.Source: §3.1 · p. 4

3.2 The Guidance Operator

  • G rewrites questions as distributions; TAPO retrieves patterns, LUFFY injects expert traces, ExPO adds answer-conditioned explanations, and PAPO adds perception-error feedback.Source: §3.2 · p. 4
  • Guidance divergence is defined for an operator relative to a policy, with D_TV denoting total-variation distance.Source: §3.2 · p. 5

3.3 The GA-GRPO Estimator

  • The estimator mixes on-policy and guidance-augmented GRPO estimates; λ=0 recovers vanilla GRPO, while λ=1 with G_P, G_T, G_S, or G_M recovers TAPO, LUFFY, ExPO, or PAPO.Source: §3.3 · p. 5

3.4 Notation Table

4 Theoretical Analysis

GA-GRPO models external guidance as a source of distribution shift, then analyzes its bias, variance, convergence, and optimal weight under boundedness and smoothness assumptions. The theory characterizes when guidance helps and how its weight should change during training.

4.1 Assumptions

  • The analysis assumes a smooth log-policy, bounded guidance divergence and rewards, and a bounded second moment for the estimator.The paper notes that smoothness is standard for softmax policies and that bounded rewards are immediate for verifiable binary rewards.Source: §4.1 · pp. 5–6

4.2 Bias and Variance of GA-GRPO

  • Guidance mixes policies on rewritten questions; the estimator’s bias is bounded by total-variation divergence, while its variance is separately bounded under the assumptions.The per-rollout policy-gradient signal is the log-policy gradient multiplied by the policy advantage.Source: §4.2 · p. 6

4.3 Main Theorems

  • Under the stated assumptions, GA-GRPO converges at O(1/sqrt(T)) to an O(RmaxδG sqrt(T))-neighbourhood of the GRPO stationary point.The convergence result is paired with a minimax lower bound on the guidance-induced bias scaling.Source: §4.3 · p. 6
  • The MSE-optimal guidance weight decays as 1/T for large T, tending to full guidance as δ approaches zero and no guidance as δ grows without bound.The weight balances estimator variance against the guidance-induced bias.Source: §4.3 · p. 7, §4.4 · p. 8
  • A matching minimax lower bound shows that the guidance-bias scaling in the convergence bound is unavoidable.The lower bound uses a two-question construction and the Le Cam two-point method.Source: §4.3 · pp. 7–8
  • The framework subsumes existing methods as special cases through particular choices of the guidance operator.The listed cases include vanilla GRPO, LUFFY, TAPO, ExPO, and PAPO.Source: §4.3 · p. 8

4.4 Discussion

  • Guidance helps when δ²T is small relative to σ0², but its benefits diminish beyond a crossover step as bias grows and variance shrinks.The optimal guidance weight decays as 1/T for large T.Source: §4.4 · p. 8

5 Experiments

The experiments evaluate GA-GRPO on math and OOD reasoning benchmarks, comparing it with existing RL and guidance methods. GA-GRPO leads on reported average scores and reaches the AIME accuracy target with fewer GPU-hours; ablations measure contributions from its components.

5.1 Setup

  • The evaluation covers math, competition-level OOD, and general-reasoning benchmarks, using Qwen2.5-Math-7B-Base and seven comparison baselines.The nine reported benchmarks include MATH-500, AIME 2024, AMC, GSM8K, GPQA-Diamond, ARC-C, and MMLU-Pro.Source: §5.1 · p. 8

5.2 Main Results

  • GA-GRPO achieves the best math-benchmark average (55.8) and OOD average (50.7), and reaches 50% AIME 2024 accuracy with 31% fewer GPU-hours than TAPO.Its math average gain over GRPO is +5.6; it uses 47% fewer GPU-hours than GRPO to reach the AIME target.Source: §5.2 · p. 9, Table 2 · p. 9, Table 3 · p. 9

5.3 Ablation Study

  • Removing the optimal-weight schedule costs 2.9 AIME 2024 points, while removing guidance retrieval costs 9.4 points; random retrieval costs 5.2 points.Using λ = 1 throughout costs 3.2 points and is unstable in late training; λ = 0 throughout recovers GRPO.Source: §5.3 · p. 9, Table 5 · p. 10

6 Analysis

The analysis experiments test GA-GRPO’s parameter sensitivity, estimator bias and variance, convergence, optimal-weight rule, method stability, lower-bound tightness, and component contributions. Their results align with the theoretical predictions, including close empirical fit for the MSE and optimal-weight analyses.

A1. Parameter Sensitivity

  • Accuracy peaks near λ = 0.5, degrades as δmax grows, and improves with group size up to G = 16 before plateauing.The λ sweep also shows degraded accuracy toward both λ = 0 and λ = 1.Source: §6 · p. 10, Figure 1 · p. 11

A2. Bias-Variance Decomposition

  • The empirical MSE closely tracks the theoretical prediction across λ and δ configurations, with Pearson correlation r = 0.94.The analysis reports bias growing linearly with λ and quadratically with δ, while variance is U-shaped in λ; their sum has a unique minimizer matching λ⋆.Source: §6 · p. 10, Figure 2 · p. 11

A5. Method Comparison

  • GA-GRPO dominates GRPO, LUFFY, TAPO, and ExPO on six radar dimensions, with its largest margins on sample efficiency and stability.The six dimensions are math accuracy, OOD accuracy, sample efficiency, stability, bias, and human-rated explainability.Source: §6 · p. 11, Figure 4 · p. 12

A6. Stability Analysis

  • Final AIME 2024 accuracy has an IQR of 1.8 for GA-GRPO, versus 3.4 for TAPO and 4.7 for GRPO across five seeds.The reported tighter box is consistent with the prediction that optimal weighting controls both bias and variance.Source: §6 · p. 11, Figure 4 · p. 12

A7. Lower-Bound Tightness

  • The empirical bias curve lies between the theoretical bounds and tracks their δT scaling, supporting that the upper bound is tight in δ up to a constant factor.Source: §6 · p. 11

A4. Optimal Guidance Weight Surface

  • Empirically optimal λ values closely match the theoretical λ⋆ surface, with mean absolute deviation 0.04.The theoretical surface is overlaid with empirically optimal values obtained by grid search.Source: §6 · pp. 10–11, Figure 3 · p. 12

A3. Convergence Dynamics

  • For δ = 0, gradient norm decays as 1/√T; for δ > 0, it plateaus at a floor proportional to Rmaxδ√T, as predicted.The experiments vary δ across 0, 0.1, 0.3, and 0.5.Source: §6 · p. 10, Figure 3 · p. 12

A8. Component Contribution

  • On MATH-500, ablations show the optimal-weight schedule contributes +3.4 points, adaptive δ estimation +1.5, IS clipping +1.0, and PCC retrieval +2.5.The theory-driven components collectively account for 5.9 of the 6.6-point gain over GRPO.Source: §6 · p. 12

7 Conclusion

GA-GRPO formalizes external guidance as a stochastic operator and bounds estimator bias by guidance divergence, while yielding a convergence guarantee and a closed-form MSE-optimal weight. It recovers TAPO, LUFFY, ExPO, and PAPO as special cases, and experiments across nine benchmarks report competitive or better results with fewer GPU-hours.

7 Conclusion

  • GA-GRPO bounds policy-gradient estimator bias by guidance divergence and proves convergence to an O(1/√T)-neighbourhood of the GRPO stationary point.The framework formalizes external guidance as a stochastic guidance operator.Source: §7 · p. 12
  • The framework yields a closed-form MSE-optimal guidance weight, recovers TAPO, LUFFY, ExPO, and PAPO as special cases, and has a matching minimax lower bound.Source: §7 · pp. 12–13
  • Across nine benchmarks, optimal-weight GA-GRPO matches or surpasses TAPO, LUFFY, ExPO, and vanilla GRPO while requiring 31% fewer GPU-hours.Eight analysis experiments validate the theoretical predictions.Source: §7 · p. 13
Loading 2610.06861v1…