PaperlayerView exact revision on arXiv

Source-linked AI summary

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison

arXiv:2608.16739v1cs.LG

TL;DR

Value-function RL offers token-level credit but faces infrastructure costs and practical competition from group-relative methods. This paper introduces Privileged Value Functions and TETHER, which improve value-based training across reasoning tasks and provide a low-risk integration path with GRPO infrastructure.

  • Problem

    Group-relative LLM RL methods provide only sequence-level credit, while value functions face infrastructure costs that complicate their adoption.

  • Method

    The paper strengthens value-based advantages with privileged information and introduces TETHER, which interpolates between group-relative and value baselines.

  • Results

    Across several reasoning tasks, privileged value functions and TETHER improve over standard value-function baselines, with TETHER consistently outperforming VF across four tasks.

  • Takeaways & Limitations

    The methods offer a low-risk path for integrating value functions into GRPO pipelines and support their use as practical components of large-scale LLM post-training.

  • Takeaways & Limitations

    Experiments use 4B models and tasks with response lengths up to 32,000 tokens, limiting evidence at larger scales and in longer-horizon agentic settings.

Abstract

from arXiv · show

Reinforcement learning algorithms for Large Language Models (LLMs) are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.

1 Introduction

The introduction contrasts critic-based value functions with critic-free GRPO, whose group sampling sacrifices token-level credit and increases straggler and off-policyness costs. It motivates two methods that exploit additional information to strengthen value-based advantages for reasoning-heavy LLM RL.

  • Background: Value functions amortize noisy Monte-Carlo return prediction and support value-based control and actor-critic reinforcement learning.They estimate expected future return from states or state-action pairs.
  • Related work: LLM reinforcement learning has shifted from PPO with trained scalar-headed critics toward GRPO, which estimates advantages from groups of sampled responses.This shift reflects the practical use of critic-free methods in LLM RL.
  • Motivation: GRPO-style critic-free methods discard temporally fine-grained credit assignment, while large groups increase straggler delays and can worsen off-policyness in asynchronous RL.Training waits for the slowest response in each group, motivating methods that better justify value-function infrastructure costs.
  • Contribution: The paper introduces two methods that exploit value functions’ access to policy-unavailable information, including group information, to construct stronger value-based advantages.The methods are evaluated across several reasoning-heavy tasks and improve RL training.

1. Privileged value functions

Privileged Value Functions (PVFs) condition on more than the current LLM state by using additional information hidden from the policy. This extra information improves value estimation and, in turn, policy optimization.

  • Privileged value functions: Privileged Value Functions condition on more than the current LLM state.They extend the information available to value functions beyond the current policy state.
  • Privileged value functions: PVFs use additional information hidden from the policy.The privileged information is available to the value function but not to the policy.
  • Privileged value functions: Improved value estimation from PVFs supports policy optimization.The proposed mechanism is intended to improve value estimation and thereby policy optimization.

2. Adaptive group–value baselines

TETHER adaptively combines group and value baselines to address their complementary weaknesses: unreliable value estimates when the critic is poorly fitted and limited token-level credit from group baselines.

  • 2. Adaptive group–value baselines: TETHER smoothly interpolates between group and value baselines, combining the group baseline’s reliability with the value baseline’s token-level credit.The method is designed for cases where the critic is poorly fitted, while retaining the group baseline’s dependable behavior.

2 RL preliminaries

LLM RL methods share a policy-gradient loop but differ in advantage estimation and update stabilization. Group-relative estimators provide sequence-level credit from multiple responses, while value functions provide token-level credit and can operate with one rollout.

  • LLM RL samples responses, assigns token advantages, and stabilizes policy updates; algorithms primarily differ in how advantages are estimated.
  • Group-relative advantage estimators: GRPO estimates a prompt baseline from K sampled responses, producing sequence-level advantages repeated across every token.RLOO removes dependence on the response’s own return by using the K −1 sibling returns.
  • Baseline unbiasedness: A baseline preserves an unbiased policy gradient when auxiliary information is conditionally independent of the current token given its history.Future tokens, realized rewards, and later trajectory feedback can violate this condition because they may depend on the current token.
  • Value-based advantage estimators: Value functions predict expected returns from token histories, providing token-level credit without requiring rollout groups, including when K = 1.The prompt value equals the expected leave-one-out baseline, while later prefixes track expected reward changes during generation.
  • Temporal-difference and GAE estimators: GAE interpolates between one-step bootstrapping at λ = 0 and unbiased Monte Carlo returns at λ = 1, trading bias against variance.The experiments set λtarget = λGAE = 1.0, using Monte Carlo targets and advantages that yield unbiased policy gradients.

3 Privileged value functions

Privileged value functions route training-time information unavailable to the policy through the critic, improving value estimation and reducing policy-gradient variance without changing the original policy objective when admissibility holds. The section describes reference solutions, leave-one-out responses, and other task-specific context, while noting that usefulness depends on what the value model can learn to exploit.

  • 3 Privileged value functions: PVFs condition the critic on policy token history plus privileged training-time context, improving return prediction and reducing policy-gradient variance.Examples include oracle answers, verifier rubrics, latent environment states, and other group responses.
  • 3 Privileged value functions: Under the baseline admissibility condition, privileged context preserves an unbiased policy-gradient estimator; future tokens, realized rewards, and subsequent feedback are excluded.A fixed reference answer is admissible, while information generated by the current response is not.
  • 3 Privileged value functions: PVF benefits are not automatic: excessive context can hurt performance, whereas easier-to-use context can help even without additional information gain.PVFs can also use verifier rubrics or detailed task specifications unavailable to the policy.
  • 3 Privileged value functions: Reference solutions simplify value prediction by letting the critic assess whether a partial trajectory is progressing toward a known correct end state.Examples include oracle answers, mathematical proof sketches, and gold patches for code repair.
  • 3 Privileged value functions: Other K −1 independently sampled responses and their rewards provide leave-one-out privileged context, effectively turning value prediction into an in-context learning task.This complementary benefit later motivates TETHER’s combination of leave-one-out and learned value-function baselines.

4 Privileged value function experiments

Across four experiments in three environments, privileged conditioning improves value-based reinforcement learning, with gains varying by task and group setting. The privileged signal affects policy learning through baseline variance reduction, and PVF explains more return variance than VF in every environment.

  • Experimental setup: The experiments compare MEAN, ordinary token-level VF, and privileged PVF under matched policy-training settings using Monte Carlo targets and advantages.PVF uses the same value-training configuration as VF but conditions on task-specific privileged context; all runs use Qwen3-4B-Instruct-2507.
  • Overall results: Across all four tasks, privileged conditioning improves the value baseline, while its gains vary substantially and affect policy learning only through baseline variance reduction.The experiments use Monte Carlo advantages, so privileged information does not directly alter the policy objective.
  • Reasoning Gym: In Reasoning Gym, PVF and VF improve similarly at K = 1, VF plateaus earlier, and both value baselines outperform MEAN; the improvement remains smaller but present at K = 8.The PVF receives the ground-truth answer, with evaluations in both no-group K = 1 and grouped K = 8 settings.
  • CodeIO: In CodeIO, PVF surpasses both VF and MEAN despite using no task-specific information, instead conditioning on other rollout responses and their returns.VF slightly underperforms MEAN, while PVF’s advantage widens over training.
  • Explained variance: PVF explains more return variance than VF in every environment, and the explained-variance improvement correlates with final reward gaps.With λGAE = 1, explained variance directly measures the advantage variance reduction provided by the critic.
  • Group-size considerations: Increasing group size reduces variance through both more accurate baselines and more averaged trajectories, with the latter benefiting value methods as well as MEAN.The experiments use relatively small groups, K ∈{4,8}, alongside batch sizes of 64 and 128.

5 TETHER: A group-aware value baseline

TETHER adaptively interpolates between leave-one-out group and token-level value baselines, starting with group-relative advantages and shifting toward learned values as critic accuracy improves. It fits and smooths the mixture coefficient from return prediction while preserving an unbiased policy gradient.

  • Method: TETHER is an adaptive linear combination of the leave-one-out group baseline and learned token value baseline.It is designed for settings where value baselines outperform group means only after the critic is fit well.
  • Method: At ρ = 0, TETHER recovers the leave-one-out group advantage; at ρ = 1, it recovers the token-level value advantage.Intermediate ρ values add token-level variation while dampening critic prediction errors with the group component.
  • Adaptive coefficient: TETHER fits the mixture ratio to best predict observed return-to-go, allowing the baseline to adapt as value-function quality changes during training.The least-squares estimate is computationally cheap enough to fit every step, and EMA smoothing reduces noise in successive estimates.
  • Adaptive coefficient: Advantages for batch Bk use the previous smoothed coefficient ρk−1, while the coefficient fitted from Bk returns is applied only to Bk+1.Using the updated coefficient on Bk would make its baseline depend on its own returns and bias the policy gradient.
  • Training behavior: Training initializes with ρ = 0 at the group baseline, and the coefficient is expected to move toward ρ = 1 as critic accuracy improves.This avoids task-specific hyperparameter tuning for transitioning from group-relative to token-level value estimates.

6 TETHER experiments

TETHER adaptively combines group-relative and value-function baselines in experiments spanning four reasoning tasks, consistently improving over VF while matching or exceeding MEAN in several settings. Its mixture coefficient begins at zero and becomes task-dependent as the critic explains return variance beyond the group mean.

  • Experimental setup: TETHER is evaluated against MEAN and VF on four tasks, omitting K = 1 Reasoning Gym and adding the multi-turn Lean task MiniF2F.The shared value-function training configuration uses EMA decay d = 0.95; MiniF2F provides compiler feedback for up to three attempts.
  • Results: TETHER consistently outperforms VF across all four tasks, substantially mitigating VF’s degradation relative to MEAN on Sudoku.It does not fully recover MEAN’s performance on Sudoku, but supports using value functions in GRPO pipelines that perform well with mean baselines.
  • Results: TETHER outperforms MEAN in Reasoning Gym and MiniF2F, matches MEAN in CodeIO, and narrows the gap in Sudoku.The comparison uses seed-averaged training reward curves for MEAN, VF, and adaptive TETHER baselines.
  • Adaptive mixture: The mixture coefficient ρ starts at 0, initially favoring the LOO baseline, then moves away from zero as the critic explains return variance beyond the group mean.The converged ρ value is strongly task-dependent, with Sudoku reaching the largest value-function mixture weight despite its weakest VF performance.
  • Adaptive mixture: Values closer to ρ = 1 indicate that the value function predicts return-to-go better than the mean baseline.This interpretation defines the adaptive coefficient shown across experiments.

7 Potential value function research for future work

The paper identifies value functions as a promising direction for scalable LLM post-training, highlighting privileged information, calibrated GAE hyperparameters, and token-position-dependent baseline mixing as opportunities for improvement.

  • Motivation: Privileged value techniques, combined with engineering optimization, could make value functions practical and scalable for large-scale LLM post-training.The authors aim to motivate renewed research on value functions for LLM reinforcement learning.
  • Hyperparameter calibration: λGAE = 0.999888 significantly improves both VF and PVF rewards over λGAE = 1 on Reasoning Gym.The calibrated value retains approximately 40% of the unbiased terminal signal at an 8192-length response; main experiments still used λGAE = λtarget = 1.0.
  • Adaptive baseline mixing: A single coefficient ρ may be inadequate because relative group- and value-baseline quality can vary across token positions.In Sudoku, value prediction may become easier later, when the critic mainly validates an almost-complete grid rather than marginalizing over many completions.
  • Adaptive baseline mixing: Bucket-specific ρm coefficients can be fit with the return-prediction objective and separately smoothed with an EMA.Buckets that are too small provide fewer token samples, making coefficient estimation noisier.

8 Limitations

The study’s limitations are its added infrastructure and accelerator costs, incomplete compute matching, and restricted experimental scale. Experiments use 4B models and responses up to 32,000 tokens, leaving long-horizon agentic training for future work.

  • Compute and infrastructure: Value function inference and training add accelerator cost and require dedicated GPU allocation.The paper identifies value functions as adding infrastructure cost to the workload.
  • Compute and infrastructure: MEAN and value function baselines are not exactly compute-matched, whereas VF and PVF settings are exactly matched.The experiments match only the number of inference trajectories for MEAN and value function baselines.
  • Experimental scale: Experiments are limited to 4B models and tasks with maximum response lengths of 32,000 tokens.The 32,000-token maximum occurs on the Sudoku task; future work should extend evaluation to long-horizon agentic training.

9 Conclusion … D.1 Privileged value function experiments

The paper concludes that privileged conditioning makes value functions a reliable control-variate alternative to self-distillation, while TETHER offers a low-risk interpolation between group-relative and value-based RL. Its supporting infrastructure asynchronously trains and evaluates critics, and experiments specify task-specific privileged contexts, replay, warmup, and loss choices.

  • 9 Conclusion: Value functions may also support non-terminal learning signals for partial trajectories and inference-time scaling, especially for longer-horizon agents.These uses extend beyond the variance-reduction role studied in the paper.
  • 9 Conclusion: Privileged value conditioning and TETHER improve value-function RL while preserving a practical path from existing GRPO infrastructure.Privileged information adds task-relevant signal without biasing the policy objective, while TETHER interpolates between group-relative and value baselines.
  • A Related work: Related work spans privileged-information critics, hard endpoint selection by EVPO, critic-free branching rollouts, and on-policy self-distillation using training-time information.TETHER instead smoothly interpolates between group and value baselines.
  • B Asynchronous value function training infrastructure; B.1 Value evaluation and training; B.2 FIFO-bounded replay and controlled reuse: The implementation uses asynchronous value training, evaluator serving, replay-buffer reuse, and separate or colocated compute placements to avoid making value learning a synchronization barrier.The evaluator predicts returns for partial responses and can overlap inference with trainer updates in dedicated placement; replay reuse is bounded at N = 2.
  • B.3 Value warmup before policy training; B.4 Value loss: Critics are initialized from the base policy, receive 20 value updates before policy optimization, and use binary classification for binary rewards while predicting continuous values in [0,1].Early experiments found a small improvement over MSE, but the effects of value loss and support require more systematic study.
  • C Statistical analysis of TETHER: TETHER selects a coefficient ρ between the leave-one-out group baseline and token-level value prediction by minimizing Monte Carlo squared error, with interior mixtures helping when errors are complementary.The fitted coefficient typically remains inside [0,1] in real training runs, so TETHER uses information from both baselines.
  • C Statistical analysis of TETHER: TETHER is not the optimal policy-gradient baseline because it omits the ∥s∥2 term, and its practical advantage depends on stable coefficient estimation and complementary baseline errors.EVPO may be preferable near an endpoint, with small batches, or when the critic changes too quickly; finite-data fitting can overfit.
  • D Experimental hyperparameters and settings; D.1 Privileged value function experiments: Experiments keep policy configurations shared across baselines and hold VF and PVF value-training compute constant, varying only critic context across Reasoning Gym, CodeIO, and Sudoku.The PVF receives a reference answer in Reasoning Gym, the other three group responses and returns in CodeIO, and a complete solved grid in Sudoku.

D.2 TETHER experiments · D.3 Common training hyperparameters

TETHER experiments use task-specific model, rollout, batch, group-size, and compute configurations, while common policy and value-training settings are held fixed across baseline comparisons. Value-backed runs consistently add dedicated value-training infrastructure beyond mean-baseline runs.

  • D.2 TETHER experiments: Reasoning Gym uses two seeds, whereas CodeIO, Sudoku, and MiniF2F use three seeds.All TETHER runs use EMA decay d = 0.95 for the fitted mixture coefficient.
  • D.2 TETHER experiments: Across tasks, TETHER uses task-specific policy steps, batch sizes, group sizes, token limits, and model configurations.Reasoning Gym uses Qwen3-4B-Instruct-2507, batch size 128, K = 8, and 800 steps; CodeIO uses K = 4 and 650 steps; Sudoku uses K = 4 and 600 steps; MiniF2F uses Qwen3.5-4B, K = 4, and 500 steps.
  • D.2 TETHER experiments: Value-backed VF and TETHER runs add one value trainer and one dedicated value evaluator relative to mean-baseline infrastructure.This yields four total nodes for Reasoning Gym, five for CodeIO, six for Sudoku, and six for MiniF2F.
  • D.2 TETHER experiments: Reasoning Gym uses 8,192 total sequence tokens and 6,144 completion tokens, while CodeIO uses 4,096 input tokens and up to 8,192 generated tokens.These task-specific token limits are part of the reported rollout configurations.
  • D.2 TETHER experiments: Sudoku limits trajectories to 32,768 tokens and each turn output to 8,192 tokens; MiniF2F permits 4,096 generated tokens per attempt and 24,576 tokens per trajectory.MiniF2F additionally permits three proof attempts with compiler feedback for each theorem.
  • D.3 Common training hyperparameters: Table 1 policy and rollout settings are held fixed across baseline comparisons.These settings use PRIME-RL’s default DPPO objective: an importance-weighted token policy gradient with masking of large probability changes and a small squared log-ratio KL penalty.
  • D.3 Common training hyperparameters: Table 2 value-training-specific settings are shared by all VF, PVF, and TETHER runs.The supplied passage identifies Table 2 as the source of value-function hyperparameters shared across value-backed runs, without listing individual values.
Loading 2608.16739v1…