PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

Hongyang Li, Xiao Li, Caesar Wu, Said Mammar, Grégoire Danoy, Pascal Bouvry

arXiv:2609.37119v1cs.LGcs.AIcs.CL

TL;DR

Reward pipelines rely on annotations or reference answers, while critics are often removed despite learning to predict outcomes. RFPO reuses a frozen, calibrated critic for rewards, baselines, and unfinished-prefix forecasts; under a 4,096-token cap, it validates above matched supervised PPO with 19% fewer GPU-hours.

  • Problem

    Human feedback and reward models require annotation projects, while verifiable rewards require reference answers for training prompts.

    Source: §1 · p. 2

  • Method

    RFPO freezes and calibrates a pretrained critic to provide rollout rewards, GAE baselines, and success forecasts for unfinished prefixes.

    Source: §1 · p. 2

  • Results

    At a 4,096-token cap, RFPO validates above supervised PPO trained under the same cap on 19% fewer GPU-hours.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    A frozen critic can provide a reward before a rollout finishes, avoiding the need to wait for every trajectory to complete.

    Source: §5 · p. 8, §7 · p. 9

    Experiments use one 4B model family on mathematical reasoning; larger models and other tasks remain untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent approaches to reinforcement learning (RL) post-training for large language models increasingly remove the critic to reduce training instability and memory overhead. Even where a critic is trained, it is discarded once training ends, although it has learned to predict outcomes. We revisit this trend and show that a pretrained critic's ability to predict future outcomes can make it a valuable asset for efficient long-horizon reasoning. First, we find that instability in critic-based RL for long chain-of-thought reasoning is largely an optimization artifact: keeping policy updates small and low in variance restores stable convergence. Second, a well-pretrained critic estimates the posterior probability of eventual success from later trajectory states and unfinished prefixes. Its predictions provide outcome-derived, dense, per-prefix learning signals that, during policy optimization, require neither completed rollouts, step-level annotations, nor external reward labels. Building on this insight, we introduce Reward-Free Policy Optimization (RFPO), which repurposes a single calibrated, frozen critic as a rollout-level reward, a value baseline for generalized advantage estimation, and a success forecaster for unfinished prefixes. We further show that binarizing the debiased score stops the policy from exploiting the critic's length bias. Binarized, RFPO matches supervised PPO without a single label in the training loop, while cutting compute and memory overhead. This makes RFPO well suited to long-horizon reasoning tasks, where outcomes arrive late and generation dominates cost: because rollouts can be rewarded before they finish, training no longer has to pay for waiting on every trajectory to complete. Our findings challenge the prevailing critic-free paradigm and establish critic-based, reward-free optimization as a scalable and computationally efficient path for LLM post-training.

1 INTRODUCTION

The paper asks whether a well-trained critic can serve as reward, not only as a policy-gradient baseline. RFPO repurposes one frozen, calibrated critic for rewards, GAE baselines, and unfinished-prefix success forecasts, and reports stronger validation than supervised PPO under a tighter training cap.

1 INTRODUCTION

  • Existing reward routes rely on annotation projects or reference answers, each attaching a reward to rollouts at a cost that does not scale with desired RL.The paper contrasts human feedback and reward models with verifiable rewards requiring reference answers.Source: §1 · p. 2
  • RFPO uses one frozen, calibrated critic as rollout reward, GAE baseline, and success forecaster for unfinished prefixes, without external labels during policy optimization.The critic estimates posterior success from later states and unfinished prefixes.Source: §1 · p. 2
  • The paper argues critic instability in long chain-of-thought RL is an optimization artifact that recedes when policy updates are small and low in variance.It reports stable convergence after reducing update size and variance.Source: §1 · p. 2
  • At a 4,096-token cap, RFPO validates above supervised PPO trained under the same cap on 19% fewer GPU-hours.About half of RFPO’s training rollouts are unfinished.Source: §1 · pp. 2–3

2 RELATED WORK

The paper distinguishes its outcome-trained critic from methods that judge written steps or replace labels with other signals. It grounds its reward in verifier outcomes and uses the critic to forecast unfinished trajectories.

2 RELATED WORK

  • Process reward models give per-step signals but require step-level annotation, whereas a critic forecasts where an outcome-trained prefix will end.The critic learns from outcomes alone.Source: §2 · p. 3
  • Label-free alternatives use voting, self-rewarding, model confidence, or generated rubrics, while implicit-reward methods such as DPO and PRIME still need training labels.The passage notes that label-free alternatives without grounded supervision are prone to reward hacking.Source: §2 · p. 3
  • RFPO’s reward is grounded in verifier outcomes obtained before policy optimization, and its critic forecast can score unfinished trajectories when sparse terminal rewards cannot.The passage contrasts this with remedies that truncate rollouts or sparsify attention but cannot score unfinished trajectories.Source: §2 · p. 3

3 REWARD-FREE POLICY OPTIMIZATION

RFPO reuses a pretrained critic as reward, baseline, and success forecaster. The method relies on critic quality and calibration to make its scores useful before rollouts finish and resistant to length bias.

3.1 A CRITIC WORTH CONSULTING

  • The critic’s ideal value at each prefix estimates the final verifier reward; with sparse binary terminal rewards and γ=λ=1, that is the probability of eventual success.RFPO uses the frozen critic as rollout reward, GAE baseline, and forecaster for unfinished prefixes.Source: §3.1 · p. 3, §3 · p. 3
  • The frozen critic separates correct from incorrect rollouts, reaching AUC 0.94 on early-policy rollouts and 0.96 on converged-policy rollouts at the step-800 checkpoint.Across nine checkpoints, within-problem AUC tracks explained variance (r=0.91), and exhausted-budget rollouts remain rankable.Source: §3.1 · p. 3
  • Within-problem prefix ranking rises with visible reasoning, reaching 0.84 at 4,096 tokens and 0.84–0.92 on prefixes without an answer.Overall AUC is already 0.958 at 256 tokens, largely because the critic recognizes problem difficulty.Source: §3.1 · p. 3

3.2 FREEZING AND CALIBRATING THE CRITIC

  • RFPO freezes the critic and uses its terminal value as reward and token-level values as the GAE baseline, with both obtained in one forward pass.The reward is scored at the last observed token whether a rollout ends naturally or reaches the generation cap.Source: §3.2 · p. 4
  • Calibration subtracts length-decile offsets estimated within correctness class and applies a threshold fitted to match the verifier-positive rate; binarization blocks reward for raising scores above the cut.The fitted threshold is 0.5215 at 93.4% calibration accuracy on the deployed pair.Source: §3.2 · p. 4

4 EXPERIMENTAL SETUP

The experiments compare RFPO and supervised PPO from a shared policy-and-critic starting point under matched PPO settings. Calibration is fitted once, and evaluation uses fixed suites and budgets.

4 EXPERIMENTAL SETUP

  • The initial policy and critic come from supervised PPO training, with the policy extracted at cumulative step 700 and the critic frozen at step 800.The pipeline’s only verifier-labeled supervision is roughly 0.56M trajectories.Source: §4 · p. 6
  • Main comparison runs share the initial policy, critic, prompt order, and PPO settings; supervised PPO uses verifier labels and updates its critic, while RFPO uses Eq. (2) and freezes it.Training runs use a 5,120-token cap unless an arm specifies otherwise.Source: §4 · p. 6, Table 1 · p. 6
  • Both methods use the clipped PPO objective with 128 prompts and eight responses per prompt, while the three zero-label runs truncate 26.5–28.5% of training generations.Training generations are capped at 5,120 tokens and validation generations at 12,288.Source: §4 · p. 6
  • Calibration is fitted once on 512 on-policy rollouts, using within-correctness length-decile offsets and a threshold that matches the corrected scores’ positive rate to the verifier-positive rate.The threshold is 0.5215, and the binary rule agrees with the verifier on 93.4% of the calibration sample.Source: §4 · p. 6
  • Validation evaluates AIME 2026 and AMC 2023 every ten steps with eight samples per problem at temperature 0.7 and a 12,288-token budget.The reported training-time validation metric is the equally weighted macro average of the two suites.Source: §4 · p. 6

5 RESULTS

With no verifier labels in its optimization loop, RFPO performs on par with supervised PPO across the reported reasoning benchmarks. It also scores unfinished rollouts and, at a 4,096-token cap, matches or exceeds same-cap supervised PPO while using fewer GPU-hours.

5 RESULTS

  • Across 310–378 steps, all three binarized zero-label runs avoid degeneration, with smoothed response length staying within 13% of its start and no late held-out accuracy collapse.The continuous-reward arms decline later in training as response length drifts with residual length bias.Source: §5 · p. 7
  • Averaged over four suites, the zero-label checkpoint reaches 41.1% pass@1 against 41.8% for supervised PPO at RL step 300.Training-time validation peaks at 25.7% on AIME 2026 pass@1 against 25.0% for supervised PPO, and at 76.6% on AMC 2023 pass@1 against 78.1%.Source: §5 · p. 7
  • Under the 5,120-token protocol, 26.5–28.5% of generations hit the cap, and RFPO scores them at the last observed token rather than waiting for completion.With a 4,096-token cap, about half of every batch remains unfinished for 310 steps.Source: §5 · pp. 8–9
  • At a 4,096-token cap, validation is level with 5,120-token runs (47.2–47.4%), with 264 rather than 327 GPU-hours over 300 steps.Supervised PPO with the full 5,120-token cap reaches 48.3% for 388 GPU-hours.Source: §5 · p. 9, Table 4 · p. 9
  • On matched batches, a step takes 421 rather than 582 seconds (−28%), and peak reserved memory falls by 9.4 GB per GPU.The savings apply to the RL stage; obtaining and calibrating the critic is a separate, one-off preparation cost.Source: §5 · p. 9

6 LIMITATIONS

The experiments cover one 4B-parameter model family and mathematical reasoning; larger models and non-reasoning tasks remain untested.

6 LIMITATIONS

  • Experiments use a single 4B-parameter model family on mathematical reasoning; larger models and other tasks are left to future work.The authors cite experiment expense: one 300-step RL run takes 264–388 GPU-hours on eight A100-80GB GPUs.Source: §6 · p. 9

7 CONCLUSION

The conclusion argues that a pretrained critic can supply success predictions before outcomes arrive, rather than being discarded after baseline use. RFPO uses one frozen critic across reward, baseline, and unfinished-prefix forecasting, and its binarized form matches supervised PPO with lower per-step cost.

7 CONCLUSION

  • A single frozen critic serves as the rollout-level reward, GAE baseline, and forecaster for unfinished prefixes.Its outcome-derived predictions can score truncated rollouts before they finish.Source: §7 · p. 9
  • The conclusion attributes critic instability to the update recipe rather than the value network, and says small, low-variance policy updates restore stability.The critic estimates the likelihood of rollout success before the rollout ends.Source: §7 · p. 9
  • Binarized RFPO gives up the continuous-reward accuracy ceiling in exchange for stable training, matching supervised PPO without a verifier and at lower per-step cost.The authors present scoring unfinished trajectories as a way to avoid waiting for every outcome.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement says reported numbers derive from submission artifacts and describes what is needed to regenerate results.

REPRODUCIBILITY STATEMENT

  • The submission includes artifacts for reported numbers, with the model lineage, PPO configuration, run-to-log mapping, and critic calibration rule documented in the paper and appendices.It also includes training logs, calibration artifacts, per-cell scoring summaries, prefix scans, and benchmark outputs.Source: REPRODUCIBILITY STATEMENT · pp. 9–10
  • A script regenerates every figure and table from the archive; recomputing critic-side measurements additionally requires rollout archives and the frozen checkpoint.The file index lists the sizes and original locations of those additional items.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The appendix documents AI assistance in writing and analysis, then specifies the training and calibration recipe used to rebuild the RFPO pipeline.

AI USE STATEMENT

  • The authors used generative AI to polish writing, draft text, compute summary statistics, and plot figures; they designed and ran experiments and verified reported numbers.The authors state that they made all analysis decisions and take responsibility for the paper.Source: AI USE STATEMENT · p. 10

A TRAINING RECIPES

  • SFT uses Qwen3-4B-Base fine-tuned on 45k OpenR1-Math-220k examples; the epoch-2 checkpoint scored 25.0% on AIME 2026.The default chat template stripped the reasoning block, so the authors patched it; without the patch, AIME accuracy was 5%.Source: A TRAINING RECIPES · p. 13, Figure 5 · p. 13
  • A supervised PPO run on DAPO-Math-17k supplies the initial policy and critic, extracted at cumulative steps 700 and 800, respectively.The critic was trained for 800 steps on 563,200 verifier-labeled rollouts.Source: A TRAINING RECIPES · p. 13
  • The frozen critic is calibrated once using length-decile offsets and threshold τ = 0.5215, achieving 93.4% agreement with the verifier on the calibration sample.Offsets are estimated separately within correct and incorrect response classes; RFPO and supervised PPO share the RL-stage settings.Source: A TRAINING RECIPES · pp. 13–14, Table 6 · p. 14, Table 7 · p. 14

B INNER UPDATES PER BATCH AND TRAINING STABILITY

The appendix finds that PPO runs with multiple optimizer updates per rollout batch often collapse or diverge, whereas single-update runs are usually stable, especially at 1,024 rollouts. The evidence also has configuration confounds and residual failures.

B INNER UPDATES PER BATCH AND TRAINING STABILITY

  • 23 of 29 runs with multiple inner updates failed, while all 25 single-update runs at 1,024 rollouts stayed stable.At eight updates over 256 rollouts, 12 of 15 runs failed; Figure 6 shows multi-update collapse or divergence within 200 steps and the configuration outcomes.Source: B INNER UPDATES PER BATCH AND TRAINING… · p. 14, Figure 6 · p. 15
  • Single-update stability did not come from PPO clipping: the importance ratio was one, and clip fraction and PPO KL were zero in every such run.The authors attribute stability to small steps averaged over many rollouts.Source: B INNER UPDATES PER BATCH AND TRAINING… · p. 15
  • All eight single-update failures used 256 or 512 rollouts and a reward dropped from the final recipe, so they do not implicate the single-update rule.The failures used no calibrated replacement, raw critic scores, or a fixed 0.7 threshold without a length baseline.Source: B INNER UPDATES PER BATCH AND TRAINING… · p. 15
  • The run record supports no reliable multi-update configuration tried, but the stable 1×1,024 group also used the final reward, leaving that confound unresolved.Failures span three starting checkpoints and include all eight multi-update runs with symmetric clipping of 0.2.Source: B INNER UPDATES PER BATCH AND TRAINING… · p. 15

C EVALUATION PROTOCOL AND EXTENDED RESULTS

The appendix reports late-training validation and extended step-300 metrics alongside the evaluation protocol and estimator definitions. Zero-label results are close to supervised PPO on several measures, with some task- and metric-specific differences.

C EVALUATION PROTOCOL AND EXTENDED RESULTS

  • Validation samples eight responses per problem on AIME 2026 and AMC 2023, uses a 12,288-token budget, and scores correctness with a rule-based verifier.Separate step-300 checkpoint evaluations use 16 samples on four suites, including out-of-domain GPQA-Diamond.Source: C EVALUATION PROTOCOL AND EXTENDED… · p. 15
  • Training-time pass@k is a plug-in estimator that is biased at k=n, so 1 − pass@8 is not the fraction of problems missed by all eight samples.Separate evaluation reports pass@16 and maj@16 directly from 16 samples.Source: C EVALUATION PROTOCOL AND EXTENDED… · p. 15
  • On late-training averages, zero-label runs scored 22.1 versus 22.5 for supervised PPO on AIME 2026 pass@1 and 37.2 versus 34.4 on pass@8.Supervised PPO retained a 2.4-point edge on AMC 2023 pass@1; averages use each run’s last ten evaluations.Source: C EVALUATION PROTOCOL AND EXTENDED… · p. 16, Table 8 · p. 16
  • At step 300, supervised PPO scored 50.0 on AIME 2026 pass@16 versus 40.0 for the zero-label checkpoint, a difference of three problems out of 30.The reward regimes stayed close under maj@16; zero-label generations were somewhat longer on AIME and GPQA and truncated at most 2.2%.Source: C EVALUATION PROTOCOL AND EXTENDED… · p. 16, Table 9 · p. 17

D TRAINING DYNAMICS

The appendix examines training dynamics, critic measurements, prefix predictions, policy drift, and generation budgets. It finds useful ranking and unfinished-prefix signals, while showing that calibration and very short generation caps can limit the frozen critic’s reliability.

D TRAINING DYNAMICS

  • Across the main arms, actor gradient norms stayed near 0.09–0.10, while 26–30% of training rollouts were truncated at 5,120 tokens and fewer than 2% of validation responses hit 12,288 tokens.Figure 8 tracks training quantities, including reward-positive-rate gap and critic scores.Source: D TRAINING DYNAMICS · p. 17, Figure 8 · p. 17
  • A frozen reward’s positive-rate gap over verifier solve rate grew to 4.5–5.4 points by step 300, although the critic’s ranking remained stable in healthy arms.The half-labeled arm’s gap ended at 1.3 points, or 2.6 points on its critic-scored half.Source: D TRAINING DYNAMICS · pp. 17–18

E THE CRITIC: MEASUREMENTS AND ROBUSTNESS

  • Within-problem AUC compares correct and incorrect attempts on the same problem, while explained variance measures value-scale fit rather than ranking alone.The appendix also defines AUC variants restricted to finished or truncated rollouts and threshold-based precision.Source: E THE CRITIC: MEASUREMENTS AND… · p. 18

E.1 CRITIC TRAINING DIAGNOSTICS

  • Critic value loss fell from 19 to about 0.05, and mean predictions converged to mean return within about 20 steps, but prediction spread continued narrowing afterward.A step-200 critic ranked rollouts but did not yet produce values readable as probabilities; Figure 9 shows the training diagnostics.Source: §E.1 · p. 18, Figure 9 · p. 18

E.2 THE MEASUREMENT PLANE

  • From critic step 200 to 300, overall AUC rose while optimal threshold τ* shifted from 0.46 to 1.08 and precision at the deployed threshold fell from 0.80 to 0.54.Finished-only within-problem AUC was 0.61–0.76, and explained variance tracked within-problem AUC across checkpoints (r=0.98 offline).Source: §E.2 · pp. 18–19, Table 10 · p. 19

E.3 PREFIX SCAN

  • On prefixes without an answer yet, within-problem ranking reached 0.84–0.92 from 4,096 tokens onward, across a shrinking number of problems.At prefixes up to about 2,000 tokens, nearly all score variance was between problems; Table 11 describes the prefix-scan measures.Source: §E.3 · p. 19, Table 11 · p. 19

E.5 THE GENERATION-BUDGET AXIS

  • At a real 1,024-token cap, validation fell from 49.1% to 39.1% even as training solve rate rose from 5.5% to 19.3%; truncation-proxy measurements incorrectly suggested usability.Under real capping, only 0.7% finished, explained variance was −0.49, and precision was 0.21; Figure 11 plots training solve rate, truncation, and critic-score gap.Source: §E.5 · p. 20, Table 12 · p. 20, Figure 11 · p. 21
  • At a 4,096-token cap, the zero-label run averaged 47.6% macro validation, level with 47.2–47.4% for 5,120-token runs and above 46.6% for supervised PPO at the same cap.The run had 48% unfinished training rollouts; its median step took 395 seconds versus 490 seconds for same-cap supervised PPO, without the latter’s 134-second critic update.Source: §E.5 · pp. 20–21, Figure 11 · p. 21

F CALIBRATION ABLATIONS: TURNING THE CRITIC’S SCORE INTO A REWARD

The fully debiased continuous reward reached a higher peak than supervised PPO, but continuous-score runs did not retain their gains. Binarization constrains length-driven reward exploitation, while freezing the critic reduces RL-stage compute and memory costs.

F CALIBRATION ABLATIONS: TURNING THE CRITIC’S SCORE INTO A REWARD

  • The critic’s score mixes correctness and length: it still declines with response length among same-outcome responses, so Eq. (2) subtracts within-class offsets while preserving the between-class gap.After debiasing, some length dependence remains; the indicator caps what that remainder can earn.Source: F CALIBRATION ABLATIONS: TURNING THE… · p. 21
  • 29.2 on AIME 2026 pass@1 was achieved by the fully debiased arm, above supervised PPO’s 25.0; selection-corrected scores preserve that ordering.The fully debiased arm ended at 16.2, below the initial policy’s 21.2.Source: Figure 12 · p. 22, F CALIBRATION ABLATIONS: TURNING THE… · p. 23

F.2 CONTINUOUS-SCORE ARMS AND A GROUP-RELATIVE VARIANT

  • Across continuous-reward arms, response-length change rose with debiasing strength (r=0.983), but every arm peaked and none retained its gain.The sweep measured debiasing strengths of 0%, 16%, 34% and 56%; the strongest arm lengthened responses while the undebiased arm shortened them.Source: F CALIBRATION ABLATIONS: TURNING THE… · pp. 22–23

F.1 REMOVING THE CALIBRATION STEP BY STEP

  • Raw-score reward shortened responses by 73% and reduced held-out accuracy; a fixed threshold without debiasing also shortened responses, while using both calibration pieces held length and accuracy.The fixed-threshold arm shortened responses by 20% in 100 steps and accuracy fell 4 points; the raw-score run used an earlier actor–critic pair and an 8,192-token cap.Source: §F.1 · p. 23

F.2 CONTINUOUS-SCORE ARMS AND A GROUP-RELATIVE VARIANT

  • The group-relative 56% variant reached a similar AIME pass@1 peak of 28.3 sooner, but more than 90% of its rollouts were truncated from step 150 and it lost the shared reward-baseline identity.Its best macro average came at step 40 after 5.5 hours, versus step 110 after 16.3 hours for the GAE arm.Source: §F.2 · pp. 23–24

F.3 ANATOMY OF THE DECLINE

  • In the 56% arm, answer restatements rose from 1.8 to 68 and tracked held-out accuracy more strongly than length (r=−0.87 versus −0.51); the 16% arm stayed flat.Verbatim repetition remained at 1% of 60-character shingles, and scoring only the final boxed answer changed train-batch accuracy by at most 5.1 points with no trend.Source: §F.3 · p. 24, Table 14 · p. 24

G COMPUTATION AND MEMORY COST

  • Step time fell from 582 to 421 s (−28%) and 300-step cost from 388 to 281 GPU-hours, while peak reserved memory fell 9.4 GB per GPU.The accounting predicts a 37% step-compute reduction and 44% lower resident model state; these are RL-stage savings, while critic preparation required 901 GPU-hours plus validation.Source: G COMPUTATION AND MEMORY COST · pp. 25–26, Table 15 · p. 26

Related papers

Loading 2609.37119v1…