PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenJev-RLCD: A Working RLCD Implementation

Zhimin Gao, Pichao Wang

arXiv:2609.38850v1cs.AIcs.CL

TL;DR

Calibrated confidence is useful for deciding which model outputs can be trusted, but open-source approaches use supervised fine-tuning with post-hoc temperature scaling and correctness-only reinforcement can yield overconfidence. RLCD scores the answer distribution after a sampled rationale, first calibrating it and then reinforcing reasoning. Across two reasoning tasks it matches or beats temperature-scaled baselines on accuracy and Brier score and outperforms them in selective prediction.

  • Problem

    Open-source reproductions use supervised fine-tuning and temperature scaling, while correctness-only reinforcement is not a proper score and can produce confidently wrong reasoning models.

    Source: §1 · p. 1

  • Method

    RLCD applies a proper score to the answer distribution after a sampled rationale, first calibrating the decision and then reinforcing reasoning from the calibrated model.

    Source: §2 · p. 2, §3.3 · p. 4

  • Results

    Across MMLU-Pro and GSM8K answer verification, RLCD matches or beats temperature-scaled SFT, RFT/STaR, and GRPO in accuracy and Brier score, and beats them in risk–coverage area.

    Source: §1 · p. 2

  • Takeaways & Limitations

    At the same accuracy as GRPO, a single RLCD query can identify which answers to trust for delegated decisions.

    Source: §8 · p. 9

    Experiments use a 1.7B model and 400–800 update steps; larger models or longer training may change the balance between the two stages.

    Source: §7 · p. 9

Abstract

from arXiv · show

Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working implementation of reinforcement learning for calibrated decisions (RLCD) for reasoning models: the model samples a rationale, and we score the answer distribution it commits to afterwards with a strictly proper scoring rule. A variance identity shows that scoring the mixture of several samples rewards disagreeing rationales, and that RLVR is exactly this mixture objective without its diversity term. Optimized naively, the per-rationale objective either switches reasoning off or is drowned out by policy-gradient noise, which leads to a two-stage recipe: calibrate, then reinforce. With Qwen3-1.7B on two reasoning tasks (3 seeds, paired tests), RLCD matches or beats SFT, RFT/STaR and GRPO (each temperature-scaled) in accuracy and beats all of them in selective prediction; on GSM8K answer verification a single query decides \gvTwoCovFive\% of the items at $\le$5\% error, versus \gvGrpoCovFive\% for GRPO. When uncertainty comes from annotator disagreement, RLCD provably cannot beat cross-entropy. Code and results: https://github.com/ZimmyGao/openjev-rlcd.

1 Introduction

Calibrated confidence lets downstream systems automate decisions the model is sure about and escalate the rest, but existing open-source reproductions rely on supervised fine-tuning and post-hoc temperature scaling. The paper asks how to implement RLCD for reasoning models and outlines its analysis, training recipe, and controlled study.

1 Introduction

  • Calibrated probabilities let downstream agents automate confident decisions and escalate the rest using the model’s own confidence, without labels.Source: §1 · p. 1
  • Open-source reproductions combine supervised fine-tuning with temperature scaling, while correctness-only reinforcement rewards are not proper scores and can produce confidently wrong reasoning models.Source: §1 · p. 1
  • The paper introduces a readout formulation and variance identity: scoring a sample mixture rewards rationale disagreement, whereas scoring each rationale removes that bonus and has a calibrated optimum.Source: §1 · p. 2
  • The authors propose calibrating the readout before reinforcing reasoning, after naive end-to-end optimization either switches reasoning off or lets policy-gradient noise dominate.Source: §1 · p. 2
  • Across MMLU-Pro and GSM8K answer verification, RLCD matches or beats temperature-scaled SFT, RFT/STaR, and GRPO in accuracy and Brier score, and beats them in risk–coverage area.When uncertainty reflects annotator disagreement, reasoning cannot reduce it and RLCD cannot beat cross-entropy on the same labels.Source: §1 · p. 2

2 Setting

The setting treats a rationale as a sampled latent path and the model’s answer distribution after that rationale as its decision. A proper score evaluates that distribution against the outcome distribution, with the Brier score uniquely maximized at the true distribution.

2 Setting

  • An outcome distribution q is one-hot when an answer is objectively right and spread when annotators disagree; the model samples rationale r for question x.Source: §2 · p. 2
  • After a rationale, the model’s next-token distribution over K answer options is its decision u; averaging over rationales gives mixture p, while one query returns one u.Source: §2 · p. 2
  • The shifted, negated Brier score has expected value uniquely maximized when the reported distribution u equals the true outcome distribution q.Source: §2 · p. 2

3 Method

The method distinguishes scoring each rationale’s answer distribution from scoring their mixture: the latter rewards disagreement, while the former has a calibrated per-rationale optimum. Because naive optimization can eliminate reasoning or drown out the readout signal, RLCD separates calibration from reasoning reinforcement.

3.1 What to score: the mixture or each rationale

  • Scoring the mixture equals the average per-rationale score plus variance across rationale readouts, so it rewards disagreement even when that disagreement is ungrounded.Source: §3.1 · p. 3
  • For sampled one-hot answers, RLVR is the mixture objective without its Gini-diversity bonus; reducing the bonus coefficient below one changes the calibrated optimum.Source: §3.1 · p. 3
  • When outcomes are independent of rationales given the question, every per-rationale readout has the unique optimum u(x, r) = q(x) for λ < 1.Source: §3.1 · p. 3
  • The per-rationale objective removes the reward for disagreement and makes a single query calibrated at the optimum, while useful rationales can still help a finite-capacity model compute q.Source: §3.1 · p. 3
  • With M rationales and one observed outcome, the gradient estimator separates readout pathwise and rationale score-function terms; the estimator is unbiased, including with a leave-one-out baseline.Source: §3.1 · p. 3

3.2 Two ways the per-rationale objective fails

  • Without a KL anchor, end-to-end per-rationale training can make rationales empty and identical within about 100 steps; a KL anchor prevents collapse and preserves reasoning.Source: §3.2 · p. 3, Figure 1 · p. 4
  • Adding the score-function term from scratch worsens metrics as its weight rises; its gradient norm is 7–12× the pathwise term’s and dominates the optimizer.Source: §3.2 · p. 4

3.3 The recipe: calibrate, then reinforce

  • RLCD first trains the reported decision with a proper score using the pathwise term, then reinforces reasoning with that score, a down-weighted score-function term, and a KL anchor at the calibrated model.Source: §3.3 · p. 4

4 Experimental setup

The experiments compare calibrated decision methods across knowledge-heavy, reasoning-based verification, and human-label-distribution tasks, using matched model training and temperature-scaled baselines.

4 Experimental setup

  • Evaluation spans knowledge-heavy 10-option MMLU-Pro, GSM8K answer verification, and ChaosNLI human-label-distribution prediction.GSM8K-Verify asks whether a proposed final answer is correct; ChaosNLI uses 100 annotator labels per MNLI item.Source: §4 · p. 5
  • GSM8K-Verify uses 4,000 training and 300 development examples, with 1,260 test problems having parsable proposals.Proposals are correct on 53% of training items and 50% of test items.Source: §4 · p. 5
  • All methods start from Qwen3-1.7B with matched question sequences; RLCD uses two 400-step stages and baselines receive dev-fitted temperature scaling.Methods use AdamW at 2·10^-6, eight questions and four rationales per step; baselines include SFT, RFT/STaR, and GRPO.Source: §4 · p. 5
  • Metrics include single-query accuracy, Brier score, NLL, ECE, AURC, Cov@ϵ, and fitted temperature; results average three seeds with paired tests over test items.The analysis reports 95% bootstrap intervals and two-sided sign-flip permutation p-values.Source: §4 · p. 5

5 Results

Two-stage RLCD improves selective decisions over temperature-scaled baselines, with stage-2 reinforcement helping when reasoning changes answers. Under annotator disagreement, RLCD cannot outperform supervised training on the same labels; ablations show why calibration and reinforcement must be separated.

5.1 Main comparison

  • At a 5% error budget, RLCD decides 81% of GSM8K-Verify items versus GRPO’s 19%, despite similar accuracy after temperature scaling.RLCD improves on SFT+TS by +21.8 points in accuracy and −0.247 Brier; GRPO and RFT require fitted temperatures near 7, versus 1.4 for RLCD.Source: §5.1 · pp. 5–6, Figure 2 · p. 6
  • On MMLU-Pro, RLCD improves accuracy by +5.3 points over SFT+TS and by +5.2 over GRPO+TS, and decides 28% of items at a 20% error budget.SFT+TS decides 16% at that budget; no reasoning baseline does, and stage 1 alone carries most of RLCD’s gain on both tasks.Source: §5.1 · p. 6

5.2 What reinforcing the reasoning adds

  • On GSM8K-Verify, RLCD-RL gains +3.5 accuracy points and −0.057 Brier over a readout-only continuation; the anchor-only control is worse and unstable.The same 400-step budget and stage-1 checkpoint were used for the continuations; the anchor-only control lacks outcome reward.Source: §5.2 · pp. 6–7, Figure 3 · p. 7

5.3 Aleatoric uncertainty: when RL cannot help

  • When annotators disagree, RLCD cannot beat supervised training on the same labels; per-rationale training switches reasoning off and matches direct-answer cross-entropy quality.The reported JSD is 0.075 for the per-rationale objective versus 0.074 for cross-entropy on direct answers.Source: §5.3 · pp. 7–8

5.4 Ablations

  • Every positive score-function weight hurts when stages are not separated, and readout warm-up alone ties readout-only training rather than reproducing the two-stage gains.The gains require a converged readout before reinforcement.Source: §5.4 · p. 8
  • Scoring rationales with a frozen calibrated readout worsens GSM8K-Verify Brier by +0.021 and AURC by +0.030 in seed 17.Reasoning shortened and the policy’s own readout became overconfident.Source: §5.4 · p. 8
  • Without a stage-2 KL anchor, reasoning degenerates after about 330 steps because the readout-based reward does not constrain rationale format.The model stops writing the marker and repeats its conclusion to the length limit.Source: §5.4 · p. 8
  • For direct scorers, explicit-probability RLCD never beats cross-entropy on CLINC150 or ChaosNLI, while RLVR collapses to the mode.On CLINC150, RLCD reaches 94.6% accuracy versus 96.3% for cross-entropy.Source: §5.4 · p. 8

6 Related work

The paper situates its contribution among calibration methods, reinforcement-learning approaches to reasoning, and latent-rationale training. It distinguishes its approach by scoring the model’s full answer distribution after reasoning.

Calibration of language models

  • Temperature scaling and verbalized or sampled confidence calibrate confidence levels but not their ranking, which selective prediction evaluates.The paper locates its difference from these methods in selective prediction results.Source: §6 · p. 8

RL for reasoning and calibration

  • RLVR and GRPO optimize correct sampled answers, while this paper scores the answer distribution the model assigns; concurrent work scores verbalized confidence with the Brier score [5].The paper also analyzes group rewards and identifies RLVR as the mixture objective without its diversity term [15].Source: §6 · p. 9

Rationales as latent variables

  • Stage 1 resembles latent-rationale training methods TRICE and LaTRO [3] [13] and STaR [20], but uses a proper score of the full distribution; the RFT/STaR baseline inherits RLVR’s overconfidence.Source: §6 · p. 9

7 Limitations

The evaluation is limited to a 1.7B model, 400–800 update steps, and three seeds; the authors note that larger models or longer training may change the balance between stages.

  • All experiments use a 1.7B model, 400–800 update steps, and three seeds; larger models or longer training may change the balance between stages.Source: §7 · p. 9
  • GSM8K-Verify is constructed from GSM8K using a 0.6B proposer.Source: §7 · p. 9
  • The Jev comparison is behavioral because its training data are not public, and the paper makes no claim about Jev’s internal algorithm.Source: §7 · p. 9
  • Stage 2 helps on the reasoning-essential task and is neutral on the knowledge-heavy task, but the authors do not know how to predict its benefit without running it.Source: §7 · p. 9

8 Conclusion

The conclusion presents RLCD as scoring the answer distribution after a rationale with a proper score, first to calibrate the decision and then to reinforce its reasoning. It argues that a single query can identify which answers to trust at the same accuracy as GRPO.

  • RLCD scores the answer distribution after the model’s own rationale with a proper score, calibrating the decision before reinforcing the reasoning that feeds it.Source: §8 · p. 9
  • Scoring groups of samples rewards disagreement, while a correctness reward undoes calibration; the paper relates both effects through one identity.Source: §8 · p. 9
  • At the same accuracy as GRPO, a single RLCD query identifies which of its answers to trust.Source: §8 · p. 9

A Estimators and their exact checks

The appendix describes an estimator combining readout and rationale gradients, with leave-one-out baselines, and checks its variants against exact gradients by enumeration.

A Estimators and their exact checks

  • The per-rationale objective allows shaping terms that depend only on the rationale and outcome.The supplied example is a term of the form si = −β…, whose remainder is not specified in the passage.Source: A Estimators and their exact checks · p. 11
  • The estimator’s gradient combines a pathwise derivative of the mixture objective with score-function terms for individual rationales, each using a baseline that does not depend on that rationale.The baselines use only other, independent rationales.Source: A Estimators and their exact checks · p. 11
  • Exact enumeration across tabular models found maximum absolute gradient differences below 10^-15 for mixture, per-rationale, convex-combination, and shaped estimators.The enumeration covers models with at most four rationale types and four answers, using three or four rationales.Source: A Estimators and their exact checks · p. 11

B Hyperparameters

The implementation uses Qwen3-1.7B and a two-stage training recipe, then evaluates with four rationales per question and temperature scaling.

B Hyperparameters

  • Training uses 400 stage-1 steps with per-rationale Brier or log scoring, followed by 400 stage-2 steps from the stage-1 model.Stage 2 uses score-function weight c = 0.3 and KL β = 0.04 to the stage-1 model.Source: B Hyperparameters · p. 11
  • The model is Qwen3-1.7B with rationales sampled at temperature 1; batches contain 8 questions and 4 rationales per question.Rationale length is capped at 256 tokens for MMLU-Pro and 320 for GSM8K-Verify.Source: B Hyperparameters · p. 11
  • Evaluation uses four rationales per test question, with the first rationale as the single-query result, and temperature scaling selected by dev NLL.The temperature search grid is T ∈ [0.1, 50].Source: B Hyperparameters · p. 11

C Mixture readout

The table reports accuracy, Brier score, AURC, and temperature for the mean answer distribution over four rationales; averaging improves every reasoning method without changing the ordering.

C Mixture readout

  • RLCD two-stage records Acc 94.5±0.3 and 48.9±2.0, and Brier 0.092±0.005 and 0.651±0.014, across the table’s two metric blocks.The table reports the mean answer distribution over four rationales.Source: C Mixture readout · p. 12
  • In the first metric block, RLCD two-stage has Acc 94.5±0.3 versus 93.8±0.3 for GRPO + KL + TS, and Brier 0.092±0.005 versus 0.114±0.007.The second metric block reports RLCD two-stage Acc 48.9±2.0 and Brier 0.651±0.014.Source: C Mixture readout · p. 12
  • Averaging the readout over four rationales improves every reasoning method, while leaving Table 1’s ordering unchanged.Source: C Mixture readout · p. 12
Loading 2609.38850v1…