Taming Overconfidence in LLMs: Reward Calibration in RLHF

1 INTRODUCTION

• Figure 1 contrasts a vanilla reward model that favors high confidence on an incorrect answer with a calibrated model that gives higher reward to its low-confidence version.Source: Figure 1 · p. 2
• The study reports that RLHF-trained LLMs show greater verbalized overconfidence than pre-RLHF counterparts and that reward models favor high-confidence responses regardless of quality.The authors propose this reward-model bias as a possible source of poor calibration.Source: §1 · p. 2
• PPO-M incorporates explicit confidence scores into reward-model training, while PPO-C adjusts PPO rewards using an exponential average of past scores and verbalized confidence.Both methods are designed to integrate into RLHF without additional golden labels.Source: §1 · p. 2
• Across six datasets, PPO-M and PPO-C lower ECE while maintaining comparable or higher accuracy; on GSM8K, PPO-M on Llama3-8B reduces ECE by 6.44 points and increases accuracy by 2.73 points.The methods also preserve capabilities in open-ended conversational settings.Source: §1 · p. 2

2.1 RLHF-LLMS EXHIBIT OVERCONFIDENCE IN THEIR VERBALIZED CONFIDENCE

• On CommonsenseQA, PPO- and DPO-trained models show higher verbalized confidence than SFT counterparts, whose confidence scores are more broadly distributed.The analysis prompts four off-the-shelf models to report confidence from 0 to 10; Figure 2 encodes sample counts by confidence bin.Source: §2.1 · p. 3, Figure 2 · p. 3

2 EXPLORING SYSTEMATIC BIASES AND OVERCONFIDENCE IN RLHF-LLMS

• The section hypothesizes that reward-model preference for appended high-confidence scores may underlie RLHF-model overconfidence.Source: §2.2 · p. 3

2.2 REWARD MODELS ARE BIASED TOWARD HIGH CONFIDENCE SCORES

• On modified RewardBench comparisons, reward models increasingly prefer high-confidence rejected responses when confidence is reversed, and favor higher confidence for identical chosen or rejected answers.The four modes include answer-only, reversed confidence, and confidence variation on identical chosen or rejected responses; the evaluated models include an implicit reward model.Source: §2.2 · pp. 3–4, Figure 3 · p. 4

Background: Reward Modeling

• Standard reward modeling trains on pairwise human preferences, using a scalar reward to score each response and a ranking loss to favor chosen over rejected responses.The reward model typically starts from an SFT model.Source: §3 · p. 5

PPO-M: PPO with Calibrated Reward Modeling

• PPO-M adds confidence-query prompts and randomly assigned high or low confidence scores to existing pairwise preference data.The resulting dataset includes chosen and rejected responses at both confidence levels.Source: §3 · p. 5
• Its loss encourages reward models to favor high confidence for chosen responses and low confidence for rejected responses.The calibration data fine-tunes pre-existing reward models rather than training them from scratch; the calibrated model then supplies PPO rewards.Source: §3 · p. 5

PPO-C: PPO with Calibrated Reward Calculation

• PPO-C adjusts rewards during PPO without modifying the reward model, using confidence-query samples alongside ordinary prompts.Ordinary samples retain the original reward, while confidence-query samples receive a calibrated reward.Source: §3 · p. 5
• For confidence-query samples, PPO-C removes the verbalized score before calculating an unbiased reward and compares it with an exponential moving average baseline.The average uses α = 0.1; the adjustment depends on the reward’s deviation from that baseline and the rescaled confidence score, with missing confidence defaulting to 0.5.Source: §3 · pp. 5–6

4 EXPERIMENTS

• Evaluation covers Llama3-8B and Mistral-7B, Direct Answers and zero-shot CoT, and ECE, AUC, and accuracy across six datasets.The datasets span open-ended generation and multiple-choice questions.Source: §4 · p. 6, §4.1 · p. 6

4.1 EXPERIMENTAL SETUP

• The Llama3-8B reward-model checkpoint is available off the shelf, while the Mistral-7B reward model is trained on preference data; calibration data mixes chosen and rejected responses with high and low confidence.Source: §4.1 · p. 6
• RLHF uses 20,480 prompts, with confidence-query prompts added to 25% of single-turn prompts; comparisons include SFT, vanilla PPO, and PPO†.PPO† uses the modified prompt dataset but keeps the vanilla reward model.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

• Across both model families and prompting strategies, PPO-M and PPO-C achieve lower ECE and higher AUC than baselines while maintaining comparable or higher accuracy.The comparison includes vanilla PPO and PPO†, which uses confidence-query prompts but retains the vanilla reward model.Source: §4.2 · p. 7
• On rejected responses with high or low confidence, the pre-calibrated reward model favors high confidence, while the calibrated model favors low confidence.Figure 6 also compares preference distributions for chosen-with-confidence, confidence-reversed, and answer-only modes.Source: §4.2 · p. 7, Figure 6 · p. 7

5 ANALYSIS

• The analysis examines instruction-following, conversational settings, and extension of the calibration approach to DPO models.Source: §5 · p. 7

5.1 INSTRUCTION-FOLLOWING CAPABILITIES

• PPO-M and PPO-C achieve MT-Bench and Arena-Hard scores comparable to or slightly higher than standard PPO, preserving instruction-following abilities.Table 2 summarizes average scores on the two benchmarks.Source: §5.1 · p. 7, Table 2 · p. 9

5.2 EXTENSION TO DPO

• CDPO extends the CRM loss to DPO training, treating DPO models as implicit reward models.The first term preserves the original DPO objective to prevent forgetting; the experiment fine-tunes a Mistral-7B DPO model on the reward-model calibration dataset.Source: §5.2 · p. 9
• CDPO lowers ECE and raises AUC across six datasets, reducing ECE by over 50% on TruthfulQA, CommonsenseQA, and Professional Knowledge while retaining performance comparable to the original DPO checkpoint.For Mistral-7B, additional training data improves MT-Bench and Arena-Hard scores, and CDPO further amplifies these gains over DPO†.Source: §5.2 · p. 9

• Prior calibration evaluations often use logits, which may be inaccessible in proprietary models and unclear to human users.Source: §6 · p. 9
• Verbalized-confidence research prompts models to output confidence scores; prior approaches include supervised fine-tuning and prompting strategies for instruction-tuned and RLHF-trained models.Source: §6 · p. 10
• RLHF commonly combines supervised fine-tuning, reward modeling, and PPO, while DPO removes the separate reward model; the role of reward models in confidence calibration has received limited study.Source: §6 · p. 10

7 CONCLUSION

• Reward models that favor high-confidence responses regardless of quality are identified as a systematic bias; PPO-M calibrates reward modeling and PPO-C adjusts reward scores during PPO.The conclusion reports reduced expected calibration error while maintaining accuracy and robust open-ended instruction-following.Source: §7 · p. 10

A RELATED WORKS

• Prior calibration work includes scaling and binning, while verbalized-confidence studies address confidence scores through supervised fine-tuning or prompting strategies.Source: A RELATED WORKS · p. 16
• SFT-based calibration methods may require ground-truth accuracy labels, limiting applicability to open-ended generation, whereas this approach is described as preserving generalization in that setting.Source: A RELATED WORKS · p. 16
• Although DPO eliminates a separate reward model, limited research has examined reward models' role in calibration; the paper identifies confidence-score bias and proposes calibration methods.Source: A RELATED WORKS · p. 16

B.1 LIMITATION

• Directly applying CRM loss to DPO is not explicitly optimized for DPO, and some performance degradation motivates future hyperparameter tuning or a more specifically designed dataset.Source: §B.1 · p. 17

B.2 BROADER IMPACT

• The paper states that improved calibration may enhance reliability, trustworthiness, and safety in general AI systems.Source: §B.2 · p. 17

C DATASETS

• The dataset section describes data used for preliminary experiments, reward modeling, reward-model calibration, and PPO training.Source: C DATASETS · p. 17

C.1 PRELIMINARY EXPERIMENTS DATASET

• To test reward-model bias toward high-confidence responses, the authors modified RewardBench by prepending a confidence-query system prompt and appending random scores to original responses, assigning 0–3 low and 7–10 high.Figure 7 shows a rejected response with appended low confidence scores.Source: §C.1 · p. 17, Figure 7 · p. 17

C.2 REWARD MODEL TRAINING DATASETS

• Mistral-7B’s reward model is trained from scratch on the open-source Skywork-Reward-Preference-80K-v0.1 pairwise binary dataset.Source: §C.2 · p. 18

C.3 REWARD MODEL CALIBRATION DATASETS.

• Calibration samples are filtered from multiple open-source datasets, excluding long or multi-turn examples and ambiguous pairs with similar solutions, while targeting about 2,500 samples per dataset.Datasets without chosen and rejected scores contribute randomly selected samples; Table 5 lists datasets and thresholds.Source: §C.3 · p. 18, Table 5 · p. 18

C.4 PPO DATASETS

• PPO training randomly selects 20,480 prompts and adds a confidence-query system prompt to 25% of single-turn prompts.Prompts longer than 8,192 tokens are filtered to avoid truncation; Figure 8 gives an example.Source: §C.4 · p. 18, Figure 8 · p. 19

C.5 EVALUATION DATASETS.

• Evaluation covers six categories across open-ended and multiple-choice tasks, including arithmetic, commonsense, symbolic and truthful reasoning, and professional knowledge.The datasets include GSM8K, CommonsenseQA, TruthfulQA, SciQ, Object Counting, and Professional Knowledge in MMLU.Source: §C.5 · p. 19
• Professional Knowledge evaluation combines MMLU test sets for accounting, law, medicine, and teaching.Source: §C.5 · p. 19

D IMPLEMENTATION DETAILS

• The section introduces implementation details for the experiments.Source: D IMPLEMENTATION DETAILS · p. 19

D.1 REWARD MODEL TRAINING

• Llama3-8B uses an off-the-shelf reward-model checkpoint, while Mistral-7B’s reward model is trained from scratch; calibration assumes reward models are already trained and generally perform well.PPO-M uses calibrated reward models, while PPO-C adjusts reward calculation during PPO; standard PPO uses the EOS-token reward.Source: §D.1 · p. 20, §D.2 · p. 20, §D.3 · pp. 20–21

D.3 PPO TRAINING

• The policy models start from supervised fine-tuning checkpoints, and PPO, PPO-M, and PPO-C share hyperparameters for each model.The appendix also lists reward-model and PPO hyperparameters in Tables 6–8.Source: §D.3 · p. 20, Table 6 · p. 20, §D.3.1 · p. 21

D.4 DPO TRAINING

• The appendix extends calibrated reward modeling to DPO using pretrained DPO checkpoints, with the same hyperparameters for DPO and CDPO.The scaling coefficient w is not used in DPO.Source: §D.4 · p. 21, §D.4.1 · p. 21, Table 9 · p. 22

E.1 OVERCONFIDENCE IN RLHF-LLMS

• Additional results across five datasets show RLHF-trained models consistently exhibit verbalized overconfidence.Figures 12–16 compare confidence distributions before and after RLHF.Source: §E.1 · p. 24, Figure 12 · p. 24, Figure 13 · p. 24

E.2 REWARD MODELS ARE BIASED TOWARD HIGH CONFIDENCE SCORES

• Reward models show a biased preference for high-confidence responses even without confidence-query prompts, though the bias is more subtle in that setting.With prompts, all tested reward models favor high-confidence responses, with varying degrees of bias.Source: Figure 17 · p. 26, Figure 18 · p. 27, Figure 19 · p. 28

E.3 CALIBRATED REWARD MODELS

• Both calibrated reward models are less certain on high- versus low-confidence chosen responses and prefer low-confidence rejected responses.The comparison covers Llama3-8B and Mistral-7B; Figures 20 and 21 show preference distributions.Source: §E.3 · p. 29, Figure 20 · p. 29, Figure 21 · p. 29

E.4 VISUALIZATION OF THE CONFIDENCE DISTRIBUTION

• PPO-M shifts confidence scores somewhat lower than PPO, while CDPO’s distribution is more dispersed than DPO’s, with more lower-confidence scores.Figure 22 compares their confidence distributions.Source: §E.4 · p. 30, Figure 22 · p. 30

E.5 MODEL LOGITS FOR CONFIDENCE SCORES

• On TruthfulQA, PPO’s confidence-position logits favor 10, whereas PPO-M’s favor 9, indicating non-uniform preferences among confidence numbers.The analysis compares log probabilities for numbers 0–10 at the confidence-score position.Source: §E.5 · p. 30, Figure 23 · p. 30

E.6 PARAMETER SENSITIVITY

• Increasing PPO-C’s w improves calibration but slightly reduces MT-Bench and Arena-Hard scores; the authors choose w = 2.0 for the main results.The reward adjustment factor is γ = w * (r̂_i − Δr_t) * (s_i − 0.5).Source: §E.6 · p. 30, Table 10 · p. 31
• For PPO-C, α = 1.0 worsens calibration overall, while α = 0.0 performs worse than α = 0.1 on calibration and MT-Bench; increasing confidence-query prompts also lowers MT-Bench scores without a consistent calibration trend.The main settings use α = 0.1 and confidence queries on 25% of single-turn prompts.Source: Table 11 · p. 31, Table 12 · p. 31, Table 13 · p. 31

E.8 IMPACT OF COMBINING EQ. 1 AND 2

• When reward models were fine-tuned on the calibration dataset, the combined Eq. 1 and Eq. 2 loss did not yield better PPO-M calibration than Eq. 2 alone.The Eq. 2-only model had similar ability to distinguish chosen from rejected responses as the combined-loss model.Source: §E.8 · p. 32

E.9 COMPARING THRESHOLD-BASED VS. REWARD-AVERAGE DIFFERENCE APPROACHES

• Threshold-based PPO-C performs promisingly across six datasets and shows a similar calibration–instruction-following trade-off as w increases.It uses the reward exponential average as a threshold and absolute reward as the adjustment scaling factor; Table 15 compares it with Difference.Source: §E.9 · pp. 32–33, Table 15 · p. 33

E.10 CAN PPO-M AND PPO-C BE COMBINED?

• Combining PPO-M and PPO-C does not outperform either method individually and sometimes reduces performance.The authors hypothesize that removing confidence scores when adjusting rewards may be inappropriate for reward models trained on confidence-bearing responses.Source: §E.10 · pp. 33–34, Table 16 · p. 33

E.11 EXTENSION TO DPO

• For Llama3-8B, CDPO reduces ECE and increases AUC while maintaining MT-Bench performance, but DPO† and CDPO show slight Arena-Hard degradation.The authors cite insufficient tuning or calibration-dataset limitations as possible explanations.Source: §E.11 · p. 34, Table 18 · p. 34, Table 17 · p. 34