Source-linked AI summary
CalibraRL: Confidence-Calibrated Reinforcement Learning for Mitigating Hallucinations in RAG
Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, Zhiming Zheng
TL;DR
Fixed outcome-based rewards do not account for confidence differences among errors or query-level differences in answering evidence. CalibraRL incorporates both into reinforcement-learning rewards, improving answer precision and reducing hallucinations while lowering ECE across datasets and backbones.
Problem
Fixed outcome-based rewards do not distinguish incorrect responses by confidence or account for differences in answering reliability across queries.
Source: §1 · pp. 1–2Method
CalibraRL combines a Brier-based confidence-aware answer reward with a Bayesian posterior-adaptive abstention reward derived from same-query rollout evidence.
Source: §1 · p. 2Results
CalibraRL improves answer precision and reduces hallucinations compared with strong baselines, and reduces mean expected calibration error (ECE) by 30.8% across two backbones and three datasets.
Source: §1 · p. 2Takeaways & Limitations
The results highlight CalibraRL's potential to improve retrieval-augmented generation reliability in web search.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Large language models (LLMs) often produce plausible but incorrect answers, motivating reinforcement learning methods that reward correct answers, penalize incorrect answers, and allow abstention. However, fixed outcome-based rewards do not distinguish incorrect responses with different confidence levels or account for differences in answering reliability across queries. Consequently, such rewards may inadequately penalize overconfident errors and fail to discourage unnecessary abstention. We propose CalibraRL, a reinforcement learning framework that incorporates response confidence and query-level answering evidence into reward design. For non-abstaining responses, CalibraRL combines correctness-based utility with a Brier-based penalty that encourages agreement between confidence and correctness. For abstentions, it estimates conditional answering reliability using a Beta posterior derived from the correctness of non-abstaining responses within each sampled group. The abstention reward increases with the posterior probability that the model's accuracy on non-abstaining responses is below a predefined threshold. CalibraRL reuses training rollouts without requiring additional annotations indicating whether the model knows the answer to each question. Experiments on three QA datasets demonstrate that CalibraRL improves answer precision and reduces hallucinations compared with strong baselines. Further analysis shows that CalibraRL reduces mean expected calibration error (ECE) by 30.8\% across two backbone models and three datasets, highlighting its potential to improve the reliability of retrieval-augmented generation (RAG) in web search.
1 Introduction
Fixed outcome-based rewards overlook confidence differences among errors and query-level differences in answering evidence. CalibraRL incorporates both into rewards and improves answer precision and reduces hallucinations across three QA datasets.
1 Introduction
- Fixed ternary rewards depend only on outcome category, although incorrect answers can differ in confidence and abstentions can differ in same-query answering evidence.A high-confidence error has a larger confidence–correctness discrepancy, while mostly correct non-abstaining responses may indicate a missed opportunity to answer.Source: §1 · pp. 1–2
- Preliminary experiments find that incorrect responses span a broad confidence range and that non-abstaining accuracy varies across queries that elicit abstention.These observations motivate using confidence and answering evidence as training signals beyond outcome labels.Source: §1 · p. 2
- CalibraRL combines a confidence-aware answer reward with a Bayesian posterior-adaptive abstention reward based on same-query rollout evidence.It reuses training rollouts and reference answers without requiring precomputed labels indicating whether each query is known or unknown to the model.Source: §1 · p. 2
- CalibraRL improves answer precision and reduces hallucinations compared with strong baselines, while reducing mean expected calibration error (ECE) by 30.8% across two backbones and three datasets.Source: §1 · p. 2
2 Related Work
Related work addresses hallucinations through retrieval, verification, decoding, factuality training, and abstention. Knowledge-boundary research distinguishes answer correctness from knowing an answer and highlights challenges in reliable self-assessment.
2.1 LLM Hallucinations
- Hallucination mitigation includes retrieval augmentation, verification, contrastive decoding, factuality-oriented training, and methods that enable abstention.TruthRL uses GRPO with positive rewards for correct answers, negative rewards for hallucinations, and zero rewards for abstentions.Source: §2.1 · p. 2
2.2 Knowledge Boundary of LLMs
- Knowledge-boundary work distinguishes response correctness probability from the probability of knowing an answer, while noting challenges in confidence estimation and expressing possessed knowledge.The cited research also examines probing internal knowledge states and the slower learning of previously unknown facts during fine-tuning.Source: §2.2 · p. 2
- A single incorrect response does not necessarily mean knowledge is absent, so abstention decisions must avoid withholding answers the model could provide correctly.Source: §2.2 · p. 2
3 Preliminaries
The paper formulates question answering with abstention and motivates reward signals that reflect both confidence–correctness alignment and query-specific answering evidence.
3.1 Problem Formulation
- The task gives a model a question with associated documents and acceptable reference answers, and prompts it to answer or abstain.The objective encourages correct answers and appropriate abstention while discouraging incorrect answers and unnecessary abstention.Source: §3.1 · p. 3
- Mean generated-token probability is used as a response-confidence proxy without additional prompting or multiple response samples.Source: §3.1 · p. 3
3.2 Motivation
- Accuracy generally rises with confidence but remains below ideal calibration, with ECE 0.318; incorrect responses include many above 0.8 confidence.The findings motivate confidence–correctness alignment because correctness-only rewards treat errors equally despite their differing confidence.Source: §3.2 · p. 3
- Among queries eliciting IDK, reference accuracy varies substantially, with some queries exceeding 0.5 and others approaching 1.All-IDK reference groups are excluded because conditional accuracy is undefined; this variation motivates query-level answering evidence for abstention rewards.Source: §3.2 · p. 3
- The reward-design principles are to penalize confidence–correctness discrepancies and adapt abstention rewards to query-level answering performance.Source: §3.2 · p. 3
4 Methodology
CalibraRL samples response groups per query, scores answers using correctness and confidence, and scores abstentions using a Bayesian reliability estimate from answered responses.
4 Methodology
- For each query, the method samples responses, evaluates answered responses, and derives abstention rewards from a Bayesian reliability estimate based on group answers.Rewards are normalized within each group for GRPO optimization.Source: §4 · p. 3
4.1 Confidence-Aware Answer Reward
- The confidence-aware answer reward addresses the limitation of correctness-only scoring, which treats responses with the same correctness label alike despite different confidence levels.Source: §4.1 · p. 3
C. Posterior-adaptive Abstention Branch
CalibraRL adapts abstention rewards to query-level answering reliability using correctness evidence from sampled non-abstaining responses. These rewards join confidence-aware answer rewards in group-relative policy optimization, while inference requires neither reliability estimation nor reward computation.
C. Posterior-adaptive Abstention Branch
- Incorrect-answer rewards include a confidence penalty that distinguishes responses otherwise receiving the same correctness-based reward; confidence scores are fixed during policy-gradient computation.Lowering confidence cannot make an incorrect-answer reward positive.Source: C. Posterior-adaptive Abstention Branch · p. 4
4.2 Posterior-Adaptive Abstention Reward
- Abstention rewards rise with the posterior probability that answering reliability falls below a threshold, using correctness outcomes from non-abstaining responses in each sampled group.Abstentions are excluded because their correctness cannot be evaluated; all abstentions in a group share the resulting reward.Source: §4.2 · p. 4
- With a uniform prior and threshold 0.5, an all-abstention group receives zero reward and yields no reward-driven learning signal.With no non-abstaining responses, the posterior remains equal to the prior.Source: §4.2 · p. 4
4.3 Confidence-Calibrated Policy Optimization
- GRPO uses within-group relative rewards to optimize the policy, with rewards and advantages held fixed during updates and a KL term regularizing deviation from a reference policy.The confidence penalty and adaptive abstention reward affect policy updates through group-relative advantages.Source: §4.3 · pp. 4–5
4.4 Inference
- At inference, the model generates an answer or abstention from the question and associated documents without an explicit confidence threshold, ground-truth answers, reliability estimation, or reward computation.The prompt instructs it to answer from the documents only when confident its answer is correct; otherwise, it should output “I don’t know”.Source: §4.4 · p. 5
5 Experiments
CalibraRL achieves the highest precision in all six model–dataset settings and the highest truthfulness in five, with particularly strong results on HotpotQA. Ablations and analyses examine its reward components, initialization, calibration, transfer, hyperparameters, and training-data size.
5.1 Experimental Setup
- The evaluation covers open-domain NQ and TriviaQA, multi-hop HotpotQA, four reliability and calibration metrics, seven baseline approaches, and two instruction-tuned backbones under shared retrieved evidence and protocols.Precision, Truthfulness, and Effective Reliability favor higher values; ECE favors lower values.Source: §5.1.1 · p. 5, §5.1.2 · p. 5, §5.1.3 · p. 5
5.2 Main Results
- CalibraRL achieves the highest precision in all six model–dataset settings and the highest truthfulness in five; on HotpotQA it outperforms all baselines on all three metrics with both backbones.Precision and ER indicate higher accuracy among answered questions and a better balance between correct and incorrect answers across the evaluation set.Source: §5.2 · p. 6, Table 1 · p. 6
5.3.1 Ablation Study.
- Removing either confidence shaping or adaptive abstention reduces average precision, truthfulness, and ER on both backbones, although some ablations yield slightly higher ER on individual datasets.The full method has the highest precision and truthfulness in all six model–dataset settings.Source: §5.3.1 · pp. 6–7, Table 2 · p. 6
5.3.2 Effect of SFT Initialization.
- Direct CalibraRL improves average precision, truthfulness, and ER over SFT + CalibraRL by 8.58, 11.03, and 4.93 percentage points on LLaMA3, and 3.68, 5.26, and 3.06 points on Qwen2.5.These results favor applying CalibraRL directly to the Instruct backbones under the reported setup.Source: §5.3.2 · p. 7, Table 3 · p. 7
5.3.3 Confidence Calibration Analysis.
- 30.8% overall mean ECE reduction across both backbones and all three benchmarks indicates improved confidence calibration on answered examples.Average ECE decreases by 24.9% for LLaMA3-8B-Instruct and 35.1% for Qwen2.5-7B-Instruct.Source: §5.3.3 · p. 7, Table 4 · p. 7
5.3.4 Cross-Dataset Generalization.
- Cross-dataset transfer is source–target dependent: models trained on NQ and HotpotQA achieve positive ER on both out-of-domain datasets, while TriviaQA training yields ER 9.70 on NQ and −1.38 on HotpotQA.Figure 4 organizes results by source dataset in rows and target dataset in columns.Source: §5.3.4 · pp. 7–8, Figure 4 · p. 7
5.3.5–5.3.6 Sensitivity Analyses
- The effects of reliability threshold and confidence-penalty weight vary by dataset and metric, with no tested value best in every setting.Higher thresholds raise NQ precision and truthfulness, while confidence weight 0.5 peaks on those metrics for NQ and weight 1 peaks on TriviaQA; ER is relatively stable in several settings.Source: §5.3.5 · p. 8, §5.3.6 · p. 8, Figure 5 · p. 8
5.3.7 Effect of Training Data Size.
- On NQ, metrics generally improve as training data grows to 10k examples; beyond 10,000, ER rises while precision and truthfulness fall.At 10k, precision is 68.65%, truthfulness 87.89%, and ER 14.40%; the main experiments use this size.Source: §5.3.7 · p. 8, Figure 7 · p. 8
6 Conclusion
The conclusion presents CalibraRL as a response to fixed rewards that overlook confidence and query-level answerability, reporting improved precision, truthfulness, and calibration. The algorithm derives response rewards from sampled group outcomes and then updates the policy with fixed rewards and advantages.
6 Conclusion
- Fixed rewards overlook that incorrect answers vary in confidence and abstained-on queries differ in estimated answerability; CalibraRL combines Brier-based confidence penalties with posterior-adaptive abstention rewards from group-level correctness evidence.It reuses training rollouts without additional annotations about whether the model knows each answer.Source: §6 · p. 8
A Algorithm Details
- Algorithm 1 samples response groups, assigns confidence-aware answer or posterior-adaptive abstention rewards, computes fixed group advantages, and updates the policy using clipped policy and reference-policy KL losses.The posterior uses a fixed prior and correctness outcomes from non-abstaining responses in the group.Source: A Algorithm Details · p. 10
B.1 Datasets
The appendices describe the QA datasets, evaluation measures, training and prompt setup, and additional empirical analyses. The rollout-size sweep reports that G=32 performs best or near best overall.
B.1 Datasets
- NQ and TriviaQA represent open-domain factual QA, while HotpotQA tests multi-hop reasoning over evidence from multiple documents.The datasets cover single-hop and multi-hop question answering.Source: §B.1 · pp. 10–11
B.2 Evaluation Metrics
- Evaluation uses Precision for accuracy among answers, Truthfulness for correct answers or abstentions, ER for net answering utility, and ECE for calibration on answered examples.Abstentions contribute positively to Truthfulness, so the paper interprets it jointly with Precision and ER.Source: §B.2 · p. 11
B.3 Training Details
- CalibraRL is trained with GRPO and LoRA for one epoch, sampling 32 responses per query for each backbone model.The confidence-penalty and posterior-adaptive abstention reward scales are both 0.5.Source: §B.3 · p. 11
B.4 Prompt
- The prompt asks for concise answers grounded in retrieved documents and requires the exact response “I don’t know” when the model is uncertain.The user message supplies the question and retrieved documents as external evidence.Source: §B.4 · p. 11
C.1 Additional Empirical Motivation
- Figures 8–9 extend the NQ protocol to TriviaQA and HotpotQA, examining confidence–accuracy, incorrect-answer confidence, and reference accuracy for abstention queries.The captions report generally increasing accuracy with confidence, heterogeneous confidence among incorrect responses, and differences in reference accuracy for questions eliciting abstention.Source: §C.1 · p. 12, Figure 8 · p. 12, Figure 9 · p. 12
C.2 Effect of Rollout Group Size
- Precision rises by 10.63, 5.10, and 4.72 percentage points on NQ, TriviaQA, and HotpotQA, respectively, as G increases from 4 to 32.Truthfulness and ER also improve; G=32 achieves best or near-best performance overall, while increasing G to 64 yields inconsistent gains.Source: Figure 10 · p. 12, §C.2 · p. 12