TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

Zhepei Wei; Xiao Yang; Kai Sun; Jiaqi Wang; Rulin Shao; Jingxiang Chen; Mohammad Kachuee; Teja Gollapudi; Yiwei Liao; Nicolas Scheffer; Rakesh Wanga; Anuj Kumar; Yu Meng; Wen-tau Yih; Xin Luna Dong

Source-linked AI summary

TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Jingxiang Chen, Mohammad Kachuee, Teja Gollapudi, Yiwei Liao, Nicolas Scheffer, Rakesh Wanga, Anuj Kumar, Yu Meng, Wen-tau Yih, Xin Luna Dong

arXiv:2509.25760v2Sep 30, 2025cs.CLcs.AIcs.LG

TL;DR

LLMs often hallucinate when questions exceed their knowledge, while abstention-oriented methods can become overly conservative and accuracy-oriented methods can encourage guessing. TruthRL uses GRPO with a ternary reward to optimize truthfulness, substantially reducing hallucinations and improving truthfulness across four knowledge-intensive benchmarks. Its gains are associated with better recognition of the model’s knowledge boundary.

Problem
LLMs must provide accurate answers while recognizing uncertainty, but accuracy-driven training can encourage hallucinations and abstention-oriented methods can become overly conservative.
Method
TruthRL uses GRPO with a ternary reward that rewards correct answers, penalizes hallucinations, and treats abstentions neutrally.
Results
TruthRL achieves up to 43.4% higher truthfulness and 40.1% lower hallucination on average across four knowledge-intensive benchmarks.
Takeaways & Limitations
TruthRL encourages accurate responses and abstention under uncertainty by directly optimizing truthfulness rather than accuracy alone.
Takeaways & Limitations
Vanilla SFT memorizes training data, has limited generalizability, and encourages answers even when the model is unsure.

Abstract

from arXiv · show

While large language models (LLMs) have demonstrated strong performance on factoid question answering, they are still prone to hallucination and untruthful responses, particularly when tasks demand information outside their parametric knowledge. Indeed, truthfulness requires more than accuracy -- models must also recognize uncertainty and abstain when unsure to avoid hallucinations. This presents a fundamental challenge for existing methods: approaches that optimize for accuracy often amplify hallucinations, while those that encourage abstention can become overly conservative, sacrificing correct answers. Both extremes ultimately compromise truthfulness. In this work, we present TruthRL, a general reinforcement learning (RL) framework that directly optimizes the truthfulness of LLMs. Specifically, we implement TruthRL using GRPO with a simple yet effective ternary reward that distinguishes correct answers, hallucinations, and abstentions. It incentivizes models to reduce hallucinations not only by providing correct responses, but also by enabling abstention when uncertain, thereby improving truthfulness. Extensive experiments across four knowledge-intensive benchmarks show that TruthRL significantly reduces hallucinations (e.g., 43.5% $\rightarrow$ 19.4%) and improves truthfulness (e.g., 5.3% $\rightarrow$ 37.2%), with consistent gains across various backbone models. Analysis shows that the improvement of TruthRL arises from enhanced capability of LLMs to recognize their knowledge boundary, hence avoiding being overly conservative as the baselines are.

1. Introduction

LLMs can hallucinate when uncertain, so truthfulness requires both accurate answers and appropriate abstention. TruthRL addresses this gap by optimizing truthfulness with a ternary reward that rewards correctness, penalizes hallucinations, and treats abstentions neutrally.

LLMs often produce plausible factual errors instead of acknowledging uncertainty, creating serious risks in high-stakes domains such as law and medicine.The paper frames admitting “I don’t know” as important when questions exceed the model’s knowledge.
Existing uncertainty-training methods can require costly model-specific annotation and become overly conservative, while accuracy-driven methods encourage guessing over abstention.This leaves models struggling to provide accurate answers while acknowledging uncertainty.
TruthRL directly optimizes truthfulness with a ternary reward that rewards correct answers, penalizes hallucinations, and treats abstentions neutrally.The framework is implemented with GRPO and is designed to encourage answering when possible while abstaining under uncertainty.
The paper advocates shifting LLM training from accuracy-driven objectives toward truthfulness-driven methods.TruthRL is presented as a general reinforcement learning framework for this shift.

2. Preliminaries

The paper defines truthfulness using accuracy, uncertainty, and hallucination, then contrasts accuracy-focused fine-tuning with methods intended to preserve appropriate abstention. Preliminary scaling results show that vanilla SFT and RL can improve accuracy while suppressing uncertainty and retaining substantial hallucination.

2.1. Problem Formulation: Truthfulness = w1·Acc + w2·Unc − w3·Hall combines accuracy, abstention-based uncertainty, and hallucination rate.Accuracy is rewarded, hallucination is penalized, and the uncertainty weight is determined by deployment needs; the default is w1 = 1, w2 = 0, w3 = 1.
2.2. Vanilla Fine-tuning Methods: SFT trains models to reproduce ground-truth responses even when unsure, encouraging hallucinations and limiting generalization.Traditional RL can generalize better than SFT but remains insufficiently designed for recognizing uncertainty or abstaining appropriately.
2.3. Preliminary Findings: Increasing majority@k samples reduces hallucination while improving accuracy and abstention for the prompting baseline on CRAG.The result uses Llama3.1-8B-Instruct and indicates potential for higher truthfulness without fine-tuning.
2.3. Preliminary Findings: Vanilla SFT and RL improve accuracy but nearly eliminate abstention and provide limited or even negative hallucination reduction as k grows.These accuracy-driven methods can diminish the base model’s capacity to express uncertainty.

3. Methodology

TruthRL combines knowledge-boundary probing, GRPO, and a ternary reward to distinguish correct answers, uncertainty, and incorrect responses. This design encourages abstention over hallucination when the model lacks knowledge while supporting truthful performance across benchmarks.

Knowledge Boundary Probing: Knowledge-boundary probing marks a question as out-of-knowledge when 256 sampled responses contain no correct answer, then supports uncertainty-oriented training labels.The supplied passages describe probing and the subsequent “I don’t know” relabeling used by the R-Tuning baseline.
TruthRL reward design: TruthRL uses GRPO with a ternary reward that gives +1 for correct answers, 0 for uncertainty, and −1 for incorrect responses.The reward explicitly separates abstention from hallucination during policy optimization.
TruthRL reward design: Under GRPO, ternary rewards give abstentions a larger relative advantage than hallucinations, encouraging abstention when the model lacks knowledge.Binary rewards assign both outcomes the same negative reward and therefore conflate them, whereas ternary rewards distinguish them.
Reward-design analysis: TruthRL maintains meaningful abstention under ternary rewards, whereas binary rewards quickly suppress abstention to nearly zero.The observed divergence is attributed to ternary rewards adaptively assigning positive or negative credit to abstention through relative advantages.
TruthRL variants: TruthRL also considers knowledge-enhanced and reasoning-enhanced variants, but analyses indicate the simple ternary reward generally performs better than binary or more complicated designs.The variants modify abstention rewards using out-of-knowledge status or add reasoning-quality signals.

4. Experiment

Across four knowledge-intensive benchmarks and multiple evaluation settings, TruthRL achieves the strongest truthfulness by jointly reducing hallucinations, preserving accuracy, and enabling selective abstention. Its ternary reward and knowledge-boundary behavior explain gains over accuracy-focused, knowledge-enhanced, and binary-reward baselines.

4.2. Main Result: Knowledge-enhanced SFT reduces hallucination with little or no accuracy loss, while vanilla SFT increases both accuracy and hallucination, especially without retrieval.These results show that optimizing accuracy alone can encourage answers beyond the model’s knowledge.
4.2. Main Result: TruthRL consistently achieves the highest truthfulness, reducing hallucinations while improving accuracy, with especially strong gains under retrieval.On CRAG with retrieval, it reduces Llama3.1-8B-Instruct hallucination by 24.1% and improves truthfulness by 31.9 percentage points versus prompting.
4.3. TruthRL improves LLMs in recognizing their knowledge boundaries: TruthRL produces minimal hallucinations on difficult questions by abstaining when knowledge is lacking, reaching 15.5% hallucination and 84.5% uncertainty.Other high-accuracy methods can hallucinate nearly 100% on these questions, whereas TruthRL recognizes its knowledge boundary.
4.4. Ablation Study: TruthRL’s ternary reward achieves the best truthfulness with competitive accuracy, whereas binary reward maximizes accuracy but suppresses abstention.Knowledge enhancement atop ternary reward does not improve results, likely because static probing cannot track the model’s evolving knowledge boundary.
4.4. Ablation Study: Neutral abstention reward yields the best balance: penalizing abstention increases hallucination, while rewarding it encourages excessive abstention.The reward comparison varies abstention incentives across negative, neutral, and positive settings.
4.3. TruthRL improves LLMs in recognizing their knowledge boundaries: TruthRL remains strongest across judges, abstention-weight settings, hallucination-baiting questions, unseen QA datasets, and the out-of-domain GSM8K benchmark.Its cross-domain evaluation suggests learned knowledge-boundary recognition rather than dataset-specific behavior.

5. Related Work

Related work addresses hallucinations through retrieval, decoding-based self-correction, and fine-tuning, but these approaches can fail out of distribution. TruthRL is positioned against this broader limitation in existing mitigation strategies.

Existing hallucination mitigation uses retrieval-augmented generation, decoding-based self-correction, SFT, and RLHF, but these approaches often fail out-of-distribution.

6. Conclusion

TruthRL directly optimizes LLM truthfulness with a ternary reward that encourages accurate answers, uncertainty-based abstention, and fewer hallucinations. Across four knowledge-intensive benchmarks, it improves truthfulness and reduces hallucinations on average.

TruthRL uses a ternary reward to encourage accurate answers, abstention when uncertain, and reduced hallucination.
TruthRL achieves up to 43.4% higher truthfulness and 40.1% lower hallucination on average across four knowledge-intensive benchmarks.

Impact Statement

TruthRL targets hallucinations and untruthful responses in knowledge-intensive tasks by rewarding both accuracy and appropriate abstention. The method is presented as a positive contribution to LLM reliability and trustworthiness, while broader ethical concerns remain.

TruthRL improves LLM reliability and trustworthiness by encouraging accurate answers and abstention when uncertain.
Broader ethical concerns involving fairness, bias, and potential misuse remain outside the contribution described here.

A. Additional Analysis

TruthRL’s additional analyses show that verifier quality, backbone scale, reasoning signals, and online optimization materially affect truthfulness outcomes. Across these analyses, TruthRL reduces hallucinations and improves truthfulness, including against production-level models.

Verifier analysis: Replacing the LLM-based verifier with a rule-based verifier causes overly conservative abstention, producing extremely low hallucination but negative truthfulness.The LLM-based verifier instead supplies fine-grained, context-sensitive training signals.
Backbone scalability: TruthRL consistently reduces hallucinations and boosts truthfulness across compact, mid-scale, and large backbones, with larger relative gains for smaller models.The method also improves 32B backbones, indicating benefits for highly capable systems.
Reasoning analysis: On CRAG, accurate responses usually have 92% reasoning quality, whereas uncertain responses score 0%, linking response type to reasoning quality.The analysis reports an overall truthfulness score of 5.3% and reasoning score of 50.2% for the prompting baseline.
Reasoning-reward variants: Multiplicative and additive reasoning-reward strategies modify the outcome reward to encourage reasoning quality alongside answer outcomes.The multiplicative strategy uses r_final = r_outcome · (1 + r_reason), while the additive strategy uses r_final = r_outcome + λ · r_reason.
RL paradigm comparison: TruthRL’s online RL outperforms offline DPO and semi-online iterative DPO on four knowledge-intensive benchmarks.Offline DPO provides limited gains because its fixed dataset constrains adaptive behavioral refinement.
Production-model comparison: TruthRL matches or surpasses GPT-5 and OpenAI o3 on several datasets, achieving the lowest hallucination rate and second-highest truthfulness score on CRAG.The comparison covers four knowledge-intensive benchmarks.

B. Extended Related Work

The related-work discussion situates TruthRL among hallucination mitigation, truthfulness fine-tuning, confidence calibration, and reinforcement-learning approaches. It emphasizes that existing methods can improve accuracy or calibration while still discouraging appropriate abstention or generalizing poorly beyond training distributions.

LLM hallucination and mitigation: Hallucinations are fluent but factually incorrect statements arising partly from limited external grounding and over-reliance on parametric recall.These errors are especially consequential in high-stakes domains such as law and medicine.
LLM hallucination and mitigation: Retrieval augmentation, decoding strategies, and fine-tuning have each been explored to reduce hallucinations or promote more truthful behavior.Examples include retrieval over external knowledge, self-consistency and calibrated decoding, SFT, DPO, and RLHF.
Method comparison: Table 13 distinguishes related methods by optimization paradigm, reward type, and how uncertainty is expressed.Its comparison includes static SFT supervision and online-reward RL methods such as PPO and GRPO.
Fine-tuning approaches: SFT, DPO, and RLHF can enhance accuracy on in-distribution topics, but their generalization may degrade significantly on out-of-distribution questions.These approaches commonly rely on curated datasets of factual question-answer pairs.
Confidence calibration vs. abstention: Confidence-calibration methods train models to express uncertainty estimates, while RLCR augments correctness rewards with a Brier-score term.This line of work differs from teaching models to abstain outright.
Reinforcement learning for LLMs: RLHF aligns models through preference-based reward models, while RLVR uses verifiable rewards and has elicited sophisticated chain-of-thought reasoning.Binary correct-versus-incorrect RLVR rewards conflate abstention with error, discouraging calibrated “I don’t know” responses.

C. Implementation Details

The implementation uses retrieval sources and preference data for baselines, full-parameter training infrastructure, and separate configurations for supervised, preference, and online RL methods. DPO constructs preferences differently for answerable and out-of-knowledge questions, while iterative DPO repeatedly rebuilds those pairs.

Experimental setup: The retrieval setup uses up to 50 web pages from a search API for CRAG and a 2018 Wikipedia dump with E5 retrieval for NQ, HotpotQA, and MuSiQue.These sources follow the stated CRAG and Search-R1-style retrieval setups.
DPO: DPO pairs “I don’t know” against incorrect answers for out-of-knowledge questions, and correct against incorrect answers for answerable questions.Its objective models preference probabilities with the policy rather than a separate reward model.
Iterative DPO: Iterative DPO starts from a DPO-trained checkpoint and repeatedly constructs preference pairs over the training set.The preference construction follows the same procedure as DPO.
Training details: Training uses full-parameter fine-tuning on 8 NVIDIA H100 80GB GPUs with Open-R1, DeepSpeed ZeRO-3 offload, gradient checkpointing, FlashAttention-2, and bf16.These settings are infrastructure choices for the reported experiments.
Training details: SFT, RFT, and R-Tuning use a 5e-6 learning rate and batch size 16, whereas DPO and iterative DPO use 3e-6 and batch size 32.Both groups are trained for one epoch with the specified optimization schedules.
Training details: RL training uses VeRL with learning rate 1e-6, batch size 64, KL coefficient β = 0.001, clip ratio ϵ = 0.2, and up to 16,384 context tokens.Rollouts use vLLM with tensor parallelism and a GPU memory utilization ratio of 0.8.
Inference details: Evaluation uses greedy decoding with temperature 0, while RFT data construction samples 64 responses at temperature 0.6 and top-p 0.9.The maximum inference length is 32k tokens.

D. Prompt Template

The paper provides separate inference prompt templates for non-retrieval and retrieval conditions, alongside dedicated judge prompts for outcome and reasoning quality. These templates define the prompting and evaluation interfaces used in the experiments.

Inference prompts: Table 14 contains the inference prompt for the without-retrieval setup, while Table 15 contains the prompt for the with-retrieval setup.The two tables correspond to the paper’s distinct inference conditions.
Judge prompts: Tables 16 and 17 provide separate LLM-as-a-judge prompts for evaluating outcome quality and reasoning quality.The judge prompts support the paper’s two evaluation dimensions.

E. Case Study

The case study presents LLM-as-a-judge prompts for evaluating outcome and reasoning quality, alongside a representative judgment example.

A representative example illustrates the accuracy of the LLM-based judge and reports an example judge result.
The case study includes an LLM-as-a-judge prompt for evaluating outcome.
It also includes an LLM-as-a-judge prompt for evaluating reasoning quality.