FaithLens: Detecting and Explaining Faithfulness Hallucination

1 Introduction

• Prompt-based detectors rely on advanced LLMs for reliable performance, making them inefficient for real-world deployment.Specialized classifiers can be more cost-efficient, but the paper identifies further explainability, cross-task generalization, and training-data quality challenges.Source: §1 · pp. 1–2
• Binary-only detectors provide no explanation, while task-specific and general-purpose systems can perform unevenly across tasks and synthetic training data may lack quality controls.The stated data risks include insufficient diversity and too many simple examples, which may limit performance in complex detection scenarios.Source: §1 · pp. 1–2
• FaithLens synthesizes explanation-bearing data, filters for label correctness, explanation quality, and diversity, then uses cold-start SFT and rule-based RL rewards for prediction and explanation quality.The explanation reward checks whether an untuned model can use an explanation to predict the correct label.Source: §1 · p. 2
• Across 12 diverse tasks, 8B-parameter FaithLens achieves state-of-the-art performance, surpassing GPT-5.2 and o3 while producing informative, coherent explanations.The paper also reports lower cost than these advanced models.Source: §1 · p. 2

2 Task Formulation

• A claim is faithful when a generic reader would affirm it as true according to the grounding document; contradiction, misinterpretation, or unverifiability makes it hallucinated.The formulation assigns faithful claims label 1 and hallucinated claims label 0.Source: §2 · p. 2
• FaithLens extends binary classification to jointly output a faithfulness prediction and a textual explanation that justifies it.The formulation is intended to provide informative, explainable outputs that improve trustworthiness in hallucination detection.Source: §2 · pp. 2–3
• The training process pairs cold-start SFT on synthesized explanation-bearing data with rule-based RL rewards for prediction correctness and explanation quality.The caption identifies these as the two stages of FaithLens training.Source: Figure 2 · p. 3

3 Methodology

• FaithLens training combines synthesized, filtered data for a supervised cold start with rule-based reinforcement learning.The reinforcement-learning stage uses prediction-correctness and explanation-quality rewards.Source: §3 · p. 3

3.1 Cold-Start Supervised Fine-tuning

• The cold-start stage fine-tunes the model on training data that pass the proposed quality filters.Source: §3.1 · p. 3, §3.1.2 · p. 4

3.1.1 Data Synthesis

• Advanced reasoning models generate document-claim samples with a chain of thought, explanation, and predicted label for supervised fine-tuning.Source: §3.1.1 · p. 3

3.1.2 Data Filtering

• Filtering rejects label mismatches, retains explanations that lower label perplexity, and tests sample diversity using probe-set predictions.The diversity filter addresses distribution bias toward particular tasks or hallucination patterns and easy samples.Source: §3.1.2 · pp. 3–4

3.2 Reinforcement Learning Training

• Because SFT can memorize simple samples and produce unclear explanations, the authors further train FaithLens with rule-based reinforcement learning.Source: §3.2 · p. 5

3.2.1 Reinforcement Learning Protocol

• For each document-claim pair, the model generates groups of explanations and predictions; GRPO uses their relative performance to guide policy updates.Source: §3.2.1 · p. 5

3.2.2 Reward Design

• The composite reward combines prediction correctness, whether a novice model can use the explanation to predict correctly, and output formatting.Source: §3.2.2 · p. 5
• Across 12 datasets, FaithLens outperforms specialized models and advanced LLMs including GPT-5.2 and o3.Source: Table 1 · p. 6

4.2 Results

• FaithLens achieves state-of-the-art overall performance across 12 tasks, outperforming specialized models and advanced LLMs while showing the lowest standard deviation across tasks.The comparison includes complex reasoning detection on HoVer and cross-task scenarios on LLM-AggreFact.Source: §4.2 · p. 7
• FaithLens produces high-quality explanations across readability, helpfulness, and informativeness, and its inference cost is $0.8/GPU-hour for 1.2K samples from 12 datasets.GPT-4.1 judges the explanation dimensions; the paper reports the lowest inference cost alongside state-of-the-art performance.Source: §4.2 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.3 Analysis

• Ablations report that each designed component improves the model, with filtering dimensions affecting prediction performance, explainability, and cross-task consistency in distinct ways.The rule-based RL stage with composite rewards further improves the SFT-initialized model; the explanation-quality reward improves explanation quality and final performance.Source: Table 5 · p. 7, §4.3 · p. 8
• FaithLens does not require claim decontextualization or decomposition, and the training process consistently improves performance when applied to Qwen-2.5-Inst and Llama-3.1-Inst backbones.The paper also compares explanations from FaithLens and GPT-4o on 120 selected samples, with majority voting over readability, helpfulness, and informativeness.Source: §4.3 · p. 8, Figure 3 · p. 8, Table 6 · p. 8

• Faithfulness hallucinations are outputs inconsistent with or unsupported by their input context, and occur across settings including RAG, summarization, and simplification.The paper distinguishes these from factuality hallucinations, which contradict real-world facts.Source: §5 · pp. 8–9
• Detection approaches use advanced LLMs or cost-efficient specialized models, including NLI-based and synthetic-data-trained systems, but current systems often lack explanations and vary across tasks.Advanced-LLM approaches are described as inefficient for real-world applications, while NLI approaches can struggle with diverse error patterns and fine-grained hallucinations.Source: §5 · p. 9
• The paper positions FaithLens as addressing these gaps through filtered synthetic data and rule-based reinforcement learning, aiming to combine trustworthiness, efficiency, and effectiveness.Source: §5 · p. 9

• FaithLens predicts whether a claim is faithful or hallucinated and provides a corresponding explanation for real-world users.Source: §6 · p. 9
• The training pipeline synthesizes data with explanations, filters it for quality, uses it for a cold-start fine-tuning stage, and then applies reinforcement learning.The reinforcement-learning rewards target both prediction correctness and explanation quality.Source: §6 · p. 9
• Using a cold-start dataset optimized with reinforcement learning and rewards for prediction correctness and explanation quality, FaithLens delivers advanced detection effectiveness across 12 tasks and high-quality explanations at much lower cost.The paper describes this as a balance of trustworthiness, efficiency, and effectiveness.Source: §6 · p. 9

Limitations

• FaithLens is restricted to textual settings; multimodal extension is out of scope because it would require different grounding signals and explanation formats.Source: Limitations · pp. 9–10
• Sequentially generating the CoT, explanation, and label adds inference overhead compared with similar-sized models that output only labels.Source: Limitations · p. 10
• FaithLens outputs binary faithful-versus-hallucinated labels; finer-grained detection is left for future work because current datasets lack a unified taxonomy.Source: Limitations · p. 10

Appendix

• Further appendix sections cover explanation evaluation, ablations, claim decontextualization and decomposition, human evaluation, and generalization across foundation models.Source: Appendix · p. 16
• The appendix includes additional material on hyperparameters, variant testing, a case study, datasets, baselines, implementation, and evaluation analyses.Source: Appendix · p. 16

A Dataset Details

• The evaluation uses 12 datasets, including 11 LLM-AggreFact tasks and HoVer; it adopts cleaned benchmarks because the originals contain 9.1% ambiguous and 6.6% mislabeled examples.HoVer adds open-domain claims requiring evidence from up to four Wikipedia articles, often with long-range coreference.Source: A Dataset Details · pp. 16–17
• ClaimVerify, ExpertQA, FC-GPT, and LFQA cover supportedness judgments on search-engine responses, expert queries, decomposed response facts, and retrieved-document answers.The datasets use sentence-level or atomic-fact annotations against cited, retrieved, or corresponding documents.Source: A Dataset Details · p. 16
• RAGTruth and REVERL add manually annotated RAG responses and sentence-level assessments of open-domain reasoning-chain correctness.Tofu dialogue-summarization datasets use linguist sentence-level annotations, while WiCE is also included.Source: A Dataset Details · pp. 16–17

B Baseline Details

• The specialized baselines include entailment-based AlignScore and MiniCheck-7B, trained with synthetic claim-document data, alongside FactCG-DBT and ClearCheck-8B.ClearCheck evaluations use its chain-of-thought output as the explanation; the cited work reports that CoT and direct answers yield similar evaluation results.Source: B Baseline Details · p. 17
• Advanced-LLM baselines include GPT-4o, o1, GPT-4.1, o3, o3-mini, GPT-5.2, DeepSeek-V3.2 variants, and Claude-3.7-Sonnet.The listed API versions are specified for these models.Source: B Baseline Details · pp. 17–18

C Implementation Details

• Data filtering proceeds through label correctness, explanation quality, and diversity, with the reported data counts covering filtered and training samples.Filtering is applied only during SFT; verified, high-quality, challenging samples are selected for RL.Source: C Implementation Details · p. 18, Table 8 · p. 18
• SFT uses Adam at a 1 × 10^-5 learning rate for three epochs, while RL uses a 1 × 10^-6 actor learning rate, group size 7, and two epochs.The SFT batch size is 16; RL uses a total minibatch size of 112 across seven GPUs.Source: C Implementation Details · p. 18
• For baselines evaluated under Seo et al. (2025)'s benchmark settings, the paper reports their results; it reproduces uncovered baselines and infers both those models and FaithLens twice.Experiments run on NVIDIA A800 SXM4 80G GPUs, and FaithLens uses the same prompt template for training and evaluation.Source: C Implementation Details · p. 18

D Explainability Results Details

• GPT-4.1 scores explanations for readability, helpfulness, and informativeness, with percentages reported as the final results.The paper also provides detailed results for the ablation, claim, and generalization studies in Table 9.Source: D Explainability Results Details · p. 18, Table 9 · p. 19
• For binary-only baselines, prompts request both an explanation and a prediction; requiring explanations has little to no effect on numeric predictions.The resulting explanations make outputs available for downstream explainability evaluation.Source: D Explainability Results Details · p. 19
• FaithLens explanations surpass GPT-4o particularly in helpfulness and informativeness across the combined automatic and human evaluations.The paper reports consistent improvements across judges, while readability scores fluctuate more when GPT-4.1 judges its own outputs.Source: D Explainability Results Details · p. 19

E Ablation Study Details

• Direct SFT on all FactCG data does not greatly improve performance, suggesting that simply scaling training data is insufficient for hallucination detection.The authors argue that a specifically designed training method is needed.Source: E Ablation Study Details · pp. 19–20

F Claim Decontextualization and Claim Decomposition Study Details

• Claim decontextualization uses preceding claims as context to expand claims affected by coreference or ellipsis before detection.The study prompts GPT-4.1 for this expansion and reports detailed results in Table 9.Source: F Claim Decontextualization and Claim… · p. 20
• Claim decomposition splits each claim into typically 2–4 atomic facts; it can improve results to some extent but raises inference time and cost by 2–4 times.The authors advise against using it until it yields a significant accuracy benefit.Source: F Claim Decontextualization and Claim… · p. 20

G Human Evaluation Details

• The 120-sample human evaluation draws 10 examples from each of 12 datasets and compares explanations on readability, helpfulness, and informativeness.Example order and model outputs were randomized to reduce presentation bias.Source: G Human Evaluation Details · p. 20
• Table 10 reports ablation and generalization explanation evaluations across readability, helpfulness, and informativeness, using GPT-4.1 as judge for those studies.N/A indicates that a trained model cannot provide the corresponding explanations.Source: Table 10 · p. 20
• Three bachelor’s or master’s students compare FaithLens with GPT-4o, with majority voting and discussion followed by a revote if needed.Participants are briefed on the evaluation principles before judging.Source: G Human Evaluation Details · p. 20

H Generalization Study Details

• The generalization study explores different model backbones and reports detailed results in Tables 9 and 10; initial-model explanations use the Appendix D prompt.The backbone study is described in § 4.3.Source: H Generalization Study Details · p. 20

I Parameter Study

• Data-diversity filtering is robust to cluster count and advanced embedding-model choice, while more clusters add perplexity-calculation time without significant performance gains.The authors recommend advanced open-source embedding models, such as Llama-Embed-Nemotron-8B, to retain effectiveness and reduce cost.Source: I Parameter Study · p. 21
• Using Llama-3.1-8B-Inst as the novice reward model outperforms Qwen-2.5-7B-Inst, while an expert reward model yields suboptimal explanation quality.The authors suggest an expert can still predict correctly despite an incorrect explanation, giving that explanation an undeserved high reward.Source: I Parameter Study · p. 21
• FaithLens remains ahead of GPT-4o and o1 under few-shot prompting and requires approximately 0.1% of o1’s inference cost.Few-shot prompting improves API-based LLM accuracy and partially narrows the performance gap.Source: I Parameter Study · pp. 21–22

J Variant Methods Testing

• Removing CoT from cold-start SFT greatly impairs performance, and SFT on CoTs alone is less effective than training with both CoTs and explanations.The authors attribute the latter result partly to filtered explanations helping performance and CoTs being difficult to use as high-quality explanations.Source: J Variant Methods Testing · p. 22
• The perplexity-based explanation-quality reward is less effective than rewarding whether a novice model predicts the ground-truth answer correctly.In the variant, the reward is 1 when the explanation reduces perplexity on the ground-truth answer and 0 otherwise.Source: J Variant Methods Testing · p. 23

K Case Study

• For the Lanham Act claim, FaithLens contrasts the claim with statutes listed in the documents, while GPT-4o gives a general summary and o1 a brief negative judgment.The authors describe the document’s omission of the Lanham Act as evidence supporting the explanation.Source: K Case Study · p. 23
• In the HoVer case, FaithLens restates the claim, cites document evidence, and distinguishes the hallucinated 1940 detail from correct claim content.The comparison describes GPT-4o’s explanation as initially misleading and o1’s as insufficiently detailed.Source: K Case Study · p. 23
• The modified baseline prompt requests both an explanation and a prediction, with little to no effect on numeric prediction outcomes.The corresponding prompt is shown in Figure 10.Source: Table 13 · p. 24, Figure 10 · p. 28