PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Reason for Hallucination Span Detection

Hsuan Su, Ting-Yao Hu, Hema Swetha Koppula, Kundan Krishna, Hadi Pouransari, Cheng-Yu Hsieh, Cem Koc, Joseph Yitan Cheng, Oncel Tuzel, Raviteja Vemulapalli

arXiv:2510.02173v2cs.CLcs.AIcs.LG

TL;DR

LLMs can produce unsupported content, while binary hallucination detection does not identify which spans are unreliable. This paper develops RL4HS, which learns reasoning with span-level rewards and CAPO, and reports stronger RAGTruth performance than pretrained reasoning models and supervised fine-tuning.

  • Problem

    Hallucination detection often treats the task as binary, although applications need to locate unsupported spans and the task requires multi-step factual verification.

  • Method

    RL4HS trains a reasoning model with GRPO using span-F1 rewards and adds Class-Aware Policy Optimization to correct reward imbalance.

  • Results

    RL4HS outperforms pretrained reasoning models and supervised fine-tuning on RAGTruth across summarization, question answering, and data-to-text tasks.

  • Takeaways & Limitations

    Span-level reward reinforcement learning can align reasoning with hallucination span detection and produce faithful reasoning traces that improve accuracy and robustness.

  • Takeaways & Limitations

    GRPO’s span-F1 reward can bias models toward predicting no hallucination spans, producing high precision but severely reduced recall.

Abstract

from arXiv · show

Large language models (LLMs) often generate hallucinations -- unsupported content that undermines reliability. While most prior works frame hallucination detection as a binary task, many real-world applications require identifying hallucinated spans, which is a multi-step decision making process. This naturally raises the question of whether explicit reasoning can help the complex task of detecting hallucination spans. To answer this question, we first evaluate pretrained models with and without Chain-of-Thought (CoT) reasoning, and show that CoT reasoning has the potential to generate at least one correct answer when sampled multiple times. Motivated by this, we propose RL4HS, a reinforcement learning framework that incentivizes reasoning with a span-level reward function. RL4HS builds on Group Relative Policy Optimization and introduces Class-Aware Policy Optimization to mitigate reward imbalance issue. Experiments on the RAGTruth benchmark (summarization, question answering, data-to-text) show that RL4HS surpasses pretrained reasoning models and supervised fine-tuning, demonstrating the necessity of reinforcement learning with span-level rewards for detecting hallucination spans.

1 Introduction

The paper frames hallucination span detection as a fine-grained alternative to binary detection and asks whether learned reasoning improves this complex task. It proposes reinforcement learning with span-level rewards and class-aware optimization, reporting gains over non-reasoning and general-purpose reasoning models.

  • Motivation: Hallucination span detection requires locating unsupported spans rather than merely deciding whether an output contains any hallucination.This fine-grained formulation is motivated by applications that need to assess the reliability of specific generated content.
  • Motivation: The task is a multi-step reasoning problem because each stated fact must be extracted and checked against the input context.The paper therefore investigates whether learned reasoning can support span-level verification.
  • Related Work: Existing Chain-of-Thought studies support reasoning for binary hallucination detection but do not address fine-grained span detection or train a task-specific reasoning model.The paper positions RL-based span-level reasoning as addressing both gaps.
  • Contribution: RL with span-level rewards significantly improves hallucination span detection over supervised fine-tuning with the same training data.The authors present this as evidence that reasoning learned through reinforcement learning benefits span detection.
  • Contribution: Class-aware policy optimization addresses GRPO’s reward imbalance by downscaling advantages for non-hallucination samples.This balances hallucination and non-hallucination classes and yields higher overall span-F1.
  • Results: State-of-the-art reasoning models trained on mathematics, coding, or tool-calling data perform significantly worse than a task-specific 7B reasoning model.This comparison motivates learning reasoning specifically for hallucination span detection.

2 Hallucination Span Detection

Hallucination span detection identifies unsupported text segments in generated responses, commonly for summarization and long-form question answering. The paper follows a generative approach that outputs hallucinated segments and evaluates them with dataset-level span-F1.

  • Task Definition: Hallucination span detection identifies every generated text segment that is unsupported by the input context.Each hallucinated span is represented by its start and end positions in the generated response.
  • Model: CoT reasoning provides increasingly larger Span-F1@K gains as the number of sampled predictions K increases.At K = 1, gains are absent or limited, while multiple sampling can produce at least one accurate prediction.
  • Model: The generative model directly outputs a list of hallucinated text segments, whose start and end indices are recovered by matching them in the response.This contrasts with encoder-based token-level binary classification approaches.
  • Evaluation Metric: Dataset-level span-F1 compares predicted hallucination spans with ground-truth spans using precision and recall over span sets.The supplied metric passages define the evaluation around predicted and ground-truth span collections.

3 RL4HS: Reinforcement Learning for Hallucination Span Detection

This section motivates RL for hallucination span detection through diverse CoT sampling, defines span-F1 rewards under GRPO, and introduces CAPO to correct class-imbalanced advantages.

  • 3.1 Motivating RL with diverse CoT reasoning: As K increases, CoT reasoning’s Span-F1@K gap grows significantly, suggesting multiple samples can produce at least one accurate hallucination-span prediction.At K = 1, CoT gives no gain for Qwen2.5-7B and limited gains for Qwen3-8B; similar behavior appears at 14B.
  • 3.2 RL with GRPO: GRPO uses group-relative returns rather than a critic, with its objective applying clipped probability ratios to advantages computed from sampled trajectories.The advantage baseline is based on group performance, normalized by group standard deviation.
  • 3.2.1 Verifiable Span-F1 Reward: RL4HS defines reward directly from span-F1 between predicted spans and ground-truth spans, awarding maximum reward when both span lists are empty.Otherwise, the reward reflects overlap quality between predicted and ground-truth spans.
  • 3.3 Reward Imbalance Across Classes: Non-hallucination predictions receive systematically higher advantages than hallucination predictions, independent of correctness, revealing GRPO’s class imbalance.The imbalance is shown across task splits and advantage distributions on Qwen2.5-7B-Instruct.
  • 3.4 Class-Aware Policy Optimization: CAPO scales advantages for non-hallucination samples with α to balance classes because standardizing GRPO rewards otherwise removes simple reward scaling effects.The experiments use α = 0.5, selected using validation performance.

4 Experimental Setup

The experiments use RAGTruth’s span annotations across summarization, question answering, and data-to-text, comparing RL4HS with supervised, GRPO, attention-based, and reasoning-model baselines.

  • Experimental Questions: Experiments are structured around five questions covering RL4HS effectiveness, CAPO’s precision–recall balance, in-domain reasoning, reward scaling, and what RL4HS learns.These questions organize the results and discussion in Section 5.
  • Dataset: RAGTruth provides paired source documents, generated responses, and human-labeled hallucination spans for summarization, question answering, and data-to-text.The benchmark supports span-level hallucination detection rather than only binary classification.
  • Models: The study primarily uses Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct, while evaluating pretrained and proprietary reasoning models as additional comparisons.Additional models include Qwen3-8B, Qwen3-14B, QwQ-32B, GPT-5, o3, GPT-4o-mini, and GPT-5-mini.
  • Baselines: Baselines include SFT, RL4HS-GRPO, and Multi-View Attention, representing supervised learning, standard GRPO training, and an attention-based detector.SFT is trained with cross-entropy on hallucination-span annotations.

5 Results & Discussion

RL4HS achieves the strongest span-level hallucination detection results across RAGTruth tasks, while CAPO addresses GRPO’s reward-hacking imbalance and improves precision–recall behavior. Out-of-domain and qualitative analyses further indicate that span-level rewards learn task-specific, systematic reasoning.

  • 5.1 Q1: What is the effectiveness of RL4HS?: Pretrained instruction-tuned models remain below 30 F1, and even Qwen3-14B reaches only 35.8 summarization F1, showing generic reasoning alone is insufficient for span localization.SFT improves substantially to 55.4 F1 at 14B, while Multi-View Attention reaches 49.1 F1 at 7B but remains behind larger SFT models.
  • 5.1 Q1: What is the effectiveness of RL4HS?: RL4HS-7B outperforms SFT across all three tasks, averaging 55.9 versus 50.1, while RL4HS-14B reaches 57.6 summarization, 54.8 QA, and 62.6 Data-to-Text F1.RL4HS also surpasses pretrained reasoning and proprietary GPT-4o/5-mini, GPT-5, and o3 baselines.
  • 5.2 Q2: Does CAPO alleviate reward hacking and achieve better precision–recall balance?: CAPO stabilizes recall while preserving precision, producing consistently higher span F1 than GRPO during training.GRPO’s high precision and declining recall reflect reward hacking toward non-hallucination predictions.
  • 5.3 Q3: Is in-Domain reasoning necessary for hallucination span detection?: RL4HS-OOD-7B performs consistently better than general-purpose reasoning models across held-out tasks and exceeds the much larger GPT-4o-mini with CoT.Qwen3 and QwQ often remain below 40 Span-F1, supporting the value of in-domain reasoning learned from span-level rewards.
  • 5.4 Q4: Can simply scaling rewards solve reward hacking?: Scaling Dr.GRPO rewards changes the precision–recall trade-off, but its variants do not surpass CAPO, which directly addresses reward imbalance.CAPO’s advantage scaling targets the systematic bias favoring non-hallucination predictions.
  • 5.5 Q5: What does RL4HS learn? A case study: In a case study, RL4HS identifies an unsupported catering claim by checking explicit claims against structured data and marking the inconsistency as hallucinated.The resulting trace follows a systematic consistency-checking procedure rather than a generic explanation.

6 Related Works

Prior hallucination detection research moved from binary judgments toward fine-grained span localization, while reasoning research developed methods such as GRPO for improving complex-task reasoning.

  • Hallucination Detection: Hallucination detection has evolved from binary classification toward span-level localization, including cascade pipelines that combine atomic-fact generation with natural-language inference.The cited prior approaches address whether hallucinations occur or where unsupported content appears.
  • Reasoning Enhancement in NLP: GRPO improves reasoning by comparing groups of outputs without requiring a separate value model and has been adapted to coding, planning, and tool-calling.The paper applies this optimization family to hallucination span detection.

7 Conclusion

RL4HS aligns LLM reasoning with hallucination detection through span-level rewards, while CAPO addresses reward imbalance. The framework outperforms pretrained reasoning models and SFT and produces faithful, heuristic-like reasoning traces.

  • Conclusion: RL4HS uses span-level rewards to align LLM reasoning with hallucination detection, with CAPO addressing reward imbalance.The framework builds on reinforcement learning for span detection.
  • Conclusion: RL4HS outperforms pretrained reasoning models and SFT on RAGTruth while producing faithful, heuristic-like reasoning traces that improve accuracy and robustness.CoT provides limited single-sample gains, whereas RL4HS distills multisample advantages into stronger predictions.

A.1 Prompt

The prompts ask models to assess hallucinations in summaries, answers, or structured-data articles and return labeled hallucinated spans in a JSON format. Reasoning variants additionally require explicit step-by-step reasoning before span extraction.

  • A.1 Prompt: The summarization, question-answering, and data-to-text prompts provide source material and generated content, then ask whether hallucinations are present.The data-to-text variant frames the response as an overview article written from structured JSON data.
  • A.1 Prompt: CoT variants insert numbered reasoning steps before the JSON span list, whereas non-CoT variants omit that reasoning instruction.This distinction is applied to both question answering and summarization prompts.
  • A.1 Prompt: All prompt variants require hallucinated spans to be compiled into a JSON object whose hallucination list is either populated or empty.The output format is shared across the task variants.

A.2 Training Details

Training details cover SFT and reinforcement learning configurations, inference settings, checkpoint selection, and class-imbalance handling.

  • A.2 Training Details: Table 4 reports the training configurations for supervised fine-tuning and reinforcement learning.
  • A.2 Training Details: RL training used group size 16, temperature 1.0, top-p 1.0, top-k -1, and clip_high = 0.28 on 8 H100 GPUs.Because reasoning data was unavailable, instruct models were fine-tuned with RL directly rather than SFT first.
  • A.2 Training Details: All trained models used 5 epochs, with checkpoints selected by best performance on a self-split validation set.
  • A.2 Training Details: GPT-series inference used top-p = 0.95 and temperature = 0.7, while hallucination examples were upweighted to balance classes.

A.3 Dataset Statistic

The dataset-statistics section reports hallucination and non-hallucination example counts for RAGTruth, alongside figures covering three benchmark tasks.

  • A.3 Dataset Statistic: Table 5 summarizes RAGTruth dataset statistics, listing hallucination examples and non-hallucination examples in parentheses.
  • A.3 Dataset Statistic: Figures 6 and 7 present hallucination span detection results with and without CoT reasoning for summarization, question answering, and data-to-text.
Loading 2510.02173v2…