Source-linked AI summary
Hallucination Span Detection with Input-Side Evidence Alignment
Miyu Yamada, Yuki Arase
TL;DR
Existing hallucination detectors often assess whole outputs without locating hallucinated spans or their supporting input evidence. This paper introduces a confidence-based masked-token prediction method that jointly detects hallucinated spans and aligns output tokens to input evidence, effectively detecting baseless hallucinations while producing useful alignments.
Problem
Existing methods mainly assess factuality at the sentence or document level, without localizing hallucinated spans or identifying corresponding input evidence for verification.
Method
An encoder predicts masked output tokens from input representations, using prediction confidence for hallucination detection and token representations for input-side evidence alignment.
Results
The method effectively detects baseless hallucinations and produces useful input-side alignments, with human evaluation confirming alignment quality.
Takeaways & Limitations
The approach supports joint localization of hallucinated output spans and identification of relevant input evidence for faithful and baseless hallucinated tokens.
Takeaways & Limitations
Inference is computationally expensive because each output token requires masking and re-encoding, making cost grow linearly with output length.
Abstract
from arXiv · showhide
Hallucinations remain a major obstacle to the reliable use of large language models (LLMs) in conditional text generation. Existing methods primarily assess the factuality of an entire generated text, providing limited insight into which output spans are hallucinated or how they relate to the input. We introduce the task of hallucination span detection with input-side evidence alignment, which jointly identifies hallucinated spans and aligns output tokens with the corresponding input evidence. Our approach is based on the observation that faithful output tokens are predictable from the input, whereas hallucinated tokens are not. We therefore train an encoder-based model to predict masked output tokens from the input representation, using prediction confidence for hallucination detection while naturally producing alignments to the input. Experiments show that the proposed method effectively detects hallucinated spans and identifies meaningful input-side evidence. Human evaluation confirms the quality of the predicted alignments.
1 Introduction
The paper targets hallucination span detection with input-side evidence alignment because existing output-level detection does not localize hallucinations or identify their supporting input evidence. It proposes distant-supervision methods based on output-token predictability and reports effective span detection with human-confirmed alignment quality.
- LLMs remain prone to hallucinations that are unsupported or contradicted by the input, undermining output reliability and limiting real-world deployment.
- Existing sentence- or document-level binary classifiers estimate overall factuality but neither localize hallucinated spans nor identify evidence for individual output spans.
- The proposed formulation uses distant supervision instead of manually aligned training data, exploiting the greater predictability of faithful output tokens from supporting input context.
- Experiments show effective hallucinated-span detection and simultaneous identification of supporting input evidence, while human evaluation confirms alignment quality for faithful and hallucinated words.
2 Related Work
Prior work has mainly detected hallucinations at the text level or identified hallucinated spans, but it has not jointly aligned faithful and hallucinated output tokens with input evidence. Related alignment research includes supervised and unsupervised monolingual word-alignment methods, including OTAlign’s optimal-transport formulation.
- Hallucination Detection: Hallucination detection has traditionally been formulated as text-level classification, using consistency across sampled responses or LLM-based fact-checking against reference documents.Representative methods include Manakul et al. (2023) and Hu et al. (2024).
- Hallucination Detection: Recent studies detect hallucinated spans with chain-of-thought reasoning, reasoning models, or encoder-based token and span classifiers.The passage identifies these approaches as more recent developments in hallucination span detection.
- Hallucination Detection: Previous hallucination-detection studies do not necessarily provide correspondences between output spans and input evidence, and none considers alignments of faithful and hallucinated tokens.This limitation motivates treating hallucination detection and input-side evidence alignment together.
- Word Alignment: Word alignment identifies correspondences between two sentences and is therefore closely related to this task’s alignment component.Most monolingual word-alignment methods have relied on supervised learning, while human annotation is expensive because it requires high expertise.
- Word Alignment: Unsupervised monolingual word-alignment methods include OTAlign, which formalizes word alignment as an optimal transport problem.The passage also cites Jalili Sabet et al. (2020) and Arase et al. (2023) among unsupervised approaches.
3 Proposed Method
The method detects hallucinated spans by predicting masked output tokens from input representations, assuming faithful tokens are input-predictable while hallucinated tokens are not. It produces evidence alignments through confidence-based matching and uses token-level inference for fine-grained detection.
- Core formulation: Faithful tokens are assumed predictable from supporting input context, so hallucination span detection is formulated as input-grounded masked token prediction.The approach is inspired by Non-Parametric Masked Language Models, which recover masked tokens by matching contextual representations with a reference text.
- Inference: Training uses span-level masking for efficiency, whereas inference masks each output token to obtain fine-grained hallucination labels and input-side evidence alignments.A token is hallucinated when its maximum confidence is below λh; word-level predictions classify a word as hallucinated only if all its subwords are hallucinated.
- Span Segmentation: Output texts are segmented into semantically meaningful spans using semantic role labeling, extracting predicates and arguments while retaining the finest spans.Overlapping predicate spans are divided into the smallest spans, and adjacent predicate spans are merged for passive or progressive constructions.
- Prediction Confidence: Each masked span is matched to the input span with the highest cosine-similarity confidence between mask representations and input-token representations.The model concatenates the input with masked output text and encodes both to obtain representations for the two special mask tokens and input tokens.
- Loss Function: The loss encourages higher confidence for faithful spans and lower confidence for hallucinated spans, with extra weight for hard examples.Hard examples include hallucinated spans exceeding the faithful confidence bound and faithful spans falling below the hallucination bound.
4 Experiment Settings
Experiments use RAGTruth’s QA and news-summarization tasks, with custom development and threshold-tuning splits. The proposed method is compared with Llama-SFT, LettuceDetect, and OTAlign under character-level hallucination metrics.
- Dataset: RAGTruth provides outputs from six LLMs across QA, data-to-text, and news-summarization tasks; this study uses QA and news summarization.QA inputs contain an MS MARCO passage and question, with an answer as output.
- Data Split: The researchers randomly held out 800 training examples for development and 1,000 for tuning the hallucination threshold λh, using the remainder for training.Each held-out set contains 400 or 500 examples per task, respectively.
- Evaluation Metrics: Evaluation uses RAGTruth’s official character-level precision, recall, and F1 score.These metrics evaluate hallucination span detection.
- Training: Following NPM, training masks 15.0% of output tokens, samples mask sizes geometrically with p = 0.5, and selects SRL-based spans nearest the sampled size.The supplied implementation description continues beyond the provided passage.
- Baselines: Comparisons include Llama-SFT, LettuceDetect, and OTAlign, representing a fine-tuned larger LLM, an encoder token classifier, and an optimal-transport word aligner.Llama-SFT fine-tunes Llama-3.1-8B-Instruct for one epoch at a learning rate of 2.0e-5; LettuceDetect uses ModernBERT-large.
5 Automatic Evaluation
Automatic evaluation assesses hallucinated-span detection on QA and summarization using precision, recall, and F1. LettuceDetect leads QA F1, Llama-SFT leads summarization F1, while the proposed method achieves the best recall on both tasks.
- Detection performance: Table 3 evaluates hallucination detection on QA and summarization with precision, recall, and F1 scores.The automatic evaluation investigates the proposed method’s capability to detect hallucinated spans.
- Detection performance: LettuceDetect achieves the highest F1 on QA, whereas Llama-SFT performs best on summarization.The comparison is task-dependent: the two baselines lead on different tasks.
- Detection performance: The proposed method achieves the best recall on both QA and summarization, favoring scenarios where hallucinations should not be missed.Recall is advantageous when failing to detect a hallucination is undesirable.
- Detection performance: OTAlign performs poorly because it assumes sentence-pair alignment, whereas the task aligns sets of sentences with potentially highly unequal text lengths.The resulting alignment problem is extremely unbalanced, limiting even unbalanced optimal transport.
6 Human Evaluation
Human evaluation assessed input-side evidence alignments on 700 sampled content words, finding useful evidence for faithful and baseless hallucinated tokens while exposing difficulty with conflict hallucinations. Examples showed exact, paraphrased, and word-to-phrase alignments, but also cases where conflicting claims were not detected.
- Evaluation Setup: 700 content words from 70 output texts were manually assessed using top-1 and top-3 predicted input tokens.The evaluator randomly selected 10 content words per text and judged alignment quality using criteria in Table 4.
- Alignment Quality: Proposed alignments provided useful evidence for faithful and baseless hallucinated tokens, averaging around score 2.Score 2 indicates that the aligned input tokens were at least relevant evidence.
- Alignment Quality: OTAlign produced more exact-evidence alignments on faithful words, but also more irrelevant alignments or prediction failures.OTAlign was evaluated only on faithful output tokens because it represents hallucinated words as null-alignments.
- Limitations: Conflict hallucination cases were challenging because relevant input text can produce close token embeddings that confuse similarity-based confidence estimation.The proposed method therefore assigned score 0 to the majority of these rare cases.
- Qualitative Examples: Examples demonstrated exact-word, paraphrase, and word-to-phrase evidence alignments, while one conflict example exposed a hallucinated number and another missed a contradiction.The method aligned the contradiction-related words but failed to judge that case as a hallucination.
7 Analysis
The analysis shows that detection recall varies sharply by hallucination type, with baseless hallucinations easier to detect than conflict hallucinations. Fine-tuning separates faithful and hallucinated-token confidence distributions, with task-dependent differences in conflict-hallucination prevalence.
- Hallucination types: RAGTruth divides hallucinations into conflict and baseless types, each further classified as evident or subtle.Evident hallucinations are explicit unsupported entities or facts, whereas subtle hallucinations involve implicit or nuanced inconsistencies.
- Hallucination types: The method achieves much higher recall for baseless hallucinations than for conflict hallucinations.Recall is 75.8 for evident baseless and 82.9 for subtle baseless hallucinations, versus 15.8 and 18.3 for evident and subtle conflict hallucinations.
- Confidence distributions: Before fine-tuning, faithful and hallucinated-word confidence scores largely overlap, whereas training shifts their distributions apart.Confidence scores are defined as representation similarities against the input texts.
- Confidence distributions: Hallucinated-word confidence distributions are bimodal, with baseless cases concentrated at lower confidence and conflict cases forming a separate peak.The detection threshold shown in Figure 3 is λh = 0.68.
- Generation tasks: The separation trend is more noticeable in summarization because conflict hallucinations are twice as frequent as in QA, while baseless hallucinations are half as frequent.These task differences are reported for the summarization and QA settings.
8 Conclusion
The paper introduces hallucination span detection with input-side evidence alignment to jointly identify hallucinated spans and align output tokens with corresponding input portions. The method detects baseless hallucinations, while conflict hallucination detection remains an important direction for future work.
- The task jointly identifies hallucinated spans in generated text and aligns output tokens with corresponding portions of the input.
- The proposed method effectively detects baseless hallucinations.
- Improving conflict hallucination detection remains an important future direction through more sophisticated confidence estimation for input-grounded token prediction.
Limitations
The method’s main limitations are computationally expensive inference and evaluation restricted to question answering and news summarization on RAGTruth. Its effectiveness across other tasks, domains, languages, and LLM families remains untested.
- Computational cost: Inference is computationally expensive because each output token requires independently masking, re-encoding, and predicting, causing cost to grow linearly with output length.Repeated forward passes can become expensive for long generated texts despite the encoder model being lightweight.
- Evaluation scope: Experiments cover only question answering and news summarization using the RAGTruth benchmark, leaving other tasks, domains, languages, and LLM families uninvestigated.The framework’s effectiveness in these broader settings remains to be investigated.