PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Multi-Signal Fusion for Token-Level Hallucination Detection

Igor Itkin

arXiv:2608.18115v1cs.CLcs.AIcs.LG

TL;DR

Token-level hallucination detectors often treat tokens independently, despite hallucinations forming temporally extended spans. This paper fuses multiple external signals with sequence labeling, achieving 0.840 AUC on RAGTruth, an 11.0-point gain over independent logistic regression.

  • Problem

    Independent token-level detection overlooks temporal dependencies in hallucinated spans, limiting evidence about how hallucination signals evolve across positions.

  • Method

    The method performs sequence labeling with a BiGRU over 33-dimensional per-token features that fuse text statistics, NLI entailment, language-model surprisal, and temporal statistics.

  • Results

    0.840 AUC exceeds independent logistic regression by 11.0 points on RAGTruth across multi-seed analysis.

  • Takeaways & Limitations

    The detector is directly deployable against closed-source APIs and degrades by under 4% on unseen source models.

  • Takeaways & Limitations

    Test-time feature extraction requires DeBERTa and TinyLlama inference passes, which may limit use in latency-sensitive applications.

Abstract

from arXiv · show

Token-level hallucination detectors score each token independently from a single signal, and fail exactly when the generating model is confidently wrong. This paper instead treats hallucination as a temporally extended span and detects it by sequence labeling: each token is scored from a 33-dimensional feature stream that fuses text statistics, Natural Language Inference (NLI) entailment, and language model surprisal, with no access to model internals. A Bidirectional Gated Recurrent Unit (BiGRU) over these features reaches an AUC of 0.840 on RAGTruth (10 seeds), an 11-point gain over an independent logistic-regression baseline (p = 0.002, Wilcoxon signed-rank). A controlled decomposition attributes most of the gain to temporal order rather than model capacity: evidence propagates from confident positions to ambiguous neighbors within a span. The same 0.845 ceiling recurs across recurrent, state-space (Mamba), and attention architectures, locating the bottleneck in the feature set rather than the model. Because it reads only the generated text and external signals, the detector works on closed-source models, and it keeps working on text produced by language models it never saw during training, losing under 4% AUC.

1 Introduction

Hallucinations are temporally extended, silently generated failures that per-token detectors miss, especially when models are confidently wrong. The paper therefore frames detection as sequence labeling with temporally enriched fusion of text, NLI, and surprisal signals, achieving stronger detection than independent classifiers.

  • Motivation: A single hallucinated claim can propagate into downstream healthcare, legal, or financial decisions because generation provides no intrinsic failure signal.The introduction presents silent failure as the central practical risk motivating detection.
  • Limitations of prior work: Existing detectors fail on high-confidence hallucinations, partial subject knowledge, and out-of-distribution data.These limitations motivate a detector that does not rely solely on model confidence or internal representations.
  • Motivation: Hallucinations form characteristic runs whose statistical signatures evolve jointly as erroneous tokens bias subsequent generation.The paper reports a median hallucination span in RAGTruth, though the supplied passage truncates before giving its value.
  • Method: 33 features combine text statistics, DeBERTa NLI entailment, TinyLlama surprisal, and temporal derivatives for each token.The derivatives include running means, first-order differences, and windowed extremes; a BiGRU learns joint dynamics across positions.
  • Results: 0.840 AUC is achieved by the BiGRU, outperforming independent classifiers by 11 points.A controlled decomposition attributes 44% to temporal order, 24% to sequence aggregation, and 32% to nonlinear capacity.

2 Related Work

Prior hallucination detectors span four signal-and-access families, but all score tokens independently and therefore miss evolving neighborhood context. This work positions itself as the first token-level method to jointly fuse complementary signals and model their temporal dependencies for localization.

  • Detector families: Four detector families are distinguished by their signals and required access: internal probes, uncertainty/logit methods, token classification/NLI, and activation-based methods.The survey focuses on methods evaluated on RAGTruth.
  • Detector families: Internal probes separate hallucinated tokens in representation space but require generating-model internals, unavailable for closed-source APIs.Examples include linear hidden-state probes, sparse autoencoder features, and jointly trained detection heads.
  • Detector families: 87.5% versus 33.3%: hallucinated states corrupt continuations more readily than reverse repair restores them, supporting forward temporal modeling.The figures report corruption and repair rates, respectively.
  • Detector families: Existing token- or span-classification systems reach span F1 of 58.9% for LettuceDetect and 58.3% for Su et al. on RAGTruth.LettuceDetect fine-tunes a 395M-parameter ModernBERT encoder, while Su et al. train 7–14B generative models with reinforcement learning.
  • Structural limitation and positioning: All four families score each token in isolation, whereas this approach fuses multiple signal types and models their joint temporal evolution across the sequence.The paper identifies token-level temporal dependencies and learned fusion of complementary signals as its distinguishing contributions.

3 Method

The method formulates hallucination detection as token-level sequence labeling over 33-dimensional features derived from text, NLI, and language-model signals. It combines temporal feature enrichment with bidirectional sequence models while using only generated text and external models, not generator internals.

  • Problem formulation: Each response token receives a binary hallucination label, and a sequence labeler maps the full 33-dimensional feature matrix to token probabilities.Training minimizes class-balanced binary cross-entropy.
  • Training and evaluation: Evaluation uses token-level AUC-ROC for threshold-independent ranking quality and token-level F1 at threshold 0.5.Because approximately 5.6% of tokens are positive, the loss uses class-balanced weighting.
  • Feature construction: 33 features combine text divergence indicators, NLI contradiction and entailment signals, and language-model statistical anomalies.All features come from generated text and external models, without access to the generating LLM’s internals.
  • Feature construction: Temporal enrichment makes sequential dynamics explicit through running statistics, deltas, and windowed extremes before sequence modeling.Text features capture source divergence, NLI features semantic contradiction, and language-model features statistical anomalies.
  • Sequence labelers: The architecture comparison spans models from non-temporal baselines to structured prediction, including MLP, BiGRU, BiLSTM, 1D-CNN, and CRF variants.The MLP matches BiGRU’s parameter count to distinguish capacity from cross-token modeling.

4 Experimental Setup

Experiments use RAGTruth as the primary benchmark and evaluate signal ablations, cross-model and cross-dataset transfer, and comparisons with Lookback Lens. The setup also specifies deterministic splitting, multi-seed evaluation, and paired Wilcoxon testing for the primary comparison.

  • Primary benchmark: RAGTruth contains 15,090 training and 2,700 test examples across QA, summarization, and data-to-text, generated by six LLMs.Annotations distinguish evident conflict from baseless information before conversion to token-level labels.
  • Transfer benchmark: PsiloQA provides 5,000 training, 1,098 test, and 890 validation examples from 11 source models, with 53% token-level hallucination prevalence.The English QA subset is used as a transfer target, creating a ten-fold positive-rate shift from RAGTruth.
  • Evaluation design: Four feature configurations crossed with eight architectures produce 32 signal-ablation conditions.The configurations are text only, text + NLI, text + LM, and all signals.
  • Transfer and comparison: Cross-model transfer leaves out each of six source LLMs, while cross-dataset transfer is bidirectional and zero-shot without target-domain fine-tuning.The study also compares against Lookback Lens on RAGTruth’s open-source subset using TinyLlama-1.1B and Qwen3-14B proxy encoders.
  • Statistical testing: Primary comparisons use paired Wilcoxon signed-rank tests across 10 seeds, with a minimum achievable two-sided p-value of 0.002.The single pre-specified comparison is BiGRU versus LogReg; remaining analyses are exploratory.

5 Results

Temporal sequence modeling, especially BiGRU, substantially outperforms token-independent detection because token order carries strong bidirectional signal. Performance is further improved by combining NLI, language-model, and text features, while transfer remains strong across unseen and closed-source models.

  • Architecture and signal comparisons: 0.840 ± 0.007 AUC across 10 seeds lets BiGRU exceed LogReg’s 0.730 ± 0.001 by 11.0 points (p = 0.002).The advantage persists across every signal configuration, including text features alone.
  • Architecture and signal comparisons: +1.3 points raises BiGRU from 0.832 to 0.845 when NLI and LM features are added to the signal set.Weaker models also benefit: LogReg gains +2.8 points and 1D-CNN gains +1.9 points.
  • Architecture and signal comparisons: 0.843–0.845 AUC is shared by seven properly trained sequence models spanning recurrent, state-space, and attention architectures.This shared ceiling suggests the feature set, rather than architecture, limits performance.
  • Temporal decomposition: 76% of label uncertainty is temporally redundant, while bidirectional context captures 91% of label entropy.The mutual-information lag decay predicts that local models capture most temporal signal.
  • Transfer and applicability: 0.808 mean AUC under leave-one-out transfer represents only a 3.8% relative degradation from in-distribution performance.BiGRU consistently outperforms LogReg and Transformer across all six held-out source models.
  • Transfer and applicability: 0.840 AUC on the full benchmark shows the method remains applicable to closed-source LLMs.On the open-source subset, combining the proposed features with Lookback reaches 0.866 AUC, +2.8 over Lookback alone.

6 Discussion

The discussion attributes sequence modeling’s advantage chiefly to local temporal order and contextual information, while identifying feature fusion—not any single signal—as central to performance. It also frames the BiGRU’s ranking strength and lower span-level F1 as consequences of different objectives and model scales.

  • Temporal information: 11 AUC points separate the sequence model from an independent classifier using identical features, with 44% of the gap attributable to token order alone.Context resolves 76% of label entropy after one step and 91% after both neighbors.
  • Temporal information: Roughly 10 tokens define the mixing time, supporting local rather than long-range temporal modeling and making a 1D-CNN a latency-friendly default.The recurrent model supplies the remaining quarter of the gain.
  • Temporal information: The 11-point BiGRU–LogReg gap shows that modeling interactions between positions adds value beyond per-token temporal summaries.Forward-looking features make forward recurrence partly redundant, while backward recurrence contributes complementary following-context information.
  • Signal fusion: Fusion drives the result because surprisal alone partly tracks generator style rather than hallucination itself.The counterfactual analysis cautions against treating the signal families as interchangeable.
  • Evaluation scope: 0.840 token-level AUC coexists with 0.394 span-level F1, versus LettuceDetect’s 0.589 span F1, reflecting different optimization targets and architectural scales.The approach uses 121K parameters on pre-extracted features and optimizes ranking quality, whereas LettuceDetect fine-tunes a 395M-parameter encoder for token classification.

7 Conclusion

The conclusion identifies temporal structure as the main untapped signal in token-level hallucination detection, while showing that deployment without model access remains practical. It also highlights annotation quality, accessibility tradeoffs, and streaming detection as key lessons and directions.

  • Core findings: 11 AUC points: A BiGRU exceeds independent classifiers on identical features, with controlled decomposition attributing most gain to token order.The task is framed as sequence labeling over a multi-signal feature stream.
  • General lessons: 0.730 AUC: Multi-signal fusion stalls without temporal modeling, while annotation density matters more than dataset size for transfer.Closing the span-F1 gap to fine-tuned encoders would require white-box access and three orders of magnitude more parameters.
  • Deployment: Under 4% degradation: The detector remains effective on unseen source models and is deployable against closed-source APIs without access to the generating model.It also combines additively with attention-based signals when those are available.
  • Future work: Future work should explore streaming detection during generation, using a unidirectional variant to trigger intervention before a hallucinated span completes.

Limitations

The method incurs test-time inference overhead, is evaluated only on English data, and may miss contradictions beyond truncated NLI contexts. Its span-level boundary precision trails LettuceDetect despite using far fewer parameters.

  • Computational and data limitations: 350M-parameter DeBERTa and 1.1B-parameter TinyLlama inference passes are required for feature extraction at test time.This overhead may limit use in latency-sensitive applications.
  • Computational and data limitations: English-only experiments leave open whether hallucination patterns differ in morphologically rich languages.
  • Context and boundary limitations: 400-word NLI context truncation may miss contradictions appearing in later source content.
  • Context and boundary limitations: 0.394 span-level F1 trails LettuceDetect’s 0.589, reflecting optimization for ranking quality rather than boundary precision.The comparison uses 3000× fewer parameters.

Ethics Statement … K Robustness Across Seeds

The study combines temporally enriched token features with sequence models to detect extended hallucination spans, while evaluating task variation, feature dynamics, onset detection, hyperparameters, and seed robustness. Its analyses characterize span persistence, evidence accumulation, and the limits of model capacity given the 33-dimensional input.

  • Ethics Statement: Public benchmarks contain no personally identifiable information, and the authors judge improved hallucination detection to outweigh potential misuse risks.The acknowledged risk is that detection signals could help adversaries craft harder-to-detect hallucinations.
  • A Complete Feature List: 33 features combine 12 temporal-context features with 21 pointwise features, and BiGRU adds +11 AUC points over LogReg.The temporal features use cumulative statistics, sliding windows, or finite differences; the gain is attributed to interactions between positions.
  • B Preprocessing Details: RAGTruth annotations become word-level labels using a more-than-50% character-overlap rule, while NLI and LM signals use external models with truncation and leakage-preventing median fallbacks.NLI uses DeBERTa-v3-large with a 400-word source-context limit; LM features use TinyLlama-1.1B as a proxy observer.
  • C Hallucination Span Statistics: 13,277 hallucination spans show strong persistence: P(Ht | Ht−1) = 0.902 versus P(Ht | Ft−1) = 0.006, implying a 150:1 persistence ratio and 10.2-token expected sojourn.Across 17,790 examples and 2.3M tokens, the median span length is 5 tokens, while 10.5% are single-token spans.
  • D Per-Task Performance; E Sequence Length Effect: 0.887 ± 0.009 QA AUC, 0.828 ± 0.015 Data2txt AUC, and 0.760 ± 0.011 Summary AUC all differ pairwise significantly at Wilcoxon p = 0.002.The temporal advantage is largest for Data2txt, where multi-token fabricated statistics and entity substitutions are most common; the BiGRU–LogReg gap grows from +8.1 in Q1 to +12.2 in Q4.
  • F Probability Evolution Through Spans: BiGRU predictions are highest at every aligned position and rise from 0.50 before span onset to 0.62 at position 15, while directional GRUs reflect forward or backward evidence access.ForwardGRU rises from 0.42 to 0.57; BackwardGRU starts at 0.49 because it has read the span from the right.
  • G Feature Dynamics at Onset; H Natural Counterfactual Details: At onset, LM entropy spikes by +0.48, context overlap drops by −0.12, NLI contradiction rises by +0.012, and BiGRU probability increases from 0.45 to 0.65.Same-model pairs show NLI contradiction +0.052 versus cross-model +0.002, while LM entropy differences shrink from +0.295 to +0.090; counterfactuals show spikes up to +4.5 nats.
  • I Onset Detection; J Hyperparameter Sweep; K Robustness Across Seeds: 1D-CNN achieves the best onset-detection F1 through high recall, whereas BiGRU achieves the best precision; BiGRU performance remains 0.837–0.843 AUC for smaller configurations.For h ≥128, AUC degrades and variance increases; h=64 with 2 layers and 121K parameters is selected for controlled comparison, while temporal models consistently outperform non-temporal baselines across seeds.

L Training Recipe Ablation … V Transfer Comparison

Across training, architecture, signal, calibration, and transfer analyses, temporal models’ performance is constrained primarily by feature representations rather than optimization, while transfer behavior varies across targets and datasets. The ablations also show complementary signal contributions and task- and length-dependent temporal advantages.

  • L Training Recipe Ablation: ∼0.840 AUC remains unbroken across six BiGRU training recipes, indicating the ceiling is determined by the 33-dimensional features rather than optimization.The recipes vary BCE versus Focal Loss and constant, cosine, or ReduceLROnPlateau schedules; Focal Loss increases recall by +0.02 over BCE at a precision cost.
  • L Training Recipe Ablation: [0.796, 0.805] AUC across all Transformer recipes confirms an approximately 0.80 ceiling that is architectural, not optimization-driven.Each recipe was evaluated over 10 seeds.
  • M Full Ablation Heatmap: Non-CRF temporal models, especially BiGRU and BiLSTM, consistently outperform baselines across all architecture–signal combinations.This full heatmap compares token AUC across architecture–signal combinations using a single seed.
  • N Cross-Model Transfer Details: GPT-4 is the hardest cross-model transfer target, whereas LLaMA-2-70B is the easiest.The comparison uses leave-one-out cross-model generalization with all signals and 10 seeds.
  • P Signal Contribution Analysis: NLI and language-model features provide complementary AUC gains beyond text features alone, including in the full open-LLM Lookback Lens comparison.The signal contribution result is presented as a BiGRU waterfall decomposition; the Lookback Lens table covers the open-LLM subset over 5 seeds.
  • Q Precision-Recall Curves: 5.6% hallucination rate in RAGTruth explains the low overall precision in the non-CRF precision-recall curves.The curves use all signals and seed 42.
  • R Per-Task Comparison: +8.1 to +12.2 AUC points describe the BiGRU–LogReg gap from the shortest to longest sequence quartile, with the largest task advantage on Data2txt; CRF forward-backward marginals recover up to +17.9 AUC points over softmax scoring.These results indicate length- and task-dependent temporal benefits, while CRF scoring depends materially on calibration method.
  • U Counterfactual Feature Comparison: NLI contradiction separates otherwise matched outputs when one LLM hallucinates, whereas LM entropy does not; reverse PsiloQA-to-RAGTruth transfer outperforms forward transfer despite fewer training examples.BiGRU integrates both NLI and language-model signals in the counterfactual comparison.

W Emerging Architecture Comparison · X Probing Comparison

Across seven architectures, plateau scheduling raises the AUC ceiling to 0.845 and largely removes architecture differences, while BiGRU remains fastest for deployment. Probing generator hidden states performs far below the 33-dimensional black-box features, and concatenating hidden states degrades performance.

  • W Emerging Architecture Comparison: Seven architectures were evaluated with literature-recommended recipes and an additional ReduceLROnPlateau plus early-stopping protocol.The latter used patience 10 and a maximum of 50 epochs.
  • W Emerging Architecture Comparison: ∼0.84 AUC is reached by BiGRU, Mamba, and BiGRU+Attention with cosine scheduling, while plateau scheduling raises the ceiling to 0.845.The comparison covered 5–10 seeds for cosine scheduling and five seeds for plateau scheduling.
  • W Emerging Architecture Comparison: 0.843–0.845 AUC is achieved by the top five architectures with plateau scheduling, indicating that the ceiling is determined by the 33-dimensional features rather than architecture.The top five are BiLSTM, Mamba, BiGRU+Attention, BiGRU, and BixLSTM.
  • W Emerging Architecture Comparison: +0.031, +0.016, and +0.010 are the gains for Transformer, BixLSTM, and BiLSTM, respectively, under plateau scheduling after fixed-epoch cosine undertraining.BiGRU converges fastest at best epoch 8 and is preferred for deployment efficiency.
  • X Probing Comparison: 0.543–0.574 AUC with a linear probe and 0.631–0.637 with BiGRU are far below the black-box method’s 0.770–0.833 on generator hidden states.The probing comparison used hidden states from Mistral-7B-Instruct and evaluated each generator only on its own outputs.
  • X Probing Comparison: 0.748–0.793 from combining hidden states with the engineered features is worse than the features alone at 0.770–0.833, indicating noisy hidden states degrade performance.The representations encode sequential context via self-attention but not the hallucination-detection signal captured by external features.
  • X Probing Comparison: All 32 layers were probed separately on LLaMA-2-7B-chat token-level outputs using a smaller n=450 test subset to test whether layer selection rescues probing.The experiment addressed whether the hallucination signal is concentrated in a specific intermediate layer.

Y Per-Task Architecture Comparison

Architecture performance varies by task: QA is easiest, while Summary is hardest, and Data2txt shows the clearest benefit from temporal modeling. On Data2txt, BiGRU reaches 0.833 AUC versus 0.795 for DilatedCNN.

  • 0.88 AUC is achieved by all architectures on QA, where factual errors produce sharp signals.QA is easiest across all architectures.
  • 0.833 AUC for BiGRU on Data2txt exceeds DilatedCNN’s 0.795 by 3.8 points.Data2txt shows the largest temporal advantage, especially for multi-token hallucination spans such as fabricated statistics and entity substitutions.
  • ∼0.76 AUC makes Summary the hardest task, consistent with subtle omissions and distortions at the token level.
Loading 2608.18115v1…