PaperlayerView exact revision on arXiv

Source-linked AI summary

HalluGuard: Demystifying Data-Driven and Reasoning-Driven Hallucinations in LLMs

Xinyue Zeng, Junhong Lin, Yujun Yan, Feng Guo, Liang Shi, Jun Wu, Dawei Zhou

arXiv:2601.18753v2cs.LGcs.AI

TL;DR

LLM hallucination detectors often address either training-related knowledge errors or inference-time reasoning instability, limiting unified analysis. This paper introduces the Hallucination Risk Bound and an NTK-based HALLUGUARD score that jointly detect both sources, achieving state-of-the-art results across diverse benchmarks and model backbones.

  • Problem

    Existing hallucination detectors usually target one source of error and lack a theoretical foundation for hallucinations that evolve during generation.

  • Method

    The paper decomposes hallucination risk into training-time semantic mismatch and inference-time instability, then uses an NTK-based score combining representation and rollout signals.

  • Results

    HALLUGUARD achieves state-of-the-art detection performance across 10 benchmarks, 11 baselines, and 9 LLM backbones.

  • Takeaways & Limitations

    The framework provides a unified lens for analyzing how data-driven and reasoning-driven hallucinations emerge and evolve during generation.

Abstract

from arXiv · show

The reliability of Large Language Models (LLMs) in high-stakes domains such as healthcare, law, and scientific discovery is often compromised by hallucinations. These failures typically stem from two sources: data-driven hallucinations and reasoning-driven hallucinations. However, existing detection methods usually address only one source and rely on task-specific heuristics, limiting their generalization to complex scenarios. To overcome these limitations, we introduce the Hallucination Risk Bound, a unified theoretical framework that formally decomposes hallucination risk into data-driven and reasoning-driven components, linked respectively to training-time mismatches and inference-time instabilities. This provides a principled foundation for analyzing how hallucinations emerge and evolve. Building on this foundation, we introduce HalluGuard, an NTK-based score that leverages the induced geometry and captured representations of the NTK to jointly identify data-driven and reasoning-driven hallucinations. We evaluate HalluGuard on 10 diverse benchmarks, 11 competitive baselines, and 9 popular LLM backbones, consistently achieving state-of-the-art performance in detecting diverse forms of LLM hallucinations. We open-source our proposed \model{} model at https://github.com/Susan571/HalluGuard-ICLR2026.

1 INTRODUCTION

LLM hallucinations threaten trustworthy deployment in high-stakes domains and arise from both flawed learned knowledge and inference-time reasoning failures. HalluGuard addresses this gap with a unified risk framework and NTK-based detector.

  • Motivation: Hallucinations in healthcare, law, and scientific discovery can undermine reliability, accountability, and user safety.An incorrect medical diagnosis may delay treatment or cause harmful interventions.
  • Hallucination Sources: Data-driven hallucinations stem from flawed, biased, or incomplete training knowledge, whereas reasoning-driven hallucinations arise from logical inconsistencies or multi-step inference failures.Existing detectors commonly target one hallucination source and may fail to capture how errors evolve during generation.
  • Unified Framework: The Hallucination Risk Bound decomposes hallucination risk into training-related semantic deviations and inference-time instability.The framework characterizes hallucinations as semantic approximation gaps amplified by unstable rollout dynamics across decoding steps.
  • Detector: HALLUGUARD uses an NTK-based score to jointly identify data-driven and reasoning-driven hallucinations.The score leverages NTK-induced geometry and captured representations.

2 PRELIMINARIES

The preliminaries distinguish hallucination signals associated with predictive uncertainty from those associated with cross-sample reasoning instability. They also introduce NTK geometry as a framework for analyzing training dynamics and representations.

  • Data-Driven Detection: Data-driven hallucination detection commonly uses predictive uncertainty, including sequence-level negative log-likelihood and entropy-based measures.Perplexity recovers average sequence uncertainty, while length-normalized and semantic entropy extend uncertainty analysis across stochastic samples.
  • Data-Driven Detection: Low perplexity indicates confident predictions, whereas high perplexity indicates implausible generations associated with weak priors.Length-normalized entropy captures dispersion across independent stochastic rollouts.
  • Reasoning-Driven Detection: Reasoning-driven hallucinations are associated with brittle inference trajectories that produce inconsistent or incoherent outputs from identical contexts.Cross-sample consistency scores quantify this instability using similarity between generated responses.
  • NTK Foundations: NTK quantifies similarity in training dynamics between inputs and supports analysis of how training shapes internal representations.In the infinite-width limit, the kernel is nearly constant during training, reducing nonlinear optimization to tractable kernel regression.

3 METHODOLOGY

The methodology models hallucination risk geometrically by separating representation mismatch from rollout instability. It then turns the theoretical decomposition into a tractable NTK-based HALLUGUARD score.

  • Problem Setting: Hallucinations are modeled in a semantic embedding space where generated sequences are mapped to representations and compared with a ground-truth representation.The model’s expected semantic output is defined under its autoregressive decoding distribution.
  • Problem Setting: The mean semantic response map and inference Jacobian characterize how internal perturbations affect the model’s reasoning trajectory.The Jacobian captures sensitivity of the reasoning trajectory to local inference instabilities.
  • Hallucination Risk Bound: The Hallucination Risk Bound decomposes total risk into systematic representation bias and stochastic rollout error under stated integrability, Lipschitz, and differentiability assumptions.The decomposition distinguishes errors caused by learned representation bias from errors introduced during stochastic rollout.
  • Data-Driven Term: Data-driven hallucination risk grows with pretraining-finetuning mismatch or weak task-aligned signal in the NTK eigenspace.The NTK conditioning ratio controls how closely the model representation approximates the true generation.
  • Reasoning-Driven Term: Reasoning-driven hallucination risk grows exponentially with sequence length as instability accumulates during autoregressive decoding.The analysis models generation as a martingale process and applies a Freedman-style concentration bound.
  • HALLUGUARD Score: HALLUGUARD combines det(K), log σmax, and −log κ2 to proxy representational adequacy, rollout amplification, and spectral instability.The score is designed as a computable alternative because direct step-wise Jacobians are impractical for billion-parameter LLMs.
  • HALLUGUARD Score: The proxy uses lightweight self-supervised spectral calibration layers optimized offline to align NTK properties across heterogeneous backbones without hallucination labels or task-specific supervision.This produces a stable, comparable geometric space for detection.

4 EXPERIMENTS

The experiments evaluate HALLUGUARD across task families, backbone scales, test-time reasoning, and fine-grained semantic hallucinations. Across these settings, it consistently outperforms baselines and its component terms track task-specific hallucination trends.

  • Experimental Setup: HALLUGUARD is evaluated across 10 benchmarks, 11 competitive detectors, and 9 publicly available LLM backbones.The evaluation covers data-grounded QA, reasoning-oriented tasks, instruction-following settings, multiple backbone scales, and two detection metrics.
  • Main Results: 81.76% AUROC and 79.76% AUPRC on MATH-500 exceed the second-best method by up to 8.3%.On RAGTruth, HALLUGUARD reaches 84.59% AUROC and 81.15% AUPRC; on TruthfulQA, it reaches 77.05% AUROC and 73.79% AUPRC.
  • Main Results: 72.89% AUPRCr on HaluEval with Llama2-7B is more than 10% above the second best, while gains persist on mid-sized and large models.Representative results include 79.01% AUROCr on GSM8K with Llama2-13B and 83.8% AUROCr on SQuAD with Llama2-70B.
  • Ablation Analysis: The data-driven term follows the ground-truth AUROC decline on SQuAD, while the reasoning-driven term mirrors the monotonic drop on MATH-500 as reasoning drift increases.The ablation results show that each term is matched to its corresponding task family and tracks performance trends as hallucination rates rise.
  • Test-Time Inference: 81.00% accuracy on MATH-500 and 70.96% on Natural show HALLUGUARD’s strongest test-time reasoning gains over IO Prompt.The improvements are approximately 10% on MATH-500 and 15.72% on Natural.
  • Fine-Grained Hallucinations: On PAWS, HALLUGUARD reaches 90.18% AUROC and 87.64% AUPRC on Llama2-70B, and 91.24% AUROC and 88.53% AUPRC on QwQ-32B.These results exceed the next-best method by nearly five points and remain leading on GPT-2.

5 RELATED WORK

Related work organizes hallucination detection around data-driven flaws and reasoning-driven inconsistencies. Existing approaches use external references, uncertainty, internal representations, self-consistency, entailment, and calibration-based scores.

  • Detecting Data-Driven Hallucinations: Data-driven hallucination detectors compare outputs with retrieved documents or references, estimate uncertainty, or probe internal representations.Examples include entropy-based methods, energy scoring, EIGENSCORE, MIND, ICR Probe, and representation-drift analysis.
  • Detecting Reasoning-Driven Hallucinations: Reasoning-driven hallucination detectors target logical errors, decoding instability, and temporal drift through sampled-generation consistency and calibration-based uncertainty.SelfCheckGPT measures alignment among candidate generations using entailment and lexical overlap.

6 CONCLUSION

The conclusion presents HALLUGUARD as a unified approach to hallucinations arising from training-time knowledge flaws and inference-time instability. It reports broad evaluation across benchmarks, baselines, and model backbones with consistently state-of-the-art detection performance.

  • Conclusion: The Hallucination Risk Bound decomposes hallucination risk into data-driven and reasoning-driven components, providing a principled view of how failures emerge during generation.The two components correspond to flawed acquired knowledge and unstable multi-step inference dynamics.
  • Conclusion: HALLUGUARD uses an NTK-based score to measure semantic-perturbation sensitivity and internal instabilities for holistic detection.The method is built on the theoretical decomposition and targets both hallucination sources jointly.
  • Conclusion: Across 10 benchmarks, 11 baselines, and 9 LLM backbones, HALLUGUARD consistently achieves state-of-the-art hallucination-detection performance.The conclusion characterizes these results as demonstrating robustness and practical efficacy.

REPRODUCIBILITY STATEMENT

The reproducibility statement directs readers to detailed theoretical assumptions, proofs, experimental settings, and evaluation protocols. These materials cover the full benchmark, baseline, and backbone evaluation.

  • Theory: Formal assumptions and proofs for the Hallucination Risk Bound are provided in Section 3 and Section A.The statement identifies these sections as the complete description of the theoretical framework.
  • Experiments: Experimental settings and evaluation protocols covering 10 benchmarks, 11 baselines, and 9 LLM backbones are documented in Section 4 and Section C.1.The statement presents these resources as supporting reproducibility of the empirical results.

ETHICS STATEMENT

The study uses only public datasets and open-source LLMs, without human subjects or private data. Its theoretical framework relies on localized, standard NTK and stability assumptions to support well-posedness and interpretation.

  • The study uses publicly available datasets and open-source large language models, without human subjects or private data.
  • The setup assumes a Lipschitz task-specific encoder mapping token sequences into a Hilbert-space representation.
  • The analysis assumes integrability, bounded second moments, and local smoothness of the mean semantic response map.
  • These assumptions are restricted to a local perturbation neighborhood and expectation-level map rather than the entire token space.

A.2 PROOF OF SECTION 3.2

The proof derives the hallucination-risk bound by splitting semantic error into approximation and stochastic components, then bounding each through NTK approximation theory and concentration. It further decomposes the NTK continuity constant into baseline, pre-training, and domain-mismatch contributions.

  • Risk decomposition: The proof splits hallucination risk into approximation error and stochastic residual using the triangle inequality.The expected semantic representation is separated from the sampled representation under autoregressive decoding.
  • Approximation term: The approximation term is bounded with C´ea’s lemma applied to an NTK-induced coercive bilinear form.The expected representation is treated as a Galerkin projection of the target onto the trainable hypothesis space.
  • Variance term: The variance term is bounded with Bernstein concentration for bounded random vectors under local perturbations.The resulting bound holds with high probability over sampled reasoning trajectories.
  • NTK continuity: The NTK continuity constant decomposes as Λ = Λ0 + ∆pt + ∆mm, separating baseline, pre-training noise, and fine-tuning domain mismatch.The pre-training deviation scales logarithmically with network parameters, while mismatch depends on task signal and empirical mismatch error.

B HALLUGUARD DERIVATION AND INTERPRETATION

HalluGuard combines NTK spectral adequacy, rollout amplification, and conditioning stability into a computable surrogate. Its implementation uses log-determinants, Jacobian estimates, normalization, perturbation controls, and lightweight calibration while keeping the backbone frozen.

  • Preliminaries and notation: The NTK Gram matrix is characterized through its eigenvalues, condition number, feature matrix, and decoder Jacobian norms.These quantities provide the spectral and dynamical terms used in the surrogate score.
  • Representational adequacy: The derivation assumes a source condition and spectral envelope to bound representation adequacy through eigenvalue decay and det(K).The determinant connects the smallest eigenvalue to approximation-error bounds, with slower decay improving the bound.
  • Implementation: Implementation replaces det(K) with log det(K) when needed, estimates σmax or an average Jacobian form, and z-normalizes components.The system also uses ridge stabilization, hidden-feature clipping, frozen backbones, and trained lightweight projection layers.
  • Variance amplification: Projector perturbation analysis yields quadratic variance amplification in the NTK condition number κ(K).The κ(K)^2 dependence combines projector sensitivity with the smallest-eigenvalue factor in the mean-squared bound.
  • Compact surrogate: HALLUGUARD combines det(K), log σmax, and −log κ(K)^2 as adequacy, amplification, and conditioning terms.The surrogate is obtained by combining the determinant, rollout-amplification, and variance-amplification analyses.

C.2 ABLATION STUDY ON −log κ2

The MATH-500 ablation tests whether the stability term −log κ2 is necessary as reasoning drift increases. Removing it is competitive at low drift but produces substantial volatility and nearly doubles the error reported in Table 6.

  • Ablation setup: The ablation progressively increases perturbation budget ρ and rollout horizon T to systematize reasoning drift on MATH-500.
  • Low-drift regime: For reasoning drift d < 0.15, the score without −log κ2 remains competitive.
  • Stability term: As reasoning complexity increases, removing −log κ2 causes significant performance volatility, whereas the full score penalizes ill-conditioned representations.
  • Table 6: The error in Table 6 nearly doubles without the stability term.

C.3 ABLATION STUDY ON SEMANTIC ENCODER Φ

The semantic encoder ablation evaluates whether HALLUGUARD’s performance depends on the default representation. Across benchmarks, the default encoder remains strongest, supporting the importance of NTK geometric structure beyond surface semantic similarity.

  • Encoder ablation: HALLUGUARD was evaluated with BERT, SimCSE, and E5 across multiple backbones and benchmarks.These alternatives replace the default semantic representation in the encoder ablation.
  • Encoder ablation: Across RAGTruth, GSM8K, and TruthfulQA, HALLUGUARD consistently outperformed encoder-substituted variants.The comparison covers both AUROC and AUPRC across the reported settings.
  • Interpretation: The results indicate that performance gains arise from geometric structure in reasoning trajectories rather than surface semantic similarity of final outputs.External encoders do not fully preserve the trajectory geometry captured by HALLUGUARD.
  • Efficiency: Less than 1 ms of post-processing time per query was required for feature clipping and NTK-score computation via the Gram matrix.The latency measurement used batched parallel sampling to generate K = 10 trajectories on NVIDIA A100/H200 GPUs.
  • Overall evaluation: HALLUGUARD consistently achieved state-of-the-art detection performance across five model families and three benchmark regimes, particularly at low false-positive rates.Additional comparisons included AUROC, AUPRC, and deployment-critical operating points.
Loading 2601.18753v2…