PaperlayerView exact revision on arXiv

Source-linked AI summary

From Out-of-Distribution Detection to Hallucination Detection: A Geometric View

Litian Liu, Reza Pourreza, Yubing Jian, Yao Qin, Roland Memisevic

arXiv:2602.07253v4cs.AIcs.CL

TL;DR

Hallucination detection in reasoning tasks remains challenging, and adapting OOD detection to LLMs raises practical issues involving training statistics, large label spaces, and stochastic decoding. This paper adapts two geometric OOD scores to token generation and averages stepwise uncertainty for sequence detection. The detectors show consistent superior performance across reasoning tasks, model architectures, and scales, and perform well with greedy and stochastic decoding.

  • Problem

    Applying OOD detectors to LLMs is challenging because training statistics are impractical to estimate, label spaces are large, and stochastic decoding adds uncertainty about score applicability.

    Source: §1 · pp. 1–2

  • Method

    The paper adapts lightweight geometric OOD scores based on feature proximity to weight vectors and distance to decision boundaries, averaging stepwise scores for sequence detection.

    Source: §1 · p. 2

  • Results

    Across reasoning tasks, model architectures, and scales, the detectors show consistent superior performance over a range of baselines and perform well under greedy and stochastic decoding.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents OOD-based, training-free, single-sample detectors as a promising route toward improved language-model safety.

    Source: §7 · p. 9

    Performance advantage may diminish on short-answer TriviaQA, and the study evaluates only two geometric OOD scores.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Detecting hallucinations in large language models is a critical open problem with significant implications for safety and reliability. While existing hallucination detection methods achieve strong performance in question-answering tasks, they remain less effective on tasks requiring reasoning. In this work, we revisit hallucination detection through the lens of out-of-distribution (OOD) detection, a well-studied problem in areas like computer vision. Treating next-token prediction in language models as a classification task allows us to apply OOD techniques, provided appropriate modifications are made to account for the structural differences in large language models. We show that OOD-based approaches yield training-free, single-sample-based detectors, achieving strong accuracy in hallucination detection for reasoning tasks. Overall, our work suggests that reframing hallucination detection as OOD detection provides a promising and scalable pathway toward language model safety.

1. Introduction

Existing hallucination detectors can be costly or ill-suited to reasoning, and applying OOD methods to LLMs raises practical challenges. The paper adapts lightweight geometric OOD scores for token generation and reports strong performance across reasoning tasks, model architectures, and scales.

1. Introduction

  • Training-based detectors can be sensitive to distribution shifts and costly to train, while multi-sample training-free methods add inference cost and face difficulties comparing diverse reasoning paths.These concerns motivate training-free, single-sample detection for reasoning tasks.Source: §1 · p. 1
  • Directly applying OOD detectors to LLMs is challenging because training statistics are impractical to estimate, the vocabulary is large, and stochastic token sampling adds score variance.Large label spaces can also make OOD scores noisier and increase costs across multi-step decoding.Source: §1 · pp. 1–2
  • The paper adapts lightweight geometric scores based on feature proximity to class weights and distance to decision boundaries, averaging stepwise uncertainty and thresholding it for hallucination detection.Lower proximity or smaller boundary distance indicates greater uncertainty.Source: §1 · p. 2, Figure 1 · p. 2
  • Across reasoning tasks, model architectures, and scales, OOD-inspired detectors show consistent superior performance over a range of baselines.The methods also perform well under greedy and stochastic decoding.Source: §1 · p. 2

2. Problem Statement

The task is to decide whether a generated response is factually correct, even though the set of correct responses is unknown at inference. The paper formulates this as a thresholded binary detector using a score based on the prompt, response, and model.

2. Problem Statement

  • A response is a hallucination when it falls outside the set of factually correct responses for its prompt.The paper denotes the correct-response set as Y*(x).Source: §2 · p. 2
  • At inference, the ground-truth set of correct responses for a prompt is unknown, so the detector must use the prompt, response, and model representations..Source: §2 · p. 2
  • The binary detector labels a response as hallucinated when its score is at or below threshold τ, with lower scores typically indicating lower certainty.Scores above τ receive the not-hallucination label.Source: §2 · p. 2

3. Geometry of Token Generation under a Classification View

The section treats next-token generation as classification over vocabulary tokens and adapts two geometric OOD uncertainty measures to language-model features. Experiments show that both measures separate hallucinated from correct responses in the reported setting.

3. Geometry of Token Generation under a Classification View

  • The section adapts feature-to-weight proximity and feature-to-boundary distance from OOD detection to characterize uncertainty in token generation.The measures capture the interplay between penultimate-layer features and the language head.Source: §3 · p. 3

3.1. Classification View

  • The language head is viewed as a shared linear classifier over vocabulary tokens, mapping each decoding-step feature to token logits; stochastic decoding may select a different token than greedy decoding.The weight and bias vectors are associated with individual tokens.Source: §3.1 · p. 3

3.2. Feature Proximity to Weight Vectors

  • NCI measures the scalar projection of the most-confident token’s weight vector onto the centered feature, where higher proximity corresponds to lower uncertainty.The centering uses an estimated mean of training features.Source: §3.2 · p. 3

3.3. Feature Distance to Decision Boundaries

  • The fDBD adaptation measures feature distance to token decision boundaries; larger distance means lower uncertainty, and the method approximates the distance for alternatives rather than repeatedly solving the optimization.Under greedy decoding, the decision region is where a token attains the maximum logit; the same terminology is retained for stochastic decoding.Source: §3.3 · p. 3

3.4. Geometric Uncertainty Signals Hallucination

  • Hallucinated-response features show lower weight-vector proximity and smaller decision-boundary distance than correct-response features, supporting both geometric measures as hallucination signals.The fDBD score explicitly measures penultimate-layer boundary distance, unlike prior noise-injection probing, which assesses it implicitly.Source: §3.4 · p. 4, Figure 2 · p. 4

4. From OOD to Hallucination Detection

This section adapts geometric measures from OOD detection to hallucination detection and evaluates their use in language-model reasoning. It addresses training-statistics estimation, vocabulary scale, and stochastic decoding.

4.1. Case Study Setups

  • The case study evaluates hallucination detection on 1,319 CSQA validation questions using Llama-3.2-3B-Instruct and AUROC, with greedy decoding unless otherwise specified.CSQA is a multiple-choice commonsense-reasoning task; higher AUROC indicates better performance.Source: §4.1 · p. 4

4.2. Challenge I: Estimating Training Statistics at Scale

  • Because full-corpus training statistics are impractical to compute reliably at LLM scale, the method uses the maximal-uncertainty point as an analytical proxy for the training feature mean.The proxy is defined by minimizing logit variance across the vocabulary.Source: §4.2 · p. 4
  • NCI with the analytical proxy for the training feature mean outperforms both the empirical-mean NCI variant and Perplexity on CSQA.Table 1 reports AUROC for Llama-3.2-3B-Instruct; higher is better.Source: §4.2 · p. 5, Table 1 · p. 5

4.3. Challenge II: Effectiveness and Efficiency in Massive Vocabulary Space

  • Restricting fDBD to the top-k confident alternative tokens excludes noisy low-probability-token scores and reduces expected per-step complexity from O(dmodel|V|) to O(dmodel k + |V|).The top-ranked token is excluded because the feature already lies within its decision region.Source: §4.3 · pp. 5–6
  • fDBD remains effective across tested alternative-set sizes and peaks at k = 1 000, outperforming the Perplexity baseline (AUROC = 63.23).The evaluated sizes are All, 1, 10, 100, 1 000, 10 000, and 100 000.Source: Table 2 · p. 5, §4.3 · p. 6

4.4. Challenge III: Robustness to Stochastic Generation

  • Across temperatures 0.2, 0.5, 0.8, and 1.0, NCI and fDBD consistently outperform Perplexity under stochastic decoding.Each temperature result is the mean AUROC and standard deviation over five random seeds.Source: §4.4 · p. 6, Table 3 · p. 6

5. Experiments

The experiments evaluate NCI and fDBD across reasoning benchmarks, model families, and detector baselines. The methods generally perform strongly, with selected-k fDBD usually outperforming its full variant, low sensitivity to k, and competitive results on TruthfulQA.

5.1. Main Results

  • NCI and fDBD add minimal inference latency over standard decoding, comparable to perplexity, on CSQA with Llama-3.2-3B-Instruct.Latency is measured end-to-end on an A100 in milliseconds per token.Source: Table 4 · p. 7, §5.1 · p. 8
  • Across datasets and architectures, NCI and fDBD outperform single-sample baselines, while selected-k fDBD almost always beats the full variant.The comparison also finds both methods generally outperform multiple-sampling detectors on reasoning tasks.Source: §5 · p. 7, §5.1 · p. 8
  • Experiments evaluate CSQA validation (1,221 questions), GSM8K test (1,319), and AQuA validation split (254) with Llama-3.2-3B-Instruct and Qwen-2.5-7B-Instruct against single- and multiple-sample detector baselines, reporting AUROC.All experiments use greedy decoding (T = 0), and k is selected with a training subset. Multiple-sample baselines use temperature 0.5 and sample size three.Source: §5.1 · pp. 7–8

5.2. Hyper-parameter Sensitivity

  • fDBD performance remains stable under tested k perturbations, so selecting its order of magnitude on validation is sufficient for test-set use.The tested values perturb selected k by ±5%, ±10%, and ±20%.Source: §5.2 · pp. 8–9, Table 5 · p. 8

5.3. Generalizability to Open-ended Tasks

  • On TruthfulQA, NCI and fDBD maintain competitive AUROC performance against single-sample baselines in an open-ended setting.The evaluation uses Llama-3.2-3B-Instruct and fDBD with k = ALL.Source: §5.3 · p. 9, Table 6 · p. 9

6. Related Work

The paper connects classifier OOD detection with hallucination detection and extends geometric OOD detectors to LLMs. It contrasts this approach with prior detection strategies and identifies limitations in answer-format coverage and the range of OOD scores studied.

6.1. Out-of-Distribution Detection

  • The work extends feature-space OOD detectors NCI and fDBD to hallucination detection in LLMs.Prior classifier OOD work defines scores in output or feature space; the paper builds on the latter line.Source: §6.1 · p. 9

6.2. Hallucination Detection

  • Multi-sample detectors incur repeated-sampling overhead, while separate single-inference classifiers can require training and face training-test distribution shift.The paper instead analyzes distances between internal features and the model’s decision boundary.Source: §6.2 · p. 9
  • The methods’ advantage can diminish on short-answer datasets such as TriviaQA, and the study covers only two geometric OOD scores.The authors identify answer-format gaps and adapting broader OOD methods as directions for future work.Source: §6.2 · p. 9

7. Conclusion

The conclusion presents adapted OOD detectors as training-free, single-sample approaches for hallucination detection in reasoning tasks. Supplementary evaluations report strong results on larger models, across model scales, and on a base model.

7. Conclusion

  • Adapting OOD detectors to LLM structure yields training-free, single-sample methods that identify hallucinations in reasoning tasks.The authors frame further OOD-based approaches as a prospective route to improving LLM safety.Source: §7 · p. 9

A. Scalability to Larger Language Model

B. Generalizability Across Model Scales

C. Generalizability to Base Models

D. Generalizability to MoE Models

The section evaluates the detectors on an MoE model and reports strong results against single-sample baselines. The authors note that the dense language-modeling head keeps the methods applicable to MoE architectures.

D. Generalizability to MoE Models

  • On AQuA with Qwen-3-30B-A3B, fDBD achieves the strongest results among methods, while NCI performs strongly against single-sample baselines.The paper attributes applicability to MoE models to their dense language-modeling head; fDBD is reported with k = ALL.Source: D. Generalizability to MoE Models · p. 13, Table 10 · p. 14

E. Generalizability to Alternative Architectures

On CSQA, the OOD-inspired detectors were evaluated with Mistral-7B-Instruct-v0.3 and retained competitive performance against single-sample baselines.

E. Generalizability to Alternative Architectures

  • On CSQA, fDBD and NCI maintain competitive performance against single-sample baselines with Mistral-7B-Instruct-v0.3.The authors describe this as evidence of generalizability to an alternative architecture.Source: E. Generalizability to Alternative… · p. 14
  • Table 11 reports AUROC, where higher is better, for OOD-inspired hallucination detectors on CSQA with Mistral-7B-Instruct-v0.3.The caption says the best performance is highlighted in bold.Source: Table 11 · p. 14

F. Effect of Sample Size on Multi-sample Baselines

On AQuA, increasing SelfCheckGPT-NLI's sample size did not substantially improve its performance, and it remained below the paper's methods.

F. Effect of Sample Size on Multi-sample Baselines

  • On AQuA, SelfCheckGPT-NLI remained below the paper's methods as its sample size increased.The comparison uses Llama-3.2-3B-Instruct; the table reports AUROC, with higher values better.Source: F. Effect of Sample Size on… · p. 14, Table 12 · p. 15
  • Increasing the SelfCheckGPT-NLI sample size did not substantially improve performance in this AQuA setting.The authors report results across different sample sizes for Llama-3.2-3B-Instruct.Source: F. Effect of Sample Size on… · p. 14
  • The authors connect the difficulty of multi-sample detection on reasoning tasks to the diversity of valid reasoning paths, which complicates response-level comparisons.Source: F. Effect of Sample Size on… · p. 14

G. Implementation details for Reasoning Tasks

The implementation appendix illustrates the reasoning tasks, prompting and answer extraction procedures, format controls, and TruthfulQA judging setup used in the experiments.

G.1. Examples

G.2. Chain-of-Thought Few-shot Prompting

  • For reasoning tasks, canonical Wei et al. (2022) chain-of-thought few-shot examples elicit formatted reasoning, and correctness is determined by extracting the task-specific final answer and matching it to ground truth.Extraction uses the answer letter after “So the answer is” for CSQA, the number after “The answer is” for GSM8K, and the answer letter after “The answer is” for AQuA.Source: §G.2 · p. 16

G.3. System Prompt

  • Because few-shot prompts alone do not reliably enforce formatting for some models, system prompts reinforce the required response format and remain consistent across models.Qwen-2.5-7B-Instruct is given as an example of a model needing this reinforcement.Source: §G.3 · p. 16

G.3.1. CSQA SYSTEM PROMPT

  • The CSQA system prompt requires plain-text reasoning on one line and a final answer in the form “So the answer is (<letter>).”Source: §G.3.1 · p. 16

G.3.2. GSM8K SYSTEM PROMPT

  • The GSM8K system prompt requires one-line plain-ASCII reasoning and a final answer in the form “The answer is <number>.”Source: §G.3.2 · p. 16

G.3.3. AQUA SYSTEM PROMPT

  • The AQuA system prompt requires one-line plain-ASCII reasoning and a final answer in the form “The answer is (<letter>).”Source: §G.3.3 · p. 16

H.1. Prompt Details

  • For TruthfulQA, the few-shot prompt follows Li et al. (2023b) but omits the abstention example, and the system prompt is omitted because Llama-3.2-3B-Instruct reliably follows the format.Source: §H.1 · p. 16

H.2. LLM as Judge

  • TruthfulQA correctness is assessed with Qwen-2.5-7B-Instruct as an LLM judge, prompted to answer yes or no on factual correctness and consistency with the reference.Source: §H.2 · p. 17

I. Proof of Theorem 3.3

The appendix establishes the geometric lower bound used for decision-boundary distance, derives the relaxed decision-neutral closest point, and reports a decoded-token NCI variant with similar performance.

I. Proof of Theorem 3.3

  • Relaxing a competing token’s decision region to require only that it surpass the highest-logit token yields a hyperplane-distance lower bound for Eqn. (3).The proof identifies distance to the relaxed region’s hyperplane as the relevant Euclidean distance.Source: §I · p. 17
  • The lower bound is tight because projection onto the nearest competing-token hyperplane lies in the corresponding decision region’s closure.Source: §I · p. 17

J. Derivative of Decision-Neutral Point

  • When equal logits across the vocabulary may be infeasible, the decision-neutral closest point instead minimizes logit variance; its analytical solution follows from centering and setting the gradient to zero.Source: J. Derivative of Decision-Neutral Point · pp. 17–18

K. Decoded-token based Variant of NCI

  • Across stochastic-decoding temperatures, decoded-token-based NCI had performance similar to the max-logit-token variant.The comparison is on CSQA with Llama-3.2-3B-Instruct; Table 13 reports mean AUROC ± standard deviation over five random seeds per temperature.Source: K. Decoded-token based Variant of NCI · p. 18, Table 13 · p. 18
Loading 2602.07253v4…