PaperlayerView exact revision on arXiv

Source-linked AI summary

Steer LLM Latents for Hallucination Detection

Seongheon Park, Xuefeng Du, Min-Hsuan Yeh, Haobo Wang, Yixuan Li

arXiv:2503.01917v2cs.LGcs.AIcs.CL

TL;DR

Hallucination detectors based on pretrained representations may not separate truthful from hallucinated content because those representations favor linguistic coherence. The paper introduces TSV, an inference-time steering vector trained with labeled exemplars and augmented using unlabeled generations, and reports strong performance with minimal labeling across datasets.

  • Problem

    Pretrained LLM representations optimized for linguistic coherence may fail to clearly separate truthful and hallucinated content, while hallucination detection requires reliable factuality signals.

  • Method

    TSV reshapes intermediate latent states during inference without changing model parameters, using a two-stage process with labeled exemplars, optimal-transport pseudo-labeling, and confidence-based selection.

  • Results

    12.8% improvement in hallucination detection accuracy (AUROC) is reported on TruthfulQA versus state-of-the-art methods, with 84.2% versus 85.5% for a fully supervised upper bound using 32 examples.

  • Takeaways & Limitations

    TSV provides a lightweight approach for hallucination detection that achieves strong performance with limited human labeling and generalizes across datasets.

  • Takeaways & Limitations

    A small labeled exemplar set may not fully capture the diversity of truthful and hallucinated data distributions, motivating augmentation with unlabeled training data.

Abstract

from arXiv · show

Hallucinations in LLMs pose a significant concern to their safe deployment in real-world applications. Recent approaches have leveraged the latent space of LLMs for hallucination detection, but their embeddings, optimized for linguistic coherence rather than factual accuracy, often fail to clearly separate truthful and hallucinated content. To this end, we propose the Truthfulness Separator Vector (TSV), a lightweight and flexible steering vector that reshapes the LLM's representation space during inference to enhance the separation between truthful and hallucinated outputs, without altering model parameters. Our two-stage framework first trains TSV on a small set of labeled exemplars to form compact and well-separated clusters. It then augments the exemplar set with unlabeled LLM generations, employing an optimal transport-based algorithm for pseudo-labeling combined with a confidence-based filtering process. Extensive experiments demonstrate that TSV achieves state-of-the-art performance with minimal labeled data, exhibiting strong generalization across datasets and providing a practical solution for real-world LLM applications.

1. Introduction

The paper addresses hallucination detection by reshaping LLM representations rather than changing model parameters. TSV uses few labeled examples and unlabeled generations to improve separation and achieve strong cross-dataset performance.

  • LLM hallucinations can appear plausible but factually inaccurate, undermining trust and creating risks in high-stakes applications.
  • Pre-trained LLM embeddings often fail to clearly separate truthful and hallucinated content because they prioritize linguistic coherence over factual accuracy.The paper illustrates this overlap with TruthfulQA embeddings.
  • TSV is a lightweight inference-time vector that reshapes latent representations without modifying model parameters.It targets hallucination classification rather than mitigation of hallucinated generations.
  • The two-stage framework first trains TSV on a small labeled exemplar set, then augments it with selected unlabeled generations using pseudo-labeling.The initial objective encourages compact clusters around truthful and hallucinated class prototypes.
  • 12.8% improvement in hallucination detection accuracy (AUROC) is reported on TruthfulQA versus state-of-the-art methods.With 32 labeled examples, TSV reaches 84.2% versus 85.5% for the fully supervised upper bound and generalizes across unseen datasets.
  • The paper reports TSV as a lightweight approach that achieves strong performance with limited human labeling and supports evaluation across larger models and diverse datasets.Ablation studies examine design choices and scalability.

2. Related Works

Prior hallucination-detection methods use uncertainty scores, supervised classifiers, or activation interventions. The paper positions TSV as a detection-specific steering vector designed to separate representations with minimal human supervision.

  • Existing hallucination detectors estimate uncertainty using token probabilities, verbalized confidence, or consistency across evaluations.
  • Supervised latent-state classifiers assume truthfulness is encoded internally but require extensive labeled data.The paper contrasts this requirement with minimal-supervision detection.
  • Activation-engineering studies steer generation toward truthful behavior, while representation fine-tuning learns interventions on hidden-state subspaces.
  • TSV learns a steering vector specifically for separating truthful and hallucinated representations rather than mitigating hallucinated generations.Its stated distinction is both task-specific and supervision-efficient.
  • The framework diagram depicts exemplar training followed by pseudo-labeling, confidence selection, exemplar augmentation, and TSV retraining.

3. Problem Setup

The problem setup formalizes hallucination detection as binary prediction over prompts and generations, using a small labeled exemplar set alongside unlabeled generations collected from deployed LLMs.

  • Hallucination detection learns a binary predictor over input prompts and their corresponding model generations.
  • The prompt and generated response are treated as an ordered concatenated sequence for detection.
  • Unlabeled generations collected through user interactions contain a mixture of truthful and hallucinated content.The mixture includes an unknown hallucination fraction.
  • A labeled exemplar set consists of prompt-response pairs annotated as truthful or hallucinated.
  • The setup allows the labeled exemplar count to remain small, with 32 examples cited as a practical target for reducing annotation costs.

4. Method

TSV learns a steering vector that reshapes LLM representations for hallucination detection without changing model parameters. Its augmented training pipeline assigns pseudo-labels to unlabeled data with optimal transport, filters uncertain samples, and then uses them to refine the detector.

  • TSV intervention: TSV adds a trainable vector to intermediate latent states, influencing subsequent representations while preserving the option to restore original generation by removing it.The steering strength is controlled by λ, and the vector is shared across token positions.
  • Initial training: The initial objective learns TSV to separate truthful and hallucinated embeddings into compact class-specific clusters around learned prototypes.The model uses maximum likelihood over exemplars and a hyperspherical distribution for normalized final-layer embeddings.
  • Augmented training: Optimal transport assigns pseudo-label probabilities to unlabeled embeddings while matching the expected class proportions and minimizing transport costs.Entropy regularization enables the Sinkhorn algorithm, and the exemplar-set class distribution serves as a proxy for the target proportions under an MCAR assumption.
  • Augmented training: Confidence filtering selects unlabeled samples with the lowest predictive uncertainty before adding their pseudo-labeled examples to the training set.The augmented learning process repeats the TSV training procedure until convergence.
  • Inference: During inference, the detector scores a test embedding by its normalized probability of belonging to the truthful class and thresholds that score for classification.The learned class prototypes provide the basis for the truthfulness score.

5. Experiments

Experiments evaluate TSV across multiple models, datasets, baselines, ablations, and distribution shifts. Results show strong detection performance, label efficiency, transferability, and the importance of shaping representations.

  • Main results: TSV achieves state-of-the-art performance across LLaMA-3.1-8b and Qwen-2.5-7b, outperforming competing hallucination detection methods across four datasets.Unsupervised baselines show inconsistent performance across models and data distributions, whereas TSV is robust across both models and all datasets.
  • Main results: 13.6%: TSV outperforms HaloScope on TruthfulQA with LLaMA-3.1-8b while using the same validation set and unlabeled data.TSV shapes the latent space using a small exemplar set, whereas HaloScope relies on default LLM embeddings.
  • Label efficiency: 6.0%: TSV with only 32 labeled examples outperforms the fully supervised SAPLMA baseline on TruthfulQA.The result emphasizes TSV’s label efficiency and the role of shaping the latent space.
  • Label efficiency: 84.2% vs. 85.5%: with 32 labeled examples, TSV nearly matches the fully supervised oracle’s TruthfulQA AUROC.The comparison uses LLaMA-3.1-8b and the AUROC metric on the same test set.
  • Generalization: TSV transfers across datasets, reaching 79.8% AUROC on TriviaQA when learned from TruthfulQA, versus 84.0% when learned directly from TriviaQA.The evaluation applies TSV learned on an in-distribution source dataset directly to an out-of-distribution target dataset.
  • Ablation studies: Ablations show that TSV raises TruthfulQA AUROC by 28.7%, augmented training adds 3.3%, and the full approach performs best across datasets.The augmented phase improves performance only when supported by structured representations and accurate pseudo-labels.

6. Conclusion

TSV reshapes the LLM latent space during inference to separate truthful from hallucinated outputs without changing model parameters. A two-stage framework combines limited labeled data with unlabeled generations to achieve strong performance and generalization.

  • TSV is a lightweight, modular approach that reshapes latent representations during inference without altering model parameters.
  • The two-stage framework combines a small labeled exemplar set with unlabeled LLM generations to reduce reliance on human labeling and computational cost.
  • Experiments report state-of-the-art hallucination-detection accuracy with strong generalization across datasets.

Impact Statement

The paper frames TSV as a response to costly truthfulness annotation by learning from a small labeled exemplar set and augmenting it with pseudo-labeled unlabeled generations.

  • Large-scale human truthfulness annotations are costly and time-intensive, motivating a two-stage training framework.
  • Initial training uses labeled exemplars to learn TSV and class prototypes for truthful and hallucinated-generation clusters.
  • The augmentation stage uses Sinkhorn optimal transport to pseudo-label unlabeled data, selects confident samples, and expands the exemplar set.

B.1. Implementation details (ours)

The implementation evaluates TSV across multiple QA datasets, model scales, labeling schemes, and class-distribution settings, using fixed generation and training configurations. It also examines scalability, legacy-model performance, and alternative truthfulness labels.

  • Implementation details: Responses are generated with beam search using five beams, and TSV is trained for 20 initial plus 20 augmented epochs.
  • Implementation details: TSV is applied to the residual stream at dataset-selected steering layers, with hyperparameters selected on TruthfulQA using LLaMA-3.1-8b.
  • Datasets and evaluation: Evaluation covers TruthfulQA, TriviaQA, SciQ, and NQ Open, including TruthfulQA’s 817 question-answer pairs.
  • Scalability: Larger-model experiments apply TSV to LLaMA-3.1-70b and Qwen-2.5-14b and report consistent outperformance of four strong baselines.
  • Labeling: Truthfulness labels are obtained from BLEURT thresholding or GPT-4o semantic-equivalence judgments, with “Yes” treated as truthful.
  • Ablations: The study compares optimal-transport class distributions using uniform, estimated, oracle, and exemplar-derived alternatives.

E.5. Robustness to pseudo-label noise

The robustness analyses test TSV under pseudo-label noise and mismatch between assumed and actual class distributions. Performance decreases with noise but remains relatively robust at the tested level.

  • Pseudo-label noise: Under 25% pseudo-label noise, AUROC decreases modestly from 84.2% to 81.3% on LLaMA-3.1-8b.
  • Pseudo-label noise: The noise experiment uses selected TruthfulQA unlabeled examples and progressively flips correct pseudo-labels.
  • Class-distribution mismatch: Class-distribution robustness is evaluated with aligned, uniform, and reversed exemplar distributions relative to unlabeled data.

F. Qualitative Results

The method’s truthfulness scores align with ground-truth labels across TruthfulQA and TriviaQA examples, while the model’s final-layer embeddings are analyzed through their norms and hyperspherical structure.

  • Truthfulness scores S(x′) align with ground-truth truthful and hallucinated labels across qualitative TruthfulQA and TriviaQA examples.Green checkmarks denote truthful ground truth, while red crosses denote hallucinated ground truth.
  • The qualitative evaluation uses questions from TruthfulQA and TriviaQA with responses generated by LLaMA-3.1-8b.
  • Final-layer last-token embeddings are modeled with a unit-norm hyperspherical distribution because RMSNorm commonly keeps embedding norms consistent while directions vary.The von Mises–Fisher distribution is used in the maximum-likelihood objective.
  • Figures 7 and 8 report L2 norms of final-layer last-token embeddings for LLaMA-3.1-8b and Qwen-2.5-7b, respectively.
  • Experiments used Python 3.8.15, PyTorch 2.3.1, NVIDIA A100 GPUs, and the OpenAI API for GPT-4o evaluation.

H.2. Training and inference time

TSV has low tracked training and inference cost, with runtimes compared against other hallucination-detection methods under the same software and hardware setup.

  • Approximately 0.1 GPU-hours are required for LLaMA-3.1-8b and Qwen-2.5-7b, 0.2 GPU-hours for Qwen-2.5-14b, and 1 GPU-hour for LLaMA-3.1-70b.
  • Wall-clock time is compared on the same TruthfulQA split for HaloScope, SAPLMA, TSV, and Semantic Entropy.HaloScope, SAPLMA, and TSV require training, whereas Semantic Entropy is training-free.
  • All compared methods use the same software and hardware setup for runtime measurement.

I. Limitations and Future Work

The paper identifies limitations in evaluation granularity and answer length: TSV targets sentence-level detection in short-form QA, leaving finer-grained and long-form settings for future work.

  • Figure 9 reports AUROC and wall-clock time for training and inference.
  • Fine-grained hallucination detection: TSV focuses on sentence-level hallucination detection, whereas practical applications may require token- or phrase-level hallucinated-span identification.
  • Fine-grained hallucination detection: Adapting TSV to hidden states across token positions is proposed as a way to support finer-grained hallucination localization.One suggested direction applies TSV before and after salient entity spans and measures representation or score shifts.
  • Long-form QA: The work focuses on short-form QA, while long-form answers remain a practical setting outside the current evaluation.The paper suggests decomposing long-form generation into multiple short QA pairs for individual verification.
Loading 2503.01917v2…