PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuroActiSep: Detecting Factual Hallucinations from Feed-Forward Neurons in a Single Pass

Ali Derogar Odolou, Reza Nazari, Mostafa Salehi

arXiv:2609.14448v1cs.CLcs.AIcs.LG

TL;DR

External verification of hallucinations can add inference costs and depend on unavailable resources, motivating detection from model internals. NeuroActiSep ranks feed-forward neurons and transfers selected identities to train probes; these probes perform comparably to internal-state classifiers across evaluated QA tasks, though the findings do not establish causality.

  • Problem

    In zero-shot closed-book question answering, the task is to classify answers as truthful or hallucinated without access to a knowledge base, while external verification can add inference costs or require unavailable resources.

    Source: §I · p. 1, A. Problem Definition · p. 2

  • Method

    NeuroActiSep ranks feed-forward neurons by class-conditioned residual-space mean separation, then transfers selected identities from CounterFact to train probes on TriviaQA and TruthfulQA.

    Source: §I · p. 1, B. Neuron Selection in Language Models · p. 3

  • Results

    Across all six model–dataset combinations, NeuroActiSep ranks among the top two methods by AUROC and performs comparably to classifiers trained directly on internal states.

    Source: B. Main Results of NeuroActiSep · p. 5

  • Takeaways & Limitations

    The study demonstrates the feasibility of using selected feed-forward neurons to train classifiers for out-of-distribution factual questions.

    Source: §V · p. 6

    The findings do not establish that selected neurons causally produce or prevent hallucinations because the study did not conduct causal intervention.

    Source: §V · p. 6

Abstract

from arXiv · show

Hallucination in large language models reduces their reliability and slows adoption. Various white-box studies have used internal representations to detect patterns of truthfulness and factuality. A less-studied approach is to identify feed-forward neurons correlated with hallucination. We propose a method to rank feed-forward neurons at the final prompt token using a custom neuron selection dataset. We transfer the selected neuron identities to train hallucination classifiers on other factual question answering datasets. Our work provides empirical evidence that probes trained using the features from the selected neurons perform on par with probes trained on internal states. We also analyze the distribution of selected neurons and the effect of layer depth on detection performance.

I. INTRODUCTION

The paper asks whether feed-forward neurons at the final prompt token contain information predictive of later hallucination, and proposes ranking neurons and transferring selected identities to train classifiers. It reports comparable detection performance from selected-neuron features and residual-stream probes.

I. INTRODUCTION

  • External verification can add inference costs and require resources unavailable in closed-book settings, motivating complementary study of internal representations.Source: §I · p. 1
  • Prior internal-state probes,, including token-position probes, motivate testing whether factuality signals localize to individual neurons.Source: §I · p. 1
  • The study tests whether final-prompt-token neurons predict subsequent hallucination, rather than extracting answer spans that localize factual claims after generation.Source: §I · p. 1
  • The method ranks neurons by their contribution to the residual stream, selects them using a custom CounterFact dataset, and transfers their identities to probes for TriviaQA and TruthfulQA.Source: §I · p. 1
  • Selected-neuron probes perform comparably to probes trained on residual-stream representations.Source: §I · p. 2

II. METHODOLOGY

  • The methodology formalizes the task and defines a neuron-selection score and a classifier built from selected neurons.Source: §II · p. 2

A. Problem Definition

The task is binary factuality assessment of zero-shot, closed-book model answers without a knowledge base. Neuron selection ranks units by class-conditioned residual-space separation at the final prompt position and aggregates selections across folds.

A. Problem Definition

  • The objective is to classify each zero-shot, closed-book answer as truthful or hallucinated without access to a knowledge base.Source: A. Problem Definition · p. 2

B. Neuron Selection in Language Models

  • Each MLP neuron contributes to the MLP output in residual space, and its pre- and post-transformation activations are extracted for each token.Source: B. Neuron Selection in Language Models · p. 2
  • At the final prompt position, NeuroActiSep ranks neurons by separation between class-conditional mean contribution vectors in residual space.The score uses truthful and hallucinated labels assigned by the annotation pipeline.Source: B. Neuron Selection in Language Models · p. 3
  • Across folds, neurons are aggregated by selection votes then mean score, and the top K set transfers across datasets for the same language model.The selection dataset contains 8,000 CounterFact samples divided into three folds.Source: B. Neuron Selection in Language Models · p. 3

C. Hallucination Detection with NeuroActiSep

NeuroActiSep forms prompt-level features from transferred neurons and uses them to train hallucination classifiers. Target-dataset labels train the classifiers but do not change neuron selection.

C. Hallucination Detection with NeuroActiSep

III. DATA AND RESOURCES

The section covers the study’s models and datasets, annotation pipeline, computational resources, and hyperparameters.

III. DATA AND RESOURCES

  • The section introduces the models and datasets used in the study.Source: §III · p. 3
  • It explains the annotation pipeline used in the study.Source: §III · p. 3
  • It describes the computational resources and hyperparameters.Source: §III · p. 3

A. Datasets and Models

The experiments use three open-weight instruction-tuned models, a custom CounterFact-based neuron-selection dataset, and TriviaQA and TruthfulQA for evaluation.

A. Datasets and Models

  • Experiments use Mistral-7B-Instructv0.2, Qwen2.5-7B-Instruct, and Llama-3-8B-Instruct with greedy zero-shot decoding and each model’s standard chat template.Source: A. Datasets and Models · p. 3
  • The CounterFact-based selection dataset contains 4,000 truthful–hallucinated pairs per model, matched by relation category but drawn from different factual triplets.Refusals and unlabeled generations are excluded, and sampling is capped at 10% per category.Source: A. Datasets and Models · p. 3
  • Figure 2 shows the distribution of selected S1% neurons across layers and linear NeuroActiSep AUROC by depth cutoff on the selection dataset.Source: Figure 2 · p. 4
  • TriviaQA evaluation uses 10,000 sampled training questions and 10,000 development questions as held-out test data, while TruthfulQA uses all 817 questions.Source: A. Datasets and Models · p. 4

B. Annotation Pipeline

The annotation pipeline labels generated answers as truthful, hallucinated, refusal, or unlabeled, using answer matching and an LLM judge according to dataset.

B. Annotation Pipeline

  • For CounterFact and TriviaQA, annotation uses case-insensitive gold-answer matching, then sends unmatched responses to an LLM judge.Source: B. Annotation Pipeline · p. 4
  • TruthfulQA responses go directly to the LLM judge because truthful answers can differ substantially in wording from references.Invalid labels trigger up to five repeated evaluations; ties and cases without a valid majority are unlabeled.Source: B. Annotation Pipeline · p. 4
  • The pipeline assigns truthful, hallucinated, and refusal labels, while refusals and unlabeled samples are excluded from classifier training and evaluation.Table II reports the numbers of labels assigned to each evaluation dataset.Source: B. Annotation Pipeline · p. 4, Table II · p. 4

C. Computational Resources and Reproducibility

Experiments used fixed software, hardware, decoding, and model-training settings; dataset, prompt, and configuration materials are available in the project repository.

C. Computational Resources and Reproducibility

  • Experiments used random seed 42 on Ubuntu 22.04 with an RTX 3090, an Intel Core i7-12700K, and 32 GB of memory.Models ran in bfloat16 with greedy decoding, capped at 100 new tokens for TriviaQA and CounterFact and 256 for TruthfulQA.Source: C. Computational Resources and… · p. 4
  • Linear and MLP classifiers used class-balanced weighting; MLPs used ReLU, Adam, a 10% validation split, 20-epoch early-stopping patience, and a 500-epoch cap.Linear models used Scikit-Learn defaults with max_iter raised to 20,000.Source: C. Computational Resources and… · pp. 4–5
  • The datasets, prompts, and experiment configurations are available in the NeuroActiSep repository.Table III is identified as reporting AUROC comparisons across datasets and models, but the supplied passages do not provide its results.Source: C. Computational Resources and… · p. 5, Table III · p. 5

IV. RESULTS AND EXPERIMENTS

The results section evaluates NeuroActiSep on neuron-selection and evaluation datasets, and examines selected-neuron distributions and the role of layer depth.

IV. RESULTS AND EXPERIMENTS

  • Results are reported for hallucination detection on both the neuron-selection and evaluation datasets.The section also analyzes how selected neurons are distributed across layers and how layer depth affects detection performance.Source: §IV · p. 5
  • The section examines how chosen neurons are distributed across layers.Source: §IV · p. 5
  • The section analyzes how layer depth affects detection performance.Source: §IV · p. 5

A. Performance of Chosen Neurons on CounterFact

On CounterFact, probes using the selected 1% of neurons outperform last-layer residual-stream and MLP-block probes across all three models. Transferred neuron identities yield competitive AUROC on other factual QA datasets.

A. Performance of Chosen Neurons on CounterFact

  • On CounterFact, probes trained on the selected 1% of neurons outperform final-prompt last-layer residual-stream and MLP-block probes across all three models.The results motivate aggregating selected neurons into set S for evaluation on other factual QA datasets.Source: A. Performance of Chosen Neurons on… · p. 5

B. Main Results of NeuroActiSep

C. Distribution of Chosen Neurons

Depth-restricted results suggest predictive information is limited in lower layers and becomes more accessible as middle layers are included. The study demonstrates predictive associations, not causal effects of selected neurons.

C. Distribution of Chosen Neurons

  • Most selected neurons come from upper layers; Mistral and Llama selections begin increasing around middle layers, whereas Qwen selections concentrate considerably in upper layers.Layer concentration alone does not establish that those layers are necessary for prediction.Source: C. Distribution of Chosen Neurons · p. 6

D. Analysis Across Layer Depth

  • Predictive information appears limited in lower layers and grows more accessible as middle layers enter the eligible set, while plateaus suggest much of the signal is available before the final layers.The fixed-1%-per-layer strategy remains competitive despite a smaller intermediate-depth feature budget; later-layer neurons may encode overlapping predictive information.Source: D. Analysis Across Layer Depth · p. 6

V. CONCLUSION

  • The study finds feed-forward neurons correlated with factual hallucinations and uses them to train classifiers on out-of-distribution factual questions, but does not establish a causal relationship.No causal intervention was conducted; the authors call for further mechanistic studies.Source: §V · p. 6
Loading 2609.14448v1…