Source-linked AI summary
Hallucination Neurons and Where to Find Them: An Investigation into the existence of Hallucination Neurons
Huseyin Cavus, Sebin Sabu, Joshua Spear, Jaskaran Singh Kawatra, Pavithra Rajendran
TL;DR
Sparse-neuron localization claims are increasingly evaluated through predictive performance, despite known concerns about L1 probing with correlated features and limited tests of stability, uniqueness, and generality. This paper applies a five-step diagnostic protocol to H-Neurons across models and datasets. It finds strong detection and significant causal effects, but also non-unique selections and only partial sharing across settings.
Problem
Sparse-neuron localization claims are often evaluated mainly through predictive performance rather than stability, uniqueness, and generality, despite known vulnerabilities of L1 probing in correlated feature spaces.
Source: §1 · p. 2, §6 · p. 12Method
The paper proposes a five-step diagnostic protocol and applies it to H-Neurons using stricter response-level labels and cross-setting analyses.
Source: §1 · p. 2, §3.1 · p. 4, §6 · p. 12Results
Sparse probing identifies neuron sets with strong detection performance and significant causal effects beyond random same-layer baselines, but the sets are not uniquely localized and only partially shared across settings.
Source: §6 · p. 12Takeaways & Limitations
Detection and localization should be treated as distinct evaluation targets, with diagnostic validation included in sparse-neuron interpretability research.
Source: §6 · p. 13The evaluation covers only two models, both in the Gemma 3 4B family, so cross-architecture generality remains untested.
Source: §5.3 · p. 12
Abstract
from arXiv · showhide
Interpretable machine learning for Large Language Models (LLMs) increasingly relies on sparse probing methods that identify small sets of neurons claimed to detect and causally influence behaviors such as factuality recall, safety alignment, and hallucination. These claims have important implications for model auditing and behavioral steering, yet they are rarely tested against known failure modes of $L_1$-regularized probing in correlated, high-dimensional feature spaces. We propose a five-step diagnostic protocol covering feature correlation, bootstrap stability, sparse versus dense ranking disagreement, intervention baselines, and cross-dataset evaluation as a minimum standard for sparse-neuron localization claims. We investigate prior work using our proposed approach, specifically on H-neurons using open-source LLMs across TriviaQA, BioASQ, and NQ-Open datasets. Our results demonstrate detection replicates across both models and datasets, and exceeds the original reported AUROC gaps for TriviaQA and BioASQ datasets. Gemma 3 4B consistently outperforms MedGemma 4B on matched datasets, with AUROC gaps of +0.311 versus +0.235 on TriviaQA, +0.474 versus +0.455 on BioASQ, and +0.128 versus +0.112 on NQ-Open respectively. Causal validation at $n = 500$ with five random seeds shows statistically significant effects beyond random same-layer baselines. At the same time, the diagnostic results indicate that the selected neurons are not uniquely localized. Across the three Gemma 3 4B settings, 19 of 22 selected H-Neurons have Pearson $|r| > 0.7$ with other features, bootstrap selections show only moderate stability, and sparse and dense rankings overlap only weakly. Our findings show that sparse predictive structure can coexist with non-unique neuron selection. Routine diagnostic validation is necessary to distinguish detection claims from localization claims in mechanistic interpretability.
1 Introduction
Sparse-neuron localization claims promise useful model auditing and steering, but L1 probing can select arbitrary representatives from correlated features rather than uniquely informative units. This paper evaluates H-Neurons with a five-step diagnostic protocol, finding detection and causal effects alongside evidence that the selected units are not uniquely localized.
1 Introduction
- L1 probing can select arbitrary representatives from correlated features, so sparse predictive selections need not be uniquely localized.Earlier work identified direct neuron-set overlap across tasks or datasets as a localization diagnostic absent from the original H-Neurons report.Source: §1 · p. 2
- The five-step protocol evaluates H-Neurons [4] as a case study and is proposed as a minimum standard for sparse-neuron localization claims.The study tests the protocol across two open-source LLMs and three question-answering datasets.Source: §1 · p. 2
- The evaluation replicates and exceeds reported detection performance, and targeted H-Neuron interventions produce statistically significant causal shifts beyond random baselines.The findings refine rather than refute the original H-Neuron claims.Source: §1 · p. 2
- Identified units are drawn from highly correlated clusters and show only moderate bootstrap stability, qualifying the localization claim.The study also reports weak sparse-versus-dense ranking agreement.Source: §1 · p. 2
- Some feedforward units, especially (L16, N4146), persist across dataset and model shifts, while no single neuron is universal.The pattern suggests a partially shared core mechanism alongside substantial domain-specific structure.Source: §1 · p. 2
- Sparse-probing methods can yield selections that are predictively useful and causally effective without meeting stronger criteria for unique localization.The paper proposes diagnostic validation as a routine part of future sparse-neuron interpretability work.Source: §1 · pp. 2–3
2 Related Work
Prior work has reported sparse neuron sets associated with skills, safety, knowledge, and hallucination, while overlap analyses and probing critiques raise questions about how localization claims should be validated. This paper situates H-Neurons within those lines of work and addresses the lack of direct neuron-set overlap measurement in the original report.
2.1 Sparse-Neuron Localization in LLMs
- Skill, safety, and knowledge neuron studies report behavior-linked units and use interventions or overlap analyses to support localization, though knowledge-neuron interventions have also been questioned.Gurnee et al. formalized k-sparse linear probing while cautioning that conclusive proofs of monosemanticity remain methodologically out of reach.Source: §2.1 · p. 3
2.2 Hallucination in LLMs
- Hallucination research includes black-box uncertainty estimates and hidden-representation detection; H-Neurons [4] extend this line with a neuron-level localization claim.The H-Neurons account frames a unified over-compliance signal across hallucination, sycophancy, and jailbreak susceptibility.Source: §2.2 · p. 3
2.3 Cross-Task and Cross-Domain Neuron Overlap as Validation
- Prior studies use direct neuron-set overlap across tasks or domains as evidence for localization, but the original H-Neurons report measures classifier transfer rather than neuron-set overlap.Wang et al., Chen et al. [2], and Leng and Xiong [25] are cited as examples of overlap-based analyses.Source: §2.3 · pp. 3–4
2.4 Critiques of Probing Methodology
- Probes can score highly on random control tasks, and L1 regularization may select arbitrarily among correlated features, motivating diagnostics beyond predictive performance alone.Zou and Hastie show Lasso selection among correlated groups can be arbitrary; the interpretability consequences have not been systematically diagnosed in recent neuron-localization work.Source: §2.4 · p. 4
3 Method
The method identifies H-Neurons from response-level labels and feedforward activation features using sparse logistic regression, then applies diagnostics for correlation, stability, ranking disagreement, intervention controls, and cross-setting overlap. Experiments use two Gemma-family models, three QA datasets, and specified resampling and validation settings.
3.1 H-Neuron Identification Pipeline
- A stricter response-level labeling adaptation retains only responses unanimously faithful or hallucinated across ten generations, yielding fewer but cleaner probe-training samples.Intermediate cases with one to nine correct responses are excluded.Source: §3.1 · p. 4
- The identification pipeline extracts feedforward activations, computes CETT scores, aggregates neuron scores into feature vectors, and selects nonzero L1 logistic-regression coefficients as probable H-Neurons.For Gemma 3 4B, the feature space contains 348,160 features.Source: §3.1 · p. 5
3.2 The Five-Step Diagnostic Protocol
- The five diagnostics test whether selected neurons are unique, stable, causally privileged, and shared across datasets or models.The protocol is designed to require at most one additional probe-fitting step beyond standard identification.Source: §3.2 · p. 5
- Feature-correlation, bootstrap-stability, and L1-versus-L2 ranking checks flag localization concerns when correlated alternatives, low Jaccard stability, or low ranking overlap are found.The protocol flags correlation above 0.7, mean bootstrap Jaccard below 0.7, and L1/L2 overlap below 50%.Source: §3.2 · p. 5
- Intervention effects are compared with random neurons from the same layers, while cross-dataset and cross-model neuron-set overlap is tested against random-selection expectations.Interventions scale target-neuron down_proj input activations and assess significance with McNemar’s test; a correlated-cluster intervention baseline remains future work.Source: §3.2 · pp. 5–6
3.3 Experimental Setup
- Experiments use two 34-layer Gemma-family models and TriviaQA, BioASQ, and NQ-Open, with 2,000 samples per dataset.Identification uses L1 logistic regression with C = 1.0; causal validation uses 500 held-out samples per condition and five seeds, and bootstrap diagnostics use five resamples.Source: §3.3 · p. 6
- Diagnostics and cross-dataset overlap for TriviaQA use an independent Gemma 3 4B replicate to check whether findings depend on one experimental run.The replicate selected 10 H-Neurons and had AUROC 0.820 with gap +0.404.Source: §3.3 · p. 6
4 Results
Detection replicates across model-dataset settings, and targeted suppression significantly reduces judge accuracy beyond random same-layer baselines. Yet diagnostics show that sparse H-Neuron selections are not uniquely localized, even as probes transfer across datasets and some neurons recur across models and domains.
4.1 Detection
- Detection gaps are positive across all six model-dataset combinations, with BioASQ strongest and NQ-Open weaker (gap +0.128).The pattern holds across two checkpoints; the weaker NQ-Open result is attributed tentatively to greater response variance and a sparser supervisable signal.Source: §4.1 · p. 6, Table 1 · p. 7
4.2 Causal Validation
- Targeted H-Neuron suppression significantly decreases judge accuracy beyond random same-layer baselines on both tested conditions.The effect is statistically significant at p < 0.05; magnitudes are approximately 2.4 percentage points on TriviaQA and 0.9 on BioASQ.Source: Table 2 · p. 7, §4.2 · p. 7
4.3 Within-Dataset Diagnostics (D1–D3)
- 19 of 22 selected H-Neurons have Pearson |r| > 0.7 with other features, while bootstrap Jaccard is 0.45–0.69 and L1–L2 overlap is 0–25%.These diagnostics indicate selections from correlated clusters rather than unique localization; bootstrap stability remains below the suggested 0.7 threshold.Source: §4.3 · pp. 7–8, Table 3 · p. 8
4.4 Cross-Dataset and Cross-Model Overlap (D5)
- Three neurons are shared between Gemma 3 4B on TriviaQA and MedGemma 4B on BioASQ (Jaccard 0.231, p = 4.2 × 10−13).The overlap is evaluated against random selection from a feature pool of D = 348,160.Source: §4.4 · p. 8, Table 4 · p. 9
4.5 Cross-Dataset Classifier Transfer
- All six cross-dataset transfers have positive AUROC gaps, although transfer strength varies by source probe and target dataset.The TriviaQA probe transfers to BioASQ at 0.868 and NQ-Open at 0.721; the BioASQ probe has weaker outbound AUROCs of 0.626 and 0.646.Source: §4.5 · pp. 8–9, Table 5 · p. 9
4.6 Cross-Model Evaluation
- Gemma 3 4B outperforms MedGemma 4B on all matched datasets, while no H-Neuron is shared across all three Gemma QA datasets.BioASQ has the strongest detection for both models, and neuron overlap is partial: TriviaQA and BioASQ share four units, whereas BioASQ and NQ-Open share none.Source: Table 6 · p. 10, §4.6 · p. 10
4.7 Candidate Cross-Domain Neurons
- (L16, N4146) recurs in both the Gemma TriviaQA–BioASQ and Gemma TriviaQA–MedGemma BioASQ intersections, making it the strongest cross-domain candidate.The authors do not claim these units uniquely encode hallucination; they propose them as falsifiable targets for single-neuron ablation and SAE decomposition.Source: §4.7 · pp. 10–11
5 Discussion
H-Neuron detection and population-level causal control are supported, but the selected units are not uniquely localized. The findings motivate separating detection from localization, testing overlap directly, and treating capability preservation as an important intervention follow-up.
5.1 What the Diagnostic Reveals About H-Neurons
- Detection and population-level causal effects hold, but correlated clusters, moderate bootstrap stability, and sparse–dense ranking disagreement do not support unique localization.D5 also finds partially domain-specific selections, with no universal neurons across the three QA datasets.Source: §5.1 · p. 11
- A sparse L1 fit may identify a predictive projection of a larger correlated feedforward structure rather than the only units carrying hallucination-relevant signal.The paper relates this interpretation to feature superposition and the layer-wise distribution of context features.Source: §5.1 · p. 11
5.2 Generalizable Methodological Takeaways
- Predictive sparsity and mechanistic localization require separate evaluation because a sparse selection can be highly predictive yet non-unique.Conflating the two risks claims stronger than the evidence supports.Source: §5.2 · p. 11
- Classifier-level transfer is insufficient to establish mechanism sharing; direct neuron-set overlap tests the stronger claim that identified units are shared.A probe may transfer through distributed representation signal even when the neurons selected differ.Source: §5.2 · pp. 11–12
5.3 Limitations
- The study evaluates only two models, both in the Gemma 3 4B family, so cross-architecture generality remains untested.The authors also did not test capability preservation after intervention, leaving general-capability degradation as an alternative explanation for causal effects.Source: §5.3 · p. 12
- The protocol’s diagnostic thresholds are empirical guidelines, not principled bounds, and the authors recommend systematic study to develop them.The listed thresholds are |r| > 0.7, Jaccard > 0.7, and overlap > 0.5.Source: §5.3 · p. 12
6 Conclusion
The paper proposes a five-step diagnostic protocol and applies it to H-Neurons, finding strong detection and significant intervention effects beyond random same-layer baselines. However, the selected units are not uniquely localized, and overlap analyses find only partial sharing across datasets and models.
6 Conclusion
- The paper proposes a five-step protocol for evaluating sparse-neuron localization claims, which are often assessed mainly through predictive performance rather than stability, uniqueness, and generality.The protocol is applied to H-Neurons for hallucination in LLMs.Source: §6 · p. 12
- Sparse probing over CETT features identifies predictive neuron sets, and targeted interventions produce statistically significant causal effects beyond random same-layer baselines.The identified sets are embedded in correlated clusters, show moderate bootstrap stability, and disagree substantially with dense rankings.Source: §6 · p. 12
- Cross-dataset and cross-model analyses find no universal neuron, while (L16, N4146) recurs across dataset and model shifts.The authors interpret this as partial sharing alongside substantial domain-specific structure.Source: §6 · p. 12
- Predictive sparsity and population-level causal effects do not by themselves establish unique mechanistic localization.The authors argue for routine diagnostic validation and separate evaluation targets for detection and localization.Source: §6 · pp. 12–13