The Hallucination Signal Is a Mean Shift: Why Simple Probes Suffice

• The central question is whether the hallucination signal’s geometry necessitates increasingly complex probe architectures.Source: §1 · p. 1
• Across three 7B-scale models and three datasets, the analysis operates within the paired-example paradigm to isolate representational geometry from generation-induced distribution shifts.Source: §1 · pp. 1–2
• The hallucination detection signal is overwhelmingly dominated by a single mean-shift component.Source: §1 · pp. 1–2
• About 73% of the Fisher LDA gap is closed by shrinkage LDA, indicating that apparent complexity largely reflects covariance estimation difficulty rather than exploitable non-linearity.Source: §1 · p. 2
• L2-regularized logistic regression achieves 0.952 AUROC and strictly upper-bounds 12 controlled architectural alternatives.Source: §1 · pp. 1–2
• LayerMix aggregates the informative layer band without oracle access, matching oracle performance with about 35 seconds of computational overhead.Source: §1 · pp. 1–2

• Training-free detectors include logit contrasting, covariance eigenvalues, semantic entropy, and sampling consistency, motivating the paper’s focus on supervised probing.Source: §2.1 · p. 2
• Recent probing methods add attention-gated SVD, cross-layer tracking, and optimal-transport pseudo-labeling without characterizing the single-layer geometry they target.Source: §2.1 · p. 2
• SEP uses multiple sampled generations in an open-ended QA paradigm, which differs fundamentally from the paired-example setting and precludes direct head-to-head comparison.Source: §2.1 · p. 2
• The paper extends prior linear-representation work by decomposing hallucination detection into mean-shift and residual components and quantifying the Fisher LDA gap.Source: §2.2 · p. 2
• Cross-validation-based layer aggregation addresses the previously unquantified cost of relying on non-oracle layer heuristics.Source: §2.2 · p. 2
• LayerMix selects a layer band by cross-validation and uniformly averages independent per-layer probes, contrasting with CLAP’s input-dependent cross-layer attention.Source: §2.3 · p. 3
• Prior work reports paradigm-dependent differences in free-form generation, token-level detection, and cross-dataset generalization.Source: §2.3 · p. 3

3.1 The Signal is a Mean Shift

• The mean-shift direction δ is defined as the difference between hallucinated and factual class centroids.Source: §3 · p. 3
• Removing δ reduces mean detection to chance at 0.499 across nine conditions.Source: §3.1 · p. 3

3.2 Mean Shift is Necessary but Insufficient

• The 1D projection onto δ achieves 0.834 AUROC, recovering 78% of unregularized LR’s above-chance performance.Source: §3.1 · p. 3

3.1 The Signal is a Mean Shift

• Cohen’s d along δ ranges from 1.2–1.6 despite accounting for less than 3% of total variance, while residual orthogonal distributions overlap.Source: §3.1 · p. 3

3.2 Mean Shift is Necessary but Insufficient

• Shrinkage LDA reaches 0.920 mean AUROC and closes at least 73% of the gap, whereas ZCA whitening reduces performance to 0.790 versus 0.952 for raw L2-LR.Source: §3.2 · p. 4
• Normality tests reject Gaussianity in both classes across all nine conditions, consistent with residual distributional structure beyond LDA.Source: §3.2 · p. 4

3.3 Signal Distribution Across Layers and Neurons

• The optimal layer varies across models, while adjacent layers retain high scores and form a model-specific contiguous signal band.Source: §3.3 · p. 4
• Two hundred neurons recover 98.3% of full AUROC, but the top-20 neurons share only four across folds, indicating distributed rather than fixed-neuron signal.Source: §3.3 · p. 4

• LayerMix identifies the informative layer band through cross-validation and aggregates its per-layer predictions.Source: §4 · pp. 4–5
• Layer-wise AUROC is reported across three models and three datasets, with shaded bands showing ±1 standard deviation across CV folds and dots marking optimal layers.Source: Figure 2 · p. 5

Stage 1: Layer Scoring

• Stage 1 scores every layer with L2-regularized logistic regression using stratified five-fold CV and mean AUROC.Source: §4 · p. 5

Stage 2: Layer Selection

• Stage 2 selects the top K layers, using K=5 by default; selected layers typically form a contiguous block around the informative region.Source: §4 · p. 5

Stage 3: Aggregation

• Stage 3 trains an independent L2-regularized logistic regression probe on each selected layer and averages their predictions.Source: §4 · p. 5

Design choices

• Full-dimensional per-layer probes avoid dimensionality reduction, while prediction averaging provides ensemble-style variance reduction without scaling feature dimension with K.Source: §4 · p. 5

Computational cost

• LayerMix extracts all layers in one scoring forward pass and adds about 35 seconds of overhead for 5,000 examples on one GPU.Source: §4 · p. 5

Models & Datasets

• Experiments use Llama-3.1-8B, Mistral-7B, and Qwen2.5-7B on TruthfulQA, HaluEval-Dialogue, and FaithDial.The datasets contain 4,135, 20,000, and 5,848 examples, respectively.Source: §5 · p. 6

Baselines

• Comparisons span training-free, unsupervised or semi-supervised, cross-layer, and subspace detection methods.The baseline set includes perplexity, entropy, self-evaluation, HaloScope, CCS, TSV, ICR Probe, SVD, MeanDiff, PLS-DA, and gcPCA.Source: §5 · p. 6

Protocol

• 5-fold stratified cross-validation evaluates methods on last-token hidden states within the paired-example paradigm.LayerMix performs nested 5-fold cross-validation inside training folds, and all methods use training-data StandardScaler.Source: §5 · p. 6

Main Comparison

• 0.954 mean AUROC lets LayerMix match the single-layer oracle’s 0.952 across nine conditions without post-hoc oracle layer access.LayerMix captures the distributed mean-shift signal through multi-layer aggregation.Source: §6.1 · p. 6
• 0.013 AUROC is the statistically significant gap LayerMix closes over the 0.941 middle-layer heuristic in 8/9 conditions.At recall 0.90, the improvement corresponds to a 15–20% relative reduction in false positives.Source: §6.1 · p. 6
• 0.954 versus 0.928 mean AUROC gives LayerMix an advantage over CLAP in all 9 conditions, while two-component PLS-DA reaches 0.910.All-layer averaging reaches 0.944, and PLS-DA outperforms SVD among single-layer subspace methods, 0.941 versus 0.929.Source: §6.1 · p. 6

Fairness Ablation

• Matched L2 regularization improves SEP from 0.889 to 0.921 and oracle LR from 0.935 to 0.957, while LayerMix reaches 0.960.The comparison uses instruct models and C=0.001.Source: §6.1 · p. 6, Table 4 · p. 7

Ablation Studies

• CV-based layer selection is the only strategy that consistently matches or outperforms the single-layer oracle, unlike fixed heuristics or Cohen’s d rankings.Cohen’s d selects scattered layers rather than the optimal contiguous band.Source: §6.2 · p. 7

Robustness and Efficiency

• Differences of ≤0.002 AUROC across K ∈{3, 5, 7} support K=5 as the default, while label efficiency favors PLS-DA at N≤200 and full-dimensional LR at N≥500.Nested cross-validation finds optimistic bias ≤0.010, and random-label controls yield 0.500 ± 0.003 AUROC.Source: §6.2 · p. 7

Geometric Ablation

• An MLP differs from L2-LR by |∆|≤0.002 AUROC across nine conditions, while SAPLMA’s 2-layer MLP reaches 0.919 versus LR’s 0.952.Within the paired-example paradigm, the discriminative boundary is overwhelmingly linear.Source: §6.3 · p. 7

Scaling

• Across 25 models from 0.5B to 70B, AUROC increases monotonically from 0.86–0.91 for models below 2B toward about 0.98 for Llama-3.1-70B.LayerMix tracks or exceeds the oracle in 72 of 75 conditions, or 96.0%.Source: §6.5 · p. 8, Figure 3 · p. 8

Scope and Predictions

• Across all 12 tested alternatives, the prediction that discarding raw hidden states or generic variance-maximizing subspaces loses mean-shift information holds within the paired-example paradigm.The Fisher LDA gap is attributed primarily to covariance estimation difficulty, with shrinkage LDA closing about 73% of it.Source: §7 · p. 8

Implications for Method Design

• LayerMix exploits a contiguous informative layer band, while L2-regularized LR provides a baseline future architectural proposals must match within this paradigm.Prediction averaging succeeds in 9/9 conditions.Source: §7 · p. 8

When to Use Probing-Based Detection

• 0.941 AUROC makes middle-layer LR a zero-overhead default, while LayerMix adds 0.013 AUROC at about 35 seconds when maximal accuracy is required.Probing-based detection requires white-box access and labeled data.Source: §7 · p. 8

Is the Mean Shift an Artifact?

• Length-only AUROC is ≤0.610, regressing out length changes AUROC by ≤0.003, and removing δ collapses detection to chance across all nine conditions.The authors note that confounds within the paired-example paradigm cannot be definitively excluded without free-form generation tests.Source: §7 · pp. 8–9

Domain-Specific Geometry and Adaptation

• Mean-shift directions have cosine about 0.12 across datasets, indicating dataset-specific geometry and motivating target-domain adaptation.The paper proposes a detector over Span(δ1, . . . , δk) as a training-free generalization extension.Source: §7 · p. 9

• The hallucination detection signal is overwhelmingly dominated by a linear mean shift.Apparent structural complexity largely reflects high-dimensional covariance estimation difficulty, quantified as about 73% of the Fisher LDA gap.Source: §8 · p. 9
• Properly regularized L2-LR outperforms complex variance-targeting architectures.The conclusion presents this ordering as a framework for future probe design.Source: §8 · p. 9
• LayerMix captures the distributed hallucination signal with an efficient, oracle-free multi-layer aggregation strategy.It matches oracle performance without requiring oracle layer access.Source: §8 · p. 9
• The signal’s orthogonality across domains necessitates adaptation, while simple linear probes support sample-efficient recalibration across distributions.The conclusion links domain variation to the need for adaptation and probe simplicity to rapid recalibration.Source: §8 · p. 9

• LayerMix requires white-box hidden-state access and is restricted to open-weight models.Source: Limitations · p. 9
• In a Qwen2.5-7B pilot, paired-example transfer to model-generated hidden states collapsed to 0.477 AUROC, with cosine similarity −0.095.The paired-example oracle reached 0.943 AUROC, indicating a boundary between the controlled and dynamic-generation paradigms.Source: Limitations · p. 9
• The adapted baseline comparisons benchmark static geometric properties rather than full operational capabilities in dynamic settings.The 12 custom architectures are controlled hypothesis tests, not standalone SOTA competitors.Source: Limitations · pp. 9–10
• Whether the extreme simplicity of the hallucination signal persists in frontier-class models such as 400B+ systems remains unverified.The reported scaling coverage spans models from 0.5B to 70B parameters.Source: Limitations · p. 10
• LayerMix’s mean gain over the single-layer oracle is only +0.002 AUROC, from 0.952 to 0.954.Its primary contribution is removing held-out oracle layer selection rather than producing massive accuracy gains.Source: Limitations · p. 10

• LayerMix may expose a vulnerability because adversarial prompts could shift hidden states orthogonal to the detection direction.Source: Ethics Statement · p. 10
• False negatives could create dangerous over-reliance, especially in high-stakes domains such as healthcare or law.Source: Ethics Statement · p. 10
• The authors recommend using LayerMix only as a supportive diagnostic within a broader human-in-the-loop verification pipeline.Source: Ethics Statement · p. 10
• All experiments use publicly available models and datasets, with no human subjects involved.Source: Ethics Statement · p. 10

Per-Model Results

• The evaluation covers 25 models from five families, spanning 0.5B to 70B parameters.Source: A Per-Model Detailed Results across… · p. 13
• Cross-model variation for SVD+LR and SAPLMA remains small, at ≤0.011 across the models averaged in Table 3.Source: Table 6 · p. 13, A Per-Model Detailed Results across… · p. 13

Layer Selection and Aggregation

• C = 0.001 consistently provides the optimal L2 regularization strength across all three datasets.Source: Figure 5 · p. 13, D Regularization Analysis · p. 13

Cross-Domain Adaptation

• LayerMix’s simple linear probe rapidly recovers cross-domain performance, achieving an overall mean AUROC above 0.70 with N = 500 target-domain examples.Source: Table 9 · p. 14, E Domain Generalization and Adaptation · p. 14

Cross-Domain Geometry

• Mean-shift directions are nearly orthogonal across domains, with observed absolute cosine similarities ≤0.154 and a within-to-between cosine ratio of about 7×.Within-dataset geometry is largely shared, whereas across-dataset geometry is nearly orthogonal; residual artefacts are not excluded.Source: Table 10 · p. 14, Table 11 · p. 15, Within-dataset vs. between-dataset… · p. 15

Geometric Decomposition

• Per-class eigenvalue spectra are near-identical, with r > 0.96, supporting a mean shift rather than differential covariance.Source: Table 12 · p. 15

Sparse Neuron Probing

• Removing neurons degrades the signal gracefully, while only 4 of the top 20 neurons are shared across five folds.This indicates that no fixed neuron subset carries the full hallucination signal.Source: Table 13 · p. 15, H Sparse Neuron Probing · p. 15

Label efficiency

• PLS-DA outperforms full-dimensional LR at N ≤200 in the six-condition average, acting as an implicit regularizer.The figure reports this label-efficiency pattern across six conditions.Source: Figure 7 · p. 17

Validation and selectivity

• Nested cross-validation estimates differ by ≤0.010 from standard 5-fold cross-validation across all conditions, indicating negligible optimistic bias.The stricter protocol used outer 5-fold and inner 3-fold cross-validation.Source: P Robustness Checks · p. 17
• Random-label controls yield AUROC 0.500±0.003, confirming zero selectivity under the null.These controls follow Hewitt and Liang (2019).Source: P Robustness Checks · p. 17

Model robustness

• Instruction tuning strengthens the underlying hallucination signal relative to the base Llama-3.1-8B model.This comparison is reported in Table 19.Source: Q Instruction-Tuned Model Results · p. 17, Table 19 · p. 17

Causal intervention

• The mean-shift direction produces a direction-specific causal effect, whereas random vectors show no systematic trend and the full-dimensional LR direction is weaker and non-monotonic.Random-vector controls have mean MC1 = 0.331 ± 0.008, while the full-dimensional LR direction spans MC1 0.325–0.330.Source: R Intervention Experiment Details · p. 18
• ∆=0.058 (17.4% relative) across the intervention sweep, with α=+2 above all random controls and α=−2 below them, but this is a single-model, single-dataset pilot.The authors call for systematic intervention studies across models and datasets.Source: R Intervention Experiment Details · p. 18