PaperlayerView exact revision on arXiv

Source-linked AI summary

What LLMs Do in the Shadows: Reading Context Reliance from the Unembedding Dark Space

Alexandra Kuleshova, Andrey Volodichev, Daria Kotova, Alexey Zaytsev

arXiv:2610.23182v1cs.CL

TL;DR

Faithfulness hallucinations can occur when models prioritize parametric knowledge over source context, but measuring context reliance can require extra computation or probe training. This paper estimates context reliance from hidden-state projections onto the unembedding dark space and combines that score with parametric knowledge in DARKLENS. Across three benchmarks and six open-weight LLMs, DARKLENS matches or outperforms a wide range of baselines; its scope is faithfulness rather than factuality.

  • Problem

    Existing context-reliance approaches can require attention-map materialization, counterfactual forward passes, or auxiliary probe training to detect faithfulness hallucinations.

    Source: §1 · p. 1, §4.2 · p. 5

  • Method

    The paper measures the hidden-state norm fraction in the unembedding dark space as DMR and pairs it with a parametric knowledge score in single-pass DARKLENS.

    Source: §1 · p. 1

  • Results

    Across three benchmarks and six open-weight LLMs, DARKLENS matches or outperforms a wide range of baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dark components at non-BOS positions actively mediate context processing, making their hidden-state norm share a direct context-reliance signal.

    Source: §1 · p. 1, §7 · p. 8

    DARKLENS measures faithfulness rather than factuality, so it will not flag an output that faithfully repeats inaccurate context.

    Source: Limitations · pp. 8–9

Abstract

from arXiv · show

Detecting faithfulness hallucinations in large language models (LLMs) is crucial for their safe deployment in high-stakes domains. Recent approaches address this by contrasting contextual against parametric knowledge utilization, yet measuring context reliance typically demands noticeable overhead, such as materializing attention maps, running additional forward passes, or training auxiliary probes. In this work, we show that context reliance can be estimated by projecting hidden states onto the dark space - directions that compose weakly with the unembedding matrix. Beyond its recognized function as an initial-token attention sink, we show that this subspace actively mediates context processing, and that its share of hidden-state norm - termed the Dark Matter Ratio (DMR) - provides a direct score of context utilization. Pairing DMR with a parametric knowledge score yields DarkLens, a lightweight faithfulness detector operating strictly within a single forward pass. Evaluated across three QA and summarization benchmarks on six open-weight LLMs (4B-13B parameters), DarkLens matches or outperforms a wide range of baselines without requiring extra forward passes or probe training.

1 Introduction

Faithfulness hallucinations can arise when models favor parametric knowledge over contextual evidence, while existing context-reliance measures often add inference or training overhead. The paper shows that dark-space components mediate context processing and uses their hidden-state norm share in a lightweight detector.

1 Introduction

  • The paper frames inference-time detection of claims that contradict or cannot be verified against source text as important for reliable deployment in high-stakes domains.Examples include medicine, law, and finance.Source: §1 · p. 1
  • Prior work links faithfulness hallucinations to conflicts between parametric memory and contextual evidence, and detects them by contrasting context utilization with parametric reliance.Existing approaches may require attention-map materialization, counterfactual forward passes, or auxiliary probe training.Source: §1 · p. 1
  • Dark components at non-BOS positions actively mediate context processing and retrieval, contrary to the hypothesis that the subspace primarily forms initial-token attention sinks.The paper defines the dark space as directions that compose weakly with the unembedding matrix and minimally affect vocabulary logits.Source: §1 · pp. 1–2
  • DMR measures the fraction of hidden-state norm residing in the dark space as an attention-free context utilization score.It avoids attention-map materialization and extra forward passes.Source: §1 · pp. 1–2
  • Across three benchmarks and six open-weight LLMs, DARKLENS matches or outperforms a wide range of baselines.It combines DMR with a parametric knowledge signal in a single forward pass.Source: §1 · p. 2

2 Background

The background describes how residual-stream representations map to vocabulary logits and how weakly composing directions can remain invisible to vocabulary-level readouts. It introduces the dark-space decomposition and the prior focus on BOS attention sinks.

2 Background

  • The unembedding matrix maps residual-stream hidden representations to vocabulary logits, while logit-lens approaches overlook components that compose weakly with that matrix.Those components have little effect on output logits.Source: §2 · p. 2
  • The dark space is spanned by right singular vectors corresponding to the 5% smallest singular values of the unembedding matrix.Projections onto it induce negligible logit variation and are largely invisible to vocabulary-level readouts.Source: §2 · p. 2
  • A hidden state decomposes into a dark component and a vocabulary-visible component, and prior work found the BOS token resides almost entirely in the dark space.This allows BOS to act as an attention sink without distorting vocabulary logits; the role of dark space at subsequent tokens had remained unestablished.Source: §2 · p. 2
  • The paper examines dark representations at non-BOS positions across intermediate layers in relation to context utilization.Source: §2 · p. 2

3 Empirical Study

Across synthetic copying and contextual QA, dark-space interventions disrupt context-dependent behavior, while scaling the dark component shifts context use in both directions. The experiments leave the BOS representation unchanged, separating measured effects from its attention-sink role.

3.1 Synthetic Induction

  • On Qwen3-8B, dark swaps cut repeated-half copying accuracy by up to 95% in windows starting at layers 4–20, while matched mid-spectrum swaps largely preserve it.The task repeats a 64-token random sequence, and each four-layer intervention exchanges dark components across samples; effects on Llama-3.1-8B-Instruct are more modest. Across four standard corpora, dark interventions keep NLL near unablated levels and below smaller references.Source: §3.1 · pp. 2–3, Figure 1 · p. 3, Figure 2 · p. 3

3.2 Dark Intervention: Contextual vs Parametric

  • On Qwen3-8B, dark permutations reduce contextual preference by up to 75% relative on NQ-Swap while leaving parametric accuracy almost intact; the mid-spectrum control damages parametric performance by up to 17%.Across most layer windows, dark permutations also penalize context-aligned answers more than controls, while parametric and distractor NLLs decrease together; early-window positional disruption can affect contextual hit rates under control swaps.Source: §3.2 · p. 4, Table 1 · p. 4, Figure 3 · p. 4

3.3 Dark Intervention: Modulating Context Use

  • Dark-component enhancement raises and shrinking lowers context hit rates on NQ-Swap and ConflictQA, with Llama-3.1-8B-Instruct peaking near α = 0.2 before declining.The mid-spectrum control stays near baseline on Qwen3-8B but shows an inverse trend on Llama-3.1-8B-Instruct.Source: Figure 4 · p. 5, §3.3 · p. 5

4 Method: DarkLens

DARKLENS frames faithfulness detection as a competition between contextual grounding and parametric reliance, using the Dark Matter Ratio (DMR) as a context signal and pairing it with a parametric knowledge score. Its token- and response-level scores use hidden-state projections and can be computed in one forward pass.

4 Method: DarkLens

  • DARKLENS combines DMR with the Parametric Knowledge Score (PKS) to balance contextual grounding against parametric reliance when scoring faithfulness hallucinations.The detector builds on a framework that assigns each generated token a hallucination score from contextual and parametric signals.Source: §4 · p. 5, §4.1 · p. 5

4.2 Dark Matter Ratio for External Context Utilization

  • Existing context-reliance measures may require attention-map materialization, probe training, or a counterfactual forward pass.DARKLENS instead reads context utilization directly from residual hidden states.Source: §4.2 · p. 5
  • DMR quantifies external context reliance from the residual stream using a linear projection of the hidden state, with negligible computational overhead.The state is conditioned on the prefix tokens and is used immediately to predict the next token.Source: §4.2 · p. 5

4.3 The DARKLENS Detector

  • PKS is the Jensen–Shannon divergence between vocabulary distributions before and after an FFN block, with elevated divergence indicating strong FFN intervention.DARKLENS substitutes PKS and DMR for the parametric and contextual signals in its token-level score.Source: §4.3 · p. 6
  • DARKLENS combines PKS across evaluated FFN layers with DMR at a selected readout layer, then averages token scores for a response-level score.The readout layer and dark dimension are hyperparameters; PKS and DMR use residual hidden states and static projections, enabling a single forward pass.Source: §4.3 · p. 6

5 Results

Across QA and summarization evaluations, DARKLENS is competitive with general-purpose and RAG-specific baselines, with strengths varying by task and model. Ablations and sensitivity analyses indicate that both its component signals and the dark-space choice matter, while rank and readout-layer choices have practical bounds.

Main results

  • DARKLENS leads all three evaluated models on MS MARCO, is strongest on two CNN/DM models, and ranks first on five of six XSum models while exceeding ReDeEP and LUMINA across all six.The evaluation covers three tasks and six open-weight LLMs; on CNN/DM, LN-Entropy and LSC lead on LLaMA-2-7B.Source: §5 · p. 6, Table 2 · p. 7

Complementarity of PKS and DMR

  • Pairing DMR with PKS improves XSum AUROC from 0.61 to 0.65 for LLaMA-3.1-8B and from 0.69 to 0.71 for gemma-3-4b.DMR alone carries most discriminative power in most configurations, while both terms contribute to DARKLENS.Source: §5 · p. 6, Table 3 · p. 7

Sensitivity to dark dimension

  • Detection performance is largely stable across dark-dimension ranks between extremes, while rank 8 may fail to isolate the context signal and rank 256 on gemma-3-4b may encroach on the vocabulary-visible subspace.When validation tuning is not desired, the passage recommends a dark dimension of approximately 1.5–3% of d_model.Source: §5 · pp. 6–7

Sensitivity to readout layer

  • On MS MARCO, readout performance generally improves with depth, stays stable across mid-to-late layers, then drops near the final layer.When validation tuning is infeasible, roughly 80% of model depth is offered as a robust default readout layer.Source: §5 · p. 7

Specificity of the dark space

  • Detection degrades when the dark subspace is replaced by equal-dimension mid-spectrum or random orthogonal subspaces, approaching chance across benchmarks.This supports the hypothesis that context-reliance information is primarily carried by directions with low unembedding composition.Source: §5 · p. 7

6 Related Work

Prior representation-analysis tools largely leave the unembedding dark space understudied, while earlier work mainly linked it to attention sinks, calibration, or a small set of entropy neurons. Faithfulness detectors that measure context reliance have also incurred overhead that DARKLENS avoids by reading residual-stream geometry.

6 Related Work

  • Earlier work mainly associated the dark space with BOS attention sinks and confidence calibration; Tighidet et al. (2025) linked final-layer entropy neurons to suppressing context copying, unlike this work’s intermediate representations.Logit-lens and tuned-lens projections largely discard the dark space, leaving it sparsely studied.Source: §6 · pp. 7–8
  • Prior internal-state detectors often use learned probes or heuristics, while context-versus-parametric methods measure context reliance with attention maps, extra forward passes, or auxiliary supervision.ReDeEP materializes attention maps, LUMINA uses an additional forward pass, and SeReDeEP adds auxiliary supervision.Source: §6 · p. 8

7 Conclusion

The conclusion argues that dark components beyond the BOS position participate in context retrieval and processing. It presents DMR and DARKLENS as a lightweight way to quantify context reliance and detect faithfulness hallucinations.

7 Conclusion

  • Dark components at subsequent positions actively mediate context retrieval and processing, beyond the dark space’s previously hypothesized BOS attention-sink role.The conclusion describes this as an empirical finding in autoregressive language models.Source: §7 · p. 8
  • DMR quantifies context reliance as the fraction of hidden-state norm in the unembedding dark space.The dark space comprises residual directions that compose weakly with the unembedding matrix and minimally affect vocabulary logits.Source: §7 · p. 8
  • Pairing DMR with a parametric knowledge score yields DARKLENS, which detects faithfulness hallucinations across QA and summarization benchmarks in a single forward pass.The conclusion reports matching or outperforming a broad range of baselines without attention-map extraction or auxiliary-model training.Source: §7 · p. 8

Limitations

The limitations distinguish faithfulness from factual correctness, require access to residual activations, and leave the circuit-level mechanism for dark-space context retrieval unresolved.

Limitations

  • Faithfully repeating inaccurate context is not flagged as an error by DARKLENS.This is a faithfulness-versus-factuality boundary.Source: Limitations · pp. 8–9
  • DARKLENS requires residual-stream activations, making it inapplicable to closed-source black-box models.Source: Limitations · p. 9
  • The precise circuit-level mechanism by which the dark space mediates context retrieval remains incompletely understood.The proposed routing of information through dark directions to modulate attention heads is a hypothesis for future investigation.Source: Limitations · p. 9

A Synthetic induction: experimental details

Synthetic-induction experiments find that dark swaps impair copying across evaluated windows, while boundary-window interventions can cause broad language-modeling degradation. The appendix also reports that the copying effect persists at reduced dark-subspace ranks.

A Synthetic induction: experimental details

B Contextual vs Parametric: experimental details

The supplementary experiments test whether dark-subspace interventions selectively affect context use across layer windows and ranks, against matched-spectrum and parametric controls. They also describe the contextual-conflict and closed-book datasets used for comparison.

B Contextual vs Parametric: experimental details

  • Matched mid-spectrum controls reproduce the rank-205 dark-subspace permutation's WikiText-2 NLL increase within ±0.01 for each layer window.The dark subspace is the bottom 5% of right singular vectors of WU; the middle subspace is selected by the same procedure.Source: Table 6 · p. 12, B Contextual vs Parametric:… · p. 12
  • Dark-subspace permutations reduce context-aligned preference across layer windows while largely preserving parametric accuracy; mid-spectrum controls degrade both on Qwen3-8B.For Llama-3.1-8B-Instruct, dark permutations also lower contextual preference relative to the mid-spectrum control, while parametric recall is unaffected by either intervention.Source: B Contextual vs Parametric:… · p. 12, Table 7 · p. 13
  • 13% relative: Qwen3-8B's NQ-Swap target hit rate drops at dark rank 64, while parametric-dataset performance remains unchanged across tested dimensions.The reported effect diminishes at smaller dark ranks but remains substantial; Table 9 identifies the rank-64 comparison.Source: B Contextual vs Parametric:… · pp. 12–13, Table 9 · p. 13

C.1 Contextual vs. parametric

  • NQ-Swap and ConflictQA use passages with injected answers that contradict world knowledge, while Cities, Celebrities, and Sports are closed-book yes/no probes of stored knowledge.The contextual datasets use 400 examples each; Cities has 400 questions, and Celebrities and Sports have 200 each.Source: §C.1 · p. 13

C.2 RAG-style benchmarks

The section describes English-language QA and summarization benchmarks, implementation choices, artifact use, and a boundary on what faithfulness scores establish. It also reports how AI tools contributed to proofreading and coding assistance.

C.2 RAG-style benchmarks

  • The evaluation spans MS MARCO longform QA and CNN/DM summarization from RAGTruth, plus XSum's highly abstractive, single-sentence summaries.The authors characterize XSum as a setting where models are prone to hallucination; its generations and annotations come from Kuleshova et al. (2026).Source: §C.2 · p. 13

D Main Results: implementation details

  • DARKLENS hyperparameters were selected on the validation set by maximizing AUROC, with α fixed at 1 and searches over top-k, β, null-dim, and LM layer; all methods based on multiple generations employed 5 additional model generations.Source: D Main Results: implementation details · p. 14

E Use of scientific artifacts

F Potential risks

  • DARKLENS measures grounding in provided context, not real-world correctness; inaccurate or adversarial retrieved documents can therefore receive high confidence.Users may misinterpret that confidence as a guarantee of objective truth.Source: F Potential risks · p. 15

G Use of AI

  • Gemini 3.8 Flash assisted with proofreading, while Cursor models assisted with experiment code and result visualizations; the authors attribute the main idea and design to humans.Source: G Use of AI · p. 15
Loading 2610.23182v1…