PaperlayerView exact revision on arXiv

Source-linked AI summary

The Attribution Blind Spot: Layerwise Trajectory Diagnostics for Source Reliance in Retrieval-Augmented Language Models

Zhe Yu, Wenpeng Xing, Yunzhao Wei, Bo Yang, Chen Ye, Gaolei Li, Meng Han

arXiv:2610.09493v1cs.AI

TL;DR

Retrieved-evidence agreement does not show that a model relied on the passage, motivating controlled conflicts that expose source choice. The paper compares prediction signals with intervention directions and finds that state-change magnitude often better predicts choice, while signed PC1 better controls it selectively.

  • Problem

    Output agreement with retrieved evidence cannot show whether the passage or parametric memory supplied the answer, and prediction alone does not identify useful intervention directions.

    Source: §1 · p. 1

  • Method

    The paper compares state-change magnitude with signed LTS from a training-fitted PC1 and tests frozen PC1 interventions against matched alternatives for behavior, preservation, and transfer.

    Source: §1 · pp. 1–2

  • Results

    State-change magnitude is often the stronger source-choice predictor, whereas signed PC1 updates provide stronger selective control than tested alternatives with better non-target preservation.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Diagnosis and control require different endpoints because predictive readability does not establish that a state feature is a useful intervention direction.

    Source: §6 · p. 9

    The experiments do not recover the source of an agreeing answer or a unique source-selection circuit.

    Source: §6 · p. 9

Abstract

from arXiv · show

A retrieval-augmented model can match a document without relying on it. Controlled knowledge conflicts make source choice observable and let us ask a second question that prediction alone cannot answer: which internal-state properties define useful intervention directions? We study paired hidden-state changes with Latent Trajectory Shift (LTS), a signed projection onto a training-fitted first principal component (PC1), and keep verified training exposure separate from behavioral source choice. Across the evaluated conflicts, state-change magnitude is often the stronger predictor, whereas signed PC1 is the stronger selective controller: equal-norm interventions change source preference while better preserving non-target behavior, and the frozen direction transfers across the tested datasets and aligned model pairs. A same-system OLMo study further combines positive choice and control results with inconclusive exposure detection at the achieved power. The central result is a separation: representations that diagnose what a model will choose need not be the representations that best control that choice.

1 INTRODUCTION

Agreement with retrieved evidence cannot establish that the evidence supplied the answer, so controlled conflicts make source choice observable. The paper separates prediction of source choice from testing whether an intervention can selectively change it.

1 INTRODUCTION

  • A retrieved passage and parametric memory can support the same answer, leaving their contributions indistinguishable from output agreement alone.Replacing a passage naming Paris with one naming Rome forces a behavioral choice, but does not identify the source of the original agreeing answer.Source: §1 · p. 1
  • Prediction and control are distinct objectives: a predictive feature need not define a useful update, and an update can disrupt unrelated behavior.The paper evaluates prediction for diagnosis and behavioral change, preservation, and transfer for control.Source: §1 · p. 1
  • The study compares state-difference magnitude with signed LTS, then tests frozen PC1 updates against matched alternatives for preservation and transfer.OLMo connects exposure, choice, and control tests; Pythia extends exposure coverage, while Llama, Qwen, and Mistral extend behavioral tests.Source: §1 · pp. 1–2
  • State-change magnitude often predicts source choice better, but signed PC1 updates provide stronger selective control and better preservation than tested alternatives.The distinction is objective-dependent: diagnostic prediction and intervention utility favor different state properties.Source: §1 · p. 2

2 MEASURING AND INTERVENING ON PAIRED STATE DIFFERENCES

The method derives both a prediction score and an intervention direction from paired hidden-state differences, while keeping their uses separate. It evaluates source preference on controlled conflicts and applies a training-fitted direction to the unchanged conflict prompt.

2.1 FROM PAIRED RUNS TO A SIGNED SCORE

  • The paired runs provide a scalar feature and an intervention direction: magnitude records displacement size, while projection preserves its orientation.The layerwise axis is fit on outer-training items and prediction and intervention are evaluated separately on held-out data.Source: §2.1 · p. 2
  • For prediction, LTS projects the original paired-state displacement onto a first principal component learned from centered outer-training displacements.The axis is learned without source-choice labels, and its polarity is fixed using training data.Source: §2.1 · p. 3
  • Choice diagnosis uses conflict versus congruent pairs, while exposure uses context versus no context; these are separate fitted contrasts, not one universal direction.Predictive preprocessing is confined to outer-training folds, and layer blocks are prespecified rather than selected on held-out performance.Source: §2.1 · p. 3

2.2 OBSERVING AND CHANGING CANDIDATE PREFERENCE

  • Controlled conflicts pair a congruent passage supporting the parametric answer with a counterfactual passage supporting an incompatible candidate.The continuous candidate-score margin is distinct from the generated answer, and positive margin favors the contextual candidate.Source: §2.2 · p. 3
  • Control keeps the held-out conflict prompt fixed and applies a training-fitted, oriented PC1 update with frozen dose, without held-out congruent states or target-data refitting.Positive dose targets the parametric candidate and negative dose the contextual candidate; aligned margin shifts and discrete flips assess change and selectivity.Source: §2.2 · pp. 3–4

3 EXPERIMENTAL SETUP

The experiments separate verified exposure detection from behavioral choice and control, using exposure-labeled Pythia and OLMo data alongside conflict-based behavioral tests. They compare hidden-state signals and intervention directions under matched controls and prespecified evaluation procedures.

3.1 DATASETS AND MODELS

  • OLMo-2-1124-7B-Instruct uses 1,000 verified members and nonmembers for exposure, choice, and control, with controller transfer to ConflictQA.The shared set has 500 members and 500 nonmembers; exposure inference uses 50 paired group units.Source: §3.1 · p. 4
  • Behavioral tests use NQSwap paired passages and ConflictQA for source choice and transfer, with matched or counterbalanced conditions probing source content and structural cues.Llama, Qwen, and Mistral extend behavioral tests beyond OLMo; NQSwap flip rates are conditional on unsteered conflict answers being parametric.Source: §3.1 · p. 4
  • Earlier WikiMIA, BookMIA, and MIMIR checks use temporal labels, not verified exposure labels.Verified exposure coverage is provided by CheckMIABench for five Pythia checkpoints and by the OLMo common-support study.Source: §3.1 · p. 4

3.2 BASELINES AND CONTROLLED COMPARISONS

  • Table 1 matches intervention arms on items, layers, additive operator, and per-layer norm budget; the comparison separates direction value from update size and layer assignment.Candidate-gradient and LTS directions are fit on outer-training data under the same operator and norm budget.Source: §3.2 · p. 4, Table 1 · p. 5

3.3 ENDPOINTS AND STATISTICAL INFERENCE

  • Across five Pythia checkpoints, verified exposure is not detected by primary LTS: out-of-fold estimates have 95% cluster-bootstrap intervals overlapping chance, indicating non-detection rather than equivalence.Exposure uses ROC–AUC against verified membership; choice reports ROC–AUC and average precision (AP). Control reports direction-aligned margin effect and discrete candidate flips; congruent accuracy and no-context stability measure non-target behavior.Source: Figure 2 · p. 5, §3.3 · p. 5
  • Confirmatory analyses use grouped outer folds, 20 split seeds, cluster-bootstrap intervals, paired cluster permutation tests, and prespecified Holm correction.Fitting and threshold selection occur inside outer-training data, with control passing criteria fixed before held-out evaluation.Source: §3.3 · p. 5

4 RESULTS

The results test exposure, choice prediction, and intervention control as distinct endpoints. They show that state-change magnitude is often a stronger diagnostic predictor, while signed PC1 provides more selective control on the evaluated paths.

4.1 RQ1: WHAT CAN VERIFIED EXPOSURE ESTABLISH?

  • Pythia LTS reaches ROC–AUC 0.505–0.510 across 70M–2.8B, with every 95% interval including chance; OLMo exposure detection is also inconclusive.In OLMo, AUC is 0.508 and the design reaches 80% power only at AUC 0.570, so it cannot establish equivalence to chance.Source: §4.1 · pp. 5–6, Table 2 · p. 6

4.2 RQ2: HOW WELL DO STATE CHANGES PREDICT SOURCE CHOICE?

  • On 112 unambiguous NQSwap choices, LTS reaches AUC 0.593, versus 0.960 for ℓ2 magnitude and 0.992 for answer margin.Contextual choice remains similar under paraphrase but falls with unrelated or shuffled passages; mixed evidence increases ambiguous responses.Source: §4.2 · p. 6
  • On ConflictQA, magnitude ranks rare parametric choices better than LTS in Llama (Holm p = 0.0003), while the difference is unresolved for Qwen (p = 0.839).Answer margin is stronger in both models; supervised LTS does not reliably improve average precision.Source: §4.2 · pp. 6–7, Table 4 · p. 7

4.3 RQ3: CAN A FROZEN DIRECTION SELECTIVELY CONTROL CHOICE?

  • Under matched norms, signed PC1 shifts the margin by 0.412 [0.364, 0.460], versus 0.005 for unsigned ℓ2, which flips 0/110.Signed PC1 retains 97.8% congruent accuracy and 96.5% no-context stability, versus 70.1/65.0% for the unsigned radial update.Source: Figure 3 · p. 8, §4.3 · p. 8
  • LTS exceeds candidate-gradient steering in all eight middle/late NQSwap-to-ConflictQA transfer cells, while source steering strengthens with depth and three other targets weaken.Candidate-gradient preservation falls below both gates; source-versus-sentiment, format, and language target-by-depth interactions survive Holm correction.Source: §4.3 · p. 8
  • All 48 prespecified middle/late forward-transfer cells pass across Llama, Qwen, and Mistral, whereas none of the 24 early cells passes.Table 6 reports significant effects across seven dataset paths and aligned transport; random maps, wrong layers, and orthogonal directions have intervals containing zero.Source: §4.3 · p. 8, Table 6 · p. 9
  • On 197 valid OLMo paragraph responses, contextual steering raises contextual claim share from 0.125 to 0.510, while parametric steering lowers it to 0.042.Responses remain valid for 197/200 items, with NLI entailment of 96.0–96.2% and response-quality noninferiority interval [-0.018, 0.024].Source: §4.3 · pp. 8–9

5 RELATED WORK

Prior work measures several distinct forms of source attribution, while this paper separates behavioral choice from state prediction and intervention utility. Its intervention comparisons hold the additive operator, position, layer blocks, and norm budget fixed.

5 RELATED WORK

  • Training-history, recoverable-memorization, textual-support, and context-contribution measures address different questions from behavioral source choice.The cited approaches include membership inference, extraction, faithfulness and citation benchmarks, entailment categorization, and ablation.Source: §5 · p. 9
  • Internal analyses read out source-choice information or measure retrieval-induced state shifts, while this work compares magnitude and projection scores and tests intervention directions.Predictive information alone does not specify an intervention’s benefit or cost.Source: §5 · p. 9
  • Prior interventions alter knowledge preference through several mechanisms, whereas this comparison fixes the additive operator, position, layer blocks, and norm budget.Candidate gradients test answer steering; preservation and frozen transfer assess usefulness beyond a local margin change.Source: §5 · p. 9

6 DISCUSSION AND CONCLUSION

The discussion distinguishes state properties that predict source choice from those useful for intervention. It also limits the conclusions: these experiments do not identify the source of agreeing answers or a unique source-selection circuit.

6 DISCUSSION AND CONCLUSION

  • State-change magnitude can predict which source a model follows without making a radial update selective, while signed PC1 can aid intervention without maximizing prediction.Under the tested additive operator, intervention value is assessed through bidirectional change, matched-budget comparisons, non-target preservation, and frozen transfer.Source: §6 · p. 9
  • Conflict reveals preference and intervention tests controllability, but the experiments do not identify the source of an agreeing answer or a unique source-selection circuit.The discussion says remaining limits appear in Appendix A.1.Source: §6 · p. 9

AI USE STATEMENT

Generative AI tools assisted with research activities and manuscript work; the authors reviewed the materials and take responsibility for the final work.

AI USE STATEMENT

  • Generative AI tools assisted with literature discovery and research planning.Source: AI USE STATEMENT · p. 10
  • The tools also assisted with experimental and analysis code and interpretation of results.Source: AI USE STATEMENT · p. 10
  • The tools assisted with drafting, translation, and language editing, while the authors reviewed the materials and took responsibility for the final text, analyses, and claims.Source: AI USE STATEMENT · p. 10

A GUIDE TO THE SUPPLEMENTARY RESULTS

The supplement organizes results by endpoint: choice tables evaluate prediction, while control-related tables assess update effects and costs. It also documents alignment, statistical and replay details, and exposure tables.

A GUIDE TO THE SUPPLEMENTARY RESULTS

  • Choice tables evaluate prediction, while matched-control, preservation, and transfer tables evaluate update effects and costs.The guide cautions that these support separate claims rather than a single success score.Source: A GUIDE TO THE SUPPLEMENTARY RESULTS · p. 14
  • Appendix E specifies cross-model alignment, including 500 unlabeled anchors, layer mapping, and low-rank constraints.Source: A GUIDE TO THE SUPPLEMENTARY RESULTS · p. 14
  • The supplement places statistical and replay details before the full exposure tables, with the earlier discovery screen retained at the end.The discovery screen’s temporal-label results are distinguished from verified exposure.Source: A GUIDE TO THE SUPPLEMENTARY RESULTS · p. 14

A.1 LIMITATIONS

The paper distinguishes behavioral source-choice tests from identifying a source behind an agreeing answer, and defines intervention outcomes as changes under a specified additive operator. Its scope is constrained by the tested cohorts, controls, and endpoints.

A.1 LIMITATIONS

  • Exposure covers five Pythia checkpoints and a lower-powered OLMo bridge; intervention is linear and single-position, while preservation omits broad capabilities and nonlinear or item-specific controls.Long-form evidence covers one model and 200 items, PopQA-Conflict reuses ConflictQA, and alignment covers only Llama–Qwen.Source: §A.1 · p. 14

B OBSERVATION, COHORTS, AND MEASURED ENDPOINTS

  • If structural models agree on the observed-record distribution but disagree about the governing source, no statistic of that record alone identifies the source.Conflict and activation interventions answer additional behavioral questions; they do not identify the source from the original record alone.Source: B OBSERVATION, COHORTS, AND MEASURED… · p. 14
  • Checkpoint names, Pythia step, frozen ConflictQA files, and tokenizer-specific minimal-pair construction are specified separately from cohort filters and endpoint denominators.Structural-control passages are length matched, mixed-source order is counterbalanced, and outcomes remain in the 113-item denominator.Source: B OBSERVATION, COHORTS, AND MEASURED… · p. 14
  • The endpoint τ_B(κ) measures the expected margin change under a frozen block-set update, while directional tables report e_B aligned to the intended candidate.Greedy flips and non-target preservation are reported separately because equal margin changes can yield different answers and side effects.Source: B OBSERVATION, COHORTS, AND MEASURED… · p. 14

C SAME-SYSTEM EVIDENCE AND CONTROLS

The OLMo evaluation combines inconclusive exposure detection with positive choice, control, and transfer results. Additional controls separate signed-direction effects from magnitude and examine depth and long-form endpoints.

C SAME-SYSTEM EVIDENCE AND CONTROLS

  • 215/1,000 items flipped overall and 215/500 initially parametric items flipped; congruent accuracy remained 97.4%, while exposure was inconclusive at 50 paired group units and MDE80 AUC 0.570.The same-system evaluation reports positive choice, control, and transfer; its behavioral effects do not depend on detecting a training trace.Source: C SAME-SYSTEM EVIDENCE AND CONTROLS · pp. 14–15, Table 7 · p. 15
  • Across five prespecified PC1 comparisons, margin and flip contrasts were positive with Holm p = 5/10001, while ancillary retention and stability differences against several controls were inconclusive.Seven arms separate sign from magnitude on the same 110-item Llama/NQSwap support under a shared norm budget.Source: Table 9 · p. 15
  • Table 10 compares the source-control depth profile with three unrelated steering targets using normalized effects under Frobenius-matched intervention budgets.The caption establishes the comparison scope but does not state its outcome.Source: Table 10 · p. 15
  • The long-form OLMo endpoint uses 197 valid paired responses from 200 items and macro-averages claim-level attribution shares across multi-claim paragraphs.The study reports approximately 780 claims per condition, inter-annotator κ = 0.88, and a quality noninferiority interval of [−0.018, 0.024].Source: Table 11 · p. 16

D COMPLETE CONFIRMATORY RESULTS

The confirmatory results support a train-only PC1 controller in middle and late blocks, with transfer advantages over an equal-budget candidate-gradient direction and specified boundaries on that claim. Complete ledgers also distinguish passing middle/late controls from early failures and separate replay instances from nested token-position evaluations.

D.1 CONFIRMATORY TRAIN-ONLY DIRECTIONAL INTERVENTIONS

  • The confirmatory experiment fits PC1 on outer-training groups and executes held-out conflict prompts without held-out paired deltas, labels, candidate identities, gradients, or refitting.It covers 120 Llama items, 87 Qwen items, three layer thirds, nine doses, and five equal-budget arms; Figure 4 shows the complete dose response and exact zero-dose rows.Source: §D.1 · p. 16, Figure 4 · p. 18
  • Both models pass the train-only PC1 gate in middle and late blocks and fail it in early blocks, establishing a same-dataset, two-model controller.The gate requires positive effects against all eight comparator cells and both absolute cells, corrected p < 0.05, and at least 16/20 sign-concordant split seeds.Source: §D.1 · p. 16, Table 12 · p. 16, Table 13 · p. 17

D.2 EQUAL-BUDGET CANDIDATE-GRADIENT DIRECTION

  • LTS has a positive, Holm-significant advantage over the equal-budget candidate-gradient direction in every prespecified middle/late transfer cell for both models, and passes the two preservation point-estimate gates.LTS preservation estimates are 97.8% for congruent accuracy and 96.5% for no-context stability; the result does not identify a unique mediator or cover every control direction.Source: Table 15 · p. 18, Table 16 · p. 19, §D.2 · p. 20
  • LTS exceeds the prevalence baseline in both models but does not dominate the stronger predictive baselines in ConflictQA source-choice ranking.Average precision is averaged over 20 out-of-fold split seeds, with item-cluster intervals and paired tests.Source: Table 19 · p. 20

COMPLETE PRESERVATION AND REPLAY LEDGERS

  • E2 reports eligible flips on independent 110-item Llama and 100-item Qwen cohorts; its separate 50-item diagnostic ledger records 10 margin-shift-without-flip cases and 40 with no recorded failure.The ConflictQA preservation denominators are 777/617, and the Qwen no-context interval has a 94.7% lower bound.Source: E2: DISCRETE CHOICE AND NON-TARGET… · p. 20, Table 21 · p. 21
  • All 32 middle/late cells pass the complete E2 matched-control ledger, while all 16 early cells fail because their intervals cross zero.The ledger covers 48 cells across both models.Source: Table 20 · p. 21, Table 22 · p. 22
  • The E6 instance ledger contains 1,863 independently replayed forward passes, while its nested token-position ledger contains 12,420 candidate-position comparisons.Physical replay validates numerical execution fidelity; the uploaded summary manifest is not an independently recomputable scientific replication.Source: E6: NUMERICAL EXECUTION REPLAY AND… · p. 22, Table 23 · p. 22
  • The candidate-gradient comparison rules out reduction to the tested average gradient axis, but not every nonlinear or item-specific direction, a complete source-selection circuit, or a production provenance certificate.The intervention claim remains bounded to the tested two-candidate setting.Source: §D.2 · p. 20, WHAT THE CANDIDATE-GRADIENT COMPARISON… · p. 23

E Cross-Architecture and Cross-Dataset Transfer

Transfer tests support frozen LTS directions across the evaluated datasets and aligned model pairs, while the statistical and exposure sections specify the evidence boundaries. The supplementary discovery screen is retained as discovery evidence, not confirmatory source-choice evidence.

E Cross-Architecture and Cross-Dataset Transfer

  • Dataset transfer freezes the direction, blocks, dose, and threshold, while cross-model transport fits rectangular orthogonal maps to 500 unlabeled NQSwap anchors.The evaluation IDs are disjoint from the anchors, and target labels are unavailable during map fitting.Source: E Cross-Architecture and Cross-Dataset… · p. 23
  • The alignment maximizes a trace objective, but the SVD construction does not establish least-squares optimality for the tall map.The reported transport results evaluate the fitted maps rather than an optimality guarantee.Source: E Cross-Architecture and Cross-Dataset… · p. 23
  • Anchor data do not determine the null-space completion, and the transport controls do not isolate the effect of changing that completion.The authors therefore report empirical transport with fitted maps, not a unique cross-architecture coordinate system or invariance to null-space choice.Source: E Cross-Architecture and Cross-Dataset… · p. 23
  • All 48 middle/late forward-transfer cells pass and all 24 early cells fail; aligned maps retain 84.5% of the native Qwen effect and 83.0% of the native Llama effect.The complete transfer table reports aligned LTS-minus-comparator effects, and the cross-model table reports the retained native effects.Source: Table 24 · p. 24, Table 26 · p. 24, F STATISTICAL INFERENCE AND REPLAY… · p. 25

F STATISTICAL INFERENCE AND REPLAY VALIDATION

  • Confirmatory intervals use item/source-group cluster bootstraps; numerical replay validates execution fidelity but is not independent clean-room reproduction of every analysis table.The transfer inference retains 72 comparator effects and integer sign-flip counts, without inventing cell-specific bootstrap intervals.Source: F STATISTICAL INFERENCE AND REPLAY… · p. 25

G Exposure: Exact Checkpoints Bound the Claim

  • Five exact step-97,000 Pythia checkpoints test verified exposure with fold-local fitting; Table 27 reports pooled out-of-fold ROC-AUC on 3,994 probes and 1,997 reciprocal groups.Each of the five model-free methods is printed once because estimates and intervals are bitwise-identical at every scale.Source: G Exposure: Exact Checkpoints Bound… · p. 25, Table 27 · p. 25

H Discovery Breadth Is Not Confirmatory Attribution

  • The nine-model WikiMIA screen uses temporal labels and shared calibration, so it is retained as breadth-oriented discovery evidence rather than verified exposure.The MIMIR negative control and model-free WikiMIA audit also guard against interpreting the screen as verified exposure or direct source use.Source: Table 30 · p. 27, H Discovery Breadth Is Not… · p. 27, Table 31 · p. 27

Related papers

Loading 2610.09493v1…