PaperlayerView exact revision on arXiv

Source-linked AI summary

Context or Knowledge: How LLMs Attribute Their Own Answers

Kirill Afendulev, Alexey Dontsov, Alla Chepurova, Elena Tutubalina, Anton Korznikov

arXiv:2610.32109v1cs.CL

TL;DR

The paper asks what determines a language model’s report about the source of an answer, a question distinct from whether the document supports that answer. It intervenes on the KV cache after answering and finds that reports follow transplanted question-and-answer state rather than document edits at report time. The findings support a causal role for cached state produced while answering, within the tested setting.

  • Problem

    When a document and parametric knowledge may both support an answer, it remains unclear how the model produces a report about the answer’s source.

    Source: §1 · p. 1

  • Method

    The authors intervene on selected KV-cache entries after answer generation and before asking the model about the source.

    Source: §1 · p. 2

  • Results

    Changing or removing document cache entries leaves reports unchanged, whereas transplanting the Q+A cache makes eligible reports follow the donor.

    Source: §1 · p. 2, §2 · p. 5

  • Takeaways & Limitations

    In this controlled setting, source reporting depends causally on source-related cached state inherited from the question-answering process.

    Source: §1 · pp. 2–3

    The experiments do not establish general reporting reliability, unrestricted introspection, or a complete semantic interpretation of the cached state.

    Source: §1 · p. 3, §8 · p. 9

Abstract

from arXiv · show

Language models can answer a question from a supplied document or from their parametric knowledge. When later asked where the answer came from, what determines their source report? We study this by intervening on the model's key-value (KV) cache after the answer is generated and before the source question is asked. On a controlled factual task with four models (7B-32B), changing or removing the document's cache never changes the report. In contrast, transplanting the question-and-answer (Q+A) cache between runs with identical question and answer tokens transfers it: every eligible report follows the donor, while the recipient's document is unchanged. On the two smaller models, steering Q+A values along directions fitted before any source question reverses 98.9-100% of reports across four document conditions at a fixed strength, while norm-matched random directions flip at most 1.1%. The components of keys and values along fitted directions carry the transplant effect: copying them alone nearly reproduces the report change; holding them fixed nearly prevents it. Applied during answer generation, the same value directions also change which source supplies the answer, though more weakly and depending on the prompt. Source reporting in this setting thus depends causally on cached state produced while answering. We do not establish general reporting accuracy or a complete interpretation of this state.

1 INTRODUCTION

The paper asks how a model reports the source of an already-generated answer, distinguishing that question from whether a document supports the answer. It tests whether the report depends on the document at report time, parametric knowledge, or cached state produced while answering, using interventions on the KV cache.

1 INTRODUCTION

  • The experiments test whether reports are determined by rereading the document, judging the answer against parametric knowledge, or using a cached trace of answering.The first two are tested as sufficient explanations, not as claims that document processing or parametric knowledge cannot contribute to the cached state.Source: §1 · p. 2, Table 1 · p. 2
  • Transplanting the question-and-answer cache transfers the source report while the recipient’s question, answer, and document remain unchanged.The report follows the donor cache, supporting a causal role for source-related state produced during the earlier computation.Source: §1 · p. 2

2 SETUP: HOLD THE WORDS FIXED, CHANGE THEIR CACHED STATE

  • The controlled task uses 155 candidate facts across eight relations, with seven equal-token-count document variants per fact.Steering experiments use the two smaller models; reported success rates are conditional on runs whose initial answers and reports meet pre-set eligibility rules.Source: §2 · p. 3
  • The setup asks models to answer with a document or from knowledge, then report the source as “context” or “knowledge.”The report is determined by the higher summed token logprobability of the two labels and agrees with the freely generated reply in 99.6–100% of cases.Source: §2 · p. 3
  • Cache interventions occur after answer generation but before the source question, allowing document entries or question-and-answer entries to be edited separately.A transplant replaces keys and values from another run; the main transplant changes Q+A together, with unedited entries retained.Source: §2 · pp. 3–4

3 CHANGING THE DOCUMENT LEAVES THE REPORT IN PLACE

Changing or removing the document cache after the answer is generated leaves the source report unchanged in this setting. The replacement document remains readable, but the report does not track whether the current document supports the answer.

3 CHANGING THE DOCUMENT LEAVES THE REPORT IN PLACE

  • Figure 2 schematically shows a counterfactual-document answer with the Q+A cache kept fixed while document entries are removed or replaced.The answer shown was generated before either edit; Table 2 reports results for all four models.Source: Figure 2 · p. 4, Table 2 · p. 4

2. Drop document

  • Zero source-report flips follow document edits across the tested models, with the question-and-answer cache retained.The edits attempt both to remove a context report and to create one.Source: Table 2 · p. 4, §2 · pp. 4–5
  • Even when the replacement document is readable, removing or replacing its cache does not change the report.The reverse test also leaves knowledge reports unchanged when a supporting fact is inserted; DynamicQA shows the same pattern with rare exceptions.Source: §2 · pp. 4–5

4 THE REPORT FOLLOWS THE QUESTION-AND-ANSWER CACHE

Transplanting the joint question-and-answer cache transfers the source report across runs even when the recipient’s document and answer remain unchanged, supporting a causal role for information encoded in that cache.

4 THE REPORT FOLLOWS THE QUESTION-AND-ANSWER CACHE

  • The experiment transplants the Q+A cache after answering to test whether reporting information is retained from the earlier computation.Source: §4 · p. 5

2. Donor

  • Runs with identical question and answer tokens exchange joint Q+A caches while retaining the recipient’s document and other cache entries.Source: §2 · p. 5
  • Because the report changes while the recipient’s document and answer stay fixed, the transplant provides causal evidence that information in the Q+A cache contributes to reporting.Source: §2 · pp. 5–6
  • Every eligible report followed the Q+A donor across all four tested models, while the recipient’s document stayed unchanged.Source: §2 · p. 5, Table 3 · p. 6
  • Same-condition control transplants produced zero report flips on every model.Source: §2 · p. 6

5 STEERING QUESTION-AND-ANSWER VALUES CHANGES THE REPORT

Steering fitted directions in Q+A values changes source reports across document conditions while leaving generated answers fixed. The fitted components also nearly reproduce the report-transfer effect when transplanted and largely prevent it when held fixed.

5 STEERING QUESTION-AND-ANSWER VALUES CHANGES THE REPORT

  • Directions fitted before any source question are added to or subtracted from Q+A values after answering, with keys and other cache entries unchanged.At α = 1, intervention magnitude equals the mean difference between the fitting conditions; controls use norm-matched random directions.Source: §5 · p. 6
  • With irrelevant documents, adding the direction yields context reports in 100% of evaluations at α = 2; subtracting it yields knowledge reports in 100% by α = 3 with counterfactual documents.The answer tokens were generated before the edit and remain fixed.Source: §5 · p. 6, Figure 4 · p. 7
  • At α = 3, steering flips 100% of reports across all four document conditions, except OLMo true facts (98.9%); random controls produce no flips on Qwen and at most 1.1% on OLMo.Source: §5 · pp. 6–7
  • Every report flips at α = 2 for irrelevant documents and α = 3 for counterfactual documents while answer repetition and document quotation still succeed.Both models also retrieve the counterfactual value in every case.Source: §5 · p. 7
  • The intervention establishes that state produced during answering can change the later report, but does not distinguish evidence availability from actual source use.The direction captures a manipulable component, not necessarily the entire source-related state.Source: §5 · p. 7

6 FITTED DIRECTIONS CAPTURE THE REPORT-TRANSFER EFFECT

  • Donor projections alone flip 97.9% of OLMo and 100% of Qwen reports; joint K+V clamps reduce flips from 100% to 1.1% on OLMo and 0% on Qwen.The clamp results show that restoring recipient projections largely prevents the transfer effect.Source: §6 · p. 7

1. Start with Q+A cache

Fitted key–value components in the question-and-answer cache largely account for report transfer. The same value direction can also change which source supplies an answer when applied during generation.

1. Start with Q+A cache

  • Fitted K+V projections largely account for the transplant’s report effect on true-fact–irrelevant-fact pairs.Clamping a control subspace leaves 100% and 94.7% of reports flipped.Source: §1 · p. 8, Table 4 · p. 8

7 STEERING VALUES DURING ANSWERING CHANGES THE CHOSEN SOURCE

  • The answer-generation test applies the fitted value direction to question and generated-answer tokens, with instructions opposing the edit.It steers toward memory despite an instruction to use the document, or toward the document despite a warning that it may contain errors.Source: §7 · p. 8
  • At α = 2, steering switches 71% of OLMo and 51% of Qwen answers toward memory, and 85% and 73% toward context, respectively.Random directions switch at most 4.1% except for OLMo toward context (24%); the fitted direction switches 3.6 times as many in that condition.Source: §7 · pp. 8–9, Figure 6 · p. 9

8 DISCUSSION AND LIMITATIONS

The interventions support a causal role for state computed while answering in later source reports, within the tested setting. The meaning of that state and the separate contributions of keys, values, question, and answer remain unresolved.

8 DISCUSSION AND LIMITATIONS

  • Across the interventions, reports depend on state computed before the source question, while general attribution reliability in natural retrieval settings remains untested.Document edits leave reports unchanged; Q+A transplants, steering, and projections change or control them, with steering and projection tests limited to 7–8B models.Source: §8 · p. 9
  • The fitted directions’ meaning remains unresolved: they may encode evidence availability, actual source use, or both.Hybrid mechanisms may contribute even though document rereading and parametric-knowledge judgment fail as complete accounts.Source: §8 · p. 9
  • Joint K+V edits nearly block or reproduce report transfer, but the separate roles and interaction of keys and values remain open.Answer repetition and document read-back were not tested for these joint edits.Source: §8 · p. 9
  • Main interventions edit question and answer states together; their separate contributions vary across models and conditions, and the fitted directions do not define one global source variable.Future work should distinguish what is established during question processing from what is added during answering.Source: §8 · p. 9

A RELATED WORK

The paper distinguishes source reporting from context-support attribution and situates its interventions alongside work on explanation faithfulness, source selection, and activation steering. Its factual-item design uses controlled document variants and held-out relation splits.

A RELATED WORK

  • Prior work studies context support, explanation faithfulness, source-choice prediction, and activation steering; this study tests the later source report through post-answer interventions.Related methods include context ablations, residual-activation prediction, and sparse-autoencoder control.Source: A RELATED WORK · p. 13

B GENERATED FACTUAL ITEMS

  • Experiments use subject–relation–object facts, type-matched counterfactuals, per-model eligibility checks, equal-length documents, and relation splits with held-out evaluation relations.The extent of domain transfer depends on the split.Source: B GENERATED FACTUAL ITEMS · p. 14

C.1 DOCUMENTS AND DOCUMENT INTERVENTIONS

  • Seven equal-length document variants support aligned cache exchanges; read-back caveats include Qwen3-8B denying an irrelevant fact and Qwen3-32B reconstructing deleted text from its answer.The reported read-back rates are 77.4% and 26.2%, respectively; the passage identifies other tests as clean comparisons for those cases.Source: B GENERATED FACTUAL ITEMS · p. 14, §C.1 · p. 15, Table 6 · p. 15

C.2 QUESTION AND ANSWER TRANSPLANTS

Cache-transplant and steering tests examine which spans and fitted components carry source reports. The results distinguish joint Q+A effects from individual-span contributions and include limits on what those interventions establish.

C.2 QUESTION AND ANSWER TRANSPLANTS

  • Document swaps produce zero report flips, while question swaps often move reports and are asymmetric on 32B models.The identical-text Q+A result uses a different eligible subset and additionally requires identical answer tokens.Source: Figure 7 · p. 16, §C.2 · p. 16, Table 8 · p. 16

C.3 KEYS AND VALUES IN QUESTION-ONLY TRANSPLANTS

  • In question-only transplants, value-only edits create the context report in most cases, while removing it is more effective with keys and values together.The cited table reports OLMo-3-7B and Qwen3-8B cells.Source: Table 9 · p. 17

C.4 QUESTION AND ANSWER SPANS IN TRANSPLANTS

  • Joint question-and-answer cache swaps establish sufficiency for report transfer on the tested pairs, not that each span independently encodes a source variable.The table reports percentages by swapped span, but the supplied caption gives no cell values.Source: §C.4 · p. 17, Table 10 · p. 17

C.5 JOINT REPORT STEERING ACROSS STRENGTHS

  • At α = 4, every Q+A steering cell reaches 100%; at α = 3, only OLMo on true facts is lower, at 98.9%.Rates pool eligible item–split pairs across two retained relation splits.Source: Table 11 · p. 17, §C.5 · p. 17
  • Random means are zero in all Qwen sweep cells; OLMo reaches 1.1% on irrelevant-fact Q+A at α = 3 and 2.2% on neutral Q+A at α = 4.Other nonzero OLMo random means are 1.6% on irrelevant-fact Q+A at α = 4.Source: §C.5 · p. 17, Table 11 · p. 17

C.6 QUESTION AND ANSWER SPANS IN REPORT STEERING

  • At α = 4 in the counterfactual setting, answer-only steering flips 86% of OLMo and 90.9% of Qwen reports, while Q+A flips 100% on both.At the common α = 3 strength, joint edits are more effective than either span alone in every model–condition cell.Source: §C.6 · p. 18, Table 12 · p. 18

C.7 CONTENT CHECKS AFTER REPORT STEERING

The checks test whether report steering preserves answer and document readouts, then compare two answer-value fitting contrasts. Preservation holds at the tested settings, while the contrast effects vary by model and direction.

C.7 CONTENT CHECKS AFTER REPORT STEERING

  • Every Q+A steering evaluation flips the report and passes answer repetition; document quotation and counterfactual-value read-back also succeed throughout the reported checks.These checks establish preservation at the stated settings, not across every strength or intervention.Source: Table 13 · p. 18, §C.7 · p. 18

C.8 SUPPORTING VERSUS IRRELEVANT AND FIXED-DOCUMENT FITTING CONTRASTS

  • At α = 3, the two answer-only fitting contrasts differ by model and direction: fixed-document steering removes context more effectively on Qwen, while neither contrast dominates removal across OLMo’s full sweep.The original contrast creates context more effectively on OLMo; neither answer-only direction reliably creates context on Qwen.Source: Table 14 · p. 19, §C.8 · p. 19
  • All norm-matched random controls produce zero report flips at tested strengths α = 1, 2, 3, and 4 for both fitting contrasts.The reported comparison is between the two fitted directions and their respective random controls.Source: §C.8 · p. 19
  • Answer repetition was not measured at α = 3, so the reported evidence establishes report control at that strength, not preservation of answer readout.Source: §C.8 · p. 19, Table 14 · p. 19

C.9 SOURCE-CHOICE STEERING: INSTRUCTIONS AND CONTROLS

The source-choice experiments compare value steering and a residual-stream alternative under prompts that initially elicit the source to be steered away from. They define report flips, answer switches, and the fitted cache-coordinate edits used for evaluation.

C.9 SOURCE-CHOICE STEERING: INSTRUCTIONS AND CONTROLS

  • The two source-choice settings steer toward memory under a document-priority instruction or toward the document under a prompt warning that it may contain errors.Recipients initially answer from the source targeted for steering away from.Source: §C.9 · p. 20
  • At α = 2, value and residual-stream methods intervene during question processing and answer generation; the residual-stream comparison uses a fitted mean hidden-state difference at question positions.The residual-stream method selects a layer using fitting relations; value steering reuses Section 5 directions without refitting for the instruction.Source: §C.9 · p. 20, Table 15 · p. 20

C.10 TERMINOLOGY USED IN THE TABLES

  • “Switched” means the answer changed to the other source’s value, while “damaged” means it contains neither source’s value.“Flip” refers only to a change in the source report.Source: §C.10 · p. 20

D.1 PAIRED RUNS, DIRECTIONS AND READOUT

  • The paired runs keep the recipient’s document fixed and edit its Q+A cache after answering, before the fixed source question.Question and answer tokens match across paired runs; other cache entries remain unchanged.Source: §D.1 · p. 21
  • Directions are fitted on DEV relations and evaluated on held-out relations; report flips use forced-choice P(context), with a flip crossing 0.5 away from the original report.Rates pool both splits and directions for each pair, with a directional evaluation—not a distinct fact—as the unit.Source: §D.1 · p. 21

D.2 TWO OPERATIONS WITH THE SAME COORDINATES

  • A value edit replaces the vector’s projection onto a fitted direction with the paired vector’s target projection, leaving the orthogonal value component unchanged.The clamp first transplants Q+A K+V, then uses the recipient’s original vector as the target.Source: §D.2 · p. 21
  • Key edits undo and reapply RoPE while preserving the starting key norm, changing key orientation but not preserving attention weights.The norm-preserving rotation is performed in post-gain space; it need not remain on the model’s exact normalization-and-gain ellipsoid.Source: §D.2 · p. 21

D.3 VALUE AND KEY CONTRIBUTIONS

Transferring fitted donor coordinates in Q+A keys and values nearly reproduces the report-transplant effect, while clamping those coordinates nearly prevents flips. Energy-matched controls show that the effect is not explained simply by capturing a large cache difference.

D.3 VALUE AND KEY CONTRIBUTIONS

  • Value-coordinate clamps eliminate flips from value-only transplants: 0% flip on either model, versus 55.3% on OLMo and 73.7% on Qwen without the clamp.With donor keys also transplanted, the value-only clamp leaves 50% and 34.2% flipped, respectively, all when creating context.Source: §D.3 · p. 22, Table 17 · p. 22
  • Joint K+V clamps leave 1.1% of OLMo reports flipped and 0% of Qwen reports flipped; the remaining OLMo flips occur only when creating context.No flips remain in the removal direction on either model.Source: §D.3 · p. 22, Table 18 · p. 22
  • 97.9% of OLMo and 100% of Qwen reports flip when only donor K+V coordinates are transferred, matching the main result.Rates are report flips over directional evaluations on true-fact–irrelevant-fact pairs.Source: §D.3 · p. 22, Table 19 · p. 23

D.4 ENERGY-MATCHED CONTROLS

  • Qwen’s energy-matched control captures more held-out energy in values and keys yet leaves 94.7% of reports flipped after clamping and flips only 2.6% when donor coordinates are imposed alone.This control matches difference energy on DEV data; held-out energy need not match.Source: §D.4 · p. 23, Table 20 · p. 23
  • Answer repetition and document read-back were reported as preserved for value-only donor-coordinate edits, but not for joint K+V conditions.Fixed dialogue tokens do not establish preservation of subsequent content readouts or exclusive coordinate semantics.Source: §D.4 · p. 23

E REPLICATION ON DYNAMICQA WIKIPEDIA PARAGRAPHS

The DynamicQA replication tests document edits and cache transplants on Wikipedia-derived paragraphs, restricting items to questions answerable from memory. Document edits mostly preserve reports, while joint Q+A transplants make nearly all reports follow the donor, with exceptions.

E REPLICATION ON DYNAMICQA WIKIPEDIA PARAGRAPHS

  • The replication uses seven length-matched Wikipedia-paragraph variants per item and questions each model can answer from memory.The setup tests richer benchmark-derived passages while retaining a known memory answer.Source: E REPLICATION ON DYNAMICQA WIKIPEDIA… · p. 24

E.1 DOCUMENT INTERVENTIONS

  • After deletion or irrelevant replacement of a counterfactual paragraph, context is retained in 98.4–100% of items; inserting a true or counterfactual paragraph into an irrelevant-document run creates no context reports.This extends the earlier pattern to these paragraphs, with rare exceptions to exact invariance.Source: §E.1 · p. 24, Table 21 · p. 24

E.2 QUESTION-ONLY TRANSPLANTS

  • The question-only transplant effects vary by model and direction; the joint read-back rate counts all eligible items, not only cases where the report flipped.The cited table description identifies the reported measures but does not state their values.Source: §E.2 · p. 24, Table 22 · p. 24

E.3 QUESTION AND ANSWER CONTRIBUTIONS WITH IDENTICAL TOKENS

  • 97.2–100% of reports follow joint Q+A transplants across the 16 model–direction cells, with exceptions to the universal transfer observed on the templated task.Separate-span results again depend on model and direction; steering on DynamicQA is outside the reported experiments.Source: §E.3 · p. 25, Table 23 · p. 25

Related papers

Loading 2610.32109v1…