Source-linked AI summary
How Do Language Models Choose Between Context and Memory?
Benjamin Shih, John Winnicki, Arianna Cao
TL;DR
Language models may choose between conflicting context and parametric memory using internal authority computations, but decodability or steerability alone does not establish causal use or cross-task reuse. The paper estimates authority directions from agreement prompts and swaps their naturally occurring coordinates between matched prompts, finding strong within-task effects but substantially weaker cross-task reuse. These findings distinguish authority representation, causal use, and causal reuse across tasks.
Problem
The paper addresses whether internal authority representations causally control source choice and whether that computation transfers across tasks.
Method
The authors estimate authority directions from agreement prompts and interchange their activation coordinates between matched high- and low-authority conflict prompts.
Results
Authority-coordinate interchanges close 30-68% of the within-task authority gap across Qwen, Llama, and OLMo, while cross-task transfer is substantially weaker.
Takeaways & Limitations
The results separate authority representation, causal use within a task, and causal reuse across tasks, suggesting authority computations may be task-dependent.
Takeaways & Limitations
Weak cross-task transfer may reflect small prompt samples, differing task geometry or scaling, or shared nonlinear representations with task-specific linear realizations.
Abstract
from arXiv · showhide
When contextual information conflicts with the knowledge stored in model parameters, activation directions can be used to decode and steer which source the model follows. However, steering along a direction does not establish causality: whether the unedited model would naturally use that direction or whether the direction is reusable across tasks. We test these distinctions through counterfactual experiments in unambiguous settings. First, we estimate authority directions from agreement prompts, in which the context and parametric knowledge support the same answer. We then interchange naturally occurring coordinates along these directions between matched prompts that direct the model to prioritize either the supplied context or its parametric knowledge. Across Qwen, Llama, and OLMo models, this intervention reproduces 30-68% of the authority-induced shift in source choice, whereas matched controls reproduce almost none. To test cross-task reuse, we learn authority directions on two tasks separately and see that cross-task transferability closes only 9% of the authority gap while the local direction learned on the given task closes 57%. These results distinguish authority representation, causal use, and cross-task causal reuse, and suggest that authority computations may be task-dependent, rather than reusable across tasks.
1 Introduction
The paper asks which internal computation determines whether a language model follows supplied context or parametric memory, and whether that computation transfers across tasks. It addresses the gap between representations that are decodable or steerable and computations that the unedited model causally uses.
- Contextual information can conflict with knowledge stored in model weights, termed parametric memory.
- Prior work identifies low-dimensional representations and steering methods that influence whether predictions follow context or parametric knowledge.
- Decodability and steerability do not establish that an unedited model naturally uses a representation to choose between context and memory.
- Authority-related representations may fail to transfer across tasks or prompt formats.
- The paper tests causal influence, cross-task transfer, and the distinction between transferable authority information and causally usable information.
2 Methodology
The method creates unambiguous context-memory conflicts, estimates authority directions from agreement prompts, and swaps naturally occurring authority coordinates between matched prompts. It measures source-choice changes against normalized authority gaps and control directions.
- Prompts pair a true parametric-memory fact with a false contextual claim and score the next token over twelve color candidates.
- Authority is varied by changing only the presentation of a fixed false contextual claim, including high-authority and low-authority wording.
- The study evaluates four frozen, open-weight instruction-tuned models without fine-tuning.
- Per-layer authority directions are estimated from normalized residual-stream differences between high- and low-authority agreement prompts.
- Residual activations are perturbed by adding or subtracting the authority direction, with random-direction perturbations as controls.
- The experiment checks token-specificity across all twelve color candidates and direction stability on disjoint subject sets.
- 2.2 Causal mediation of source selection: For matched prompt pairs, the intervention replaces the authority-direction component while preserving the orthogonal residual and recomputing downstream activations.
- 2.2 Causal mediation of source selection: Gap closure combines reciprocal changes in context-following probability and normalizes them by the natural high-low authority gap.
3 Authority trajectory locally steers source control choice
Within the color task, interchanging naturally occurring authority coordinates changes which source the model follows. The effect is substantial across Qwen, Llama, and OLMo, while random interchange is near zero.
- 68% of the natural high–low authority gap is closed in Qwen, 56% in Llama, and 30% in OLMo.
- Random interchange is near zero under the same color-task intervention.
- The results provide evidence that authority-coordinate intervention mediates source choice within a task and layer.
4 Limited causal transferability across tasks
Cross-task transfer of authority directions is substantially weaker than within-task causal effects, despite the source task producing a strong authority gap.
- 0.085 of the natural effect, or 14.8%, was closed when the color direction set was imported into the materials task.The imported trajectory outperformed the strongest covariance-matched control by roughly 0.042.
- The materials task retained a natural authority gap of 0.560, while memory accuracy was 0.988.
- The strongest covariance-matched control closed 0.043, compared with 0.085 closed by the imported color trajectory.
5 Authority representation is not causal use
Authority can remain highly decodable across domains without the imported direction causing a meaningful change in source choice.
- 0.995–1.000 state/country AUROC shows that authority gaps remained highly decodable across domains.Country/state AUROC was 1.000.
- 0.014 and 0.038 were the largest imported-direction effects across all layers in the two transfer directions.These effects were negligible despite high cross-domain decoding.
- 0.454 for countries and 0.492 for states were the authority effects under the ordinary reference prompt, falling to 0.120 and 0.065 under EA0.
- Generic padding produced similar suppression, indicating that the null effect was not specific to EA0 wording.
- High cross-domain authority decoding therefore did not necessarily translate into significant causal behavior changes.
6 Discussion
The discussion distinguishes representation, within-task causal use, and cross-task causal reuse, concluding that authority computations may be task-dependent and motivating broader tests.
- Discussion: The intervention copies naturally occurring activation components between matched prompts, providing stronger evidence for inherent causal use than additive steering.
- Limitations and future work: Weak cross-task transfer may reflect small prompt samples, differing task geometry or scaling, or shared nonlinear representations with task-specific linear realizations.
- Limitations and future work: Even highly decodable authority can have little behavioral effect when an imported direction is intervened on at a single layer.
- Limitations and future work: Repeated multi-layer interchange and experiments across many tasks could test whether the observed behavior is task-specific or broader.
- Discussion: Authority-related activation directions causally influence source choice, but their effects are substantially less reusable across tasks than within a task.
- Experimental scope: The experiments use frozen Qwen2.5, Llama-3.1, and OLMo models without fine-tuning, with Qwen2.5-3B used only for readout and steering.
B Authority families
Authority cues can shift context-following behavior, and directions estimated from several cue families are aligned and effective relative to random directions.
- 0.5 to 0.9 increases in context-following occurred for authority directions from wording, source credibility, evidence, and consensus cues.These effects were several times larger than those of a random direction in Qwen and Llama.
- 0.01 was the natural behavior change for first-person confidence, making its estimated direction have little effect.“I am certain” versus “I might be wrong” served as an empirical null baseline.
C.1 The color task: authority steering
The color task estimates authority directions from agreement prompts and evaluates them on authority-conflicting prompt pairs.
- The authority direction is estimated from mean residual activations across 200 prompts per authority and evaluated on 400 conflicting pairs.Answers are selected from a fixed, balanced set of twelve colors.
- The color prompts balance colors so that no authority is systematically associated with any color.
- Figure 4 compares directions trained on wording, source, evidence, and consensus cues across responsive cue families.First-person confidence serves as a near-null baseline.
C.1.1 Color results
Color-task interventions show that authority coordinates strongly affect source choice, with effects concentrated in late layers and evidence that the authority subspace is low-dimensional.
- At layer 21, steering toward authority changes context following from 0.14 to 0.76 on low-authority prompts, while steering away changes 0.90 to 0.06 on high-authority prompts.
- The mediation index is 0.35 in Qwen, versus 0.017 for the covariance-control mean and 0.049 for its maximum.The Qwen index is 7.0× the best matched-spectrum control.
- Readout and edit effects rise after layer 16 and peak around layers 18–20, although large MLP writes there are not necessary.
- Figure 5 shows smooth source-choice changes when activation coordinates move toward the opposite authority coordinate, while the authority direction separates choices more strongly than an orthogonal component.
- Rank two recovers 0.827 of rank four with Fieller 95% CI [0.754, 0.900], but fails after shuffling labels and data.
C.3 State/country task: representation versus causal use
The state/country task separates authority representation from causal use by evaluating cross-domain direction transfer with matched interventions.
- The experiment uses single-token state and multi-token country facts because they satisfy memory, agreement-accuracy, and authority-gap requirements.
- Facts are split into 2,048 nonconflicting prompts, 2,048 conflicts, and 512 context-free memory prompts.
- Transfers compare natural behavior with matched single-layer interventions on the authority direction, an orthogonal rank-two subspace, and their joint subspace.
- Table 5 reports reciprocal single-layer authority encoding and causal effects for the state/country task.