Source-linked AI summary
Racing Thoughts: Explaining Contextualization Errors in Large Language Models
Michael A. Lepori, Michael C. Mozer, Asma Ghandeharioun
TL;DR
LLMs’ contextualization mechanisms and failure modes remain poorly understood, including errors where a model misinterprets a context-dependent entity. The paper proposes the LLM Race Conditions Hypothesis and tests it with behavioral experiments and mechanistic interpretability analyses. The evidence supports a critical contextualization window and motivates inference-time interventions, while the account does not cover all contextualization errors and current interventions require information known a priori.
Problem
The paper addresses limited understanding of the algorithm-level factors behind successful or failed LLM contextualization, a capability central to language tasks.
Method
The authors construct context-dependent question-answering tasks with distractors and use mechanistic interpretability analyses, causal attention interventions, and representation patching.
Results
The evidence supports a critical window in which question tokens integrate contextual information and shows that contextualization no longer affects downstream performance after the middle layers.
Takeaways & Limitations
The hypothesis motivates recurrent architectural changes and more advanced inference-time interventions for addressing contextualization errors.
Takeaways & Limitations
The hypothesis does not comprehensively explain all contextualization errors, and current interventions require knowing which entities and layers to patch in advance.
Abstract
from arXiv · showhide
The profound success of transformer-based language models can largely be attributed to their ability to integrate relevant contextual information from an input sequence in order to generate a response or complete a task. However, we know very little about the algorithms that a model employs to implement this capability, nor do we understand their failure modes. For example, given the prompt "John is going fishing, so he walks over to the bank. Can he make an ATM transaction?", a model may incorrectly respond "Yes" if it has not properly contextualized "bank" as a geographical feature, rather than a financial institution. We propose the LLM Race Conditions Hypothesis as an explanation of contextualization errors of this form. This hypothesis identifies dependencies between tokens (e.g., "bank" must be properly contextualized before the final token, "?", integrates information from "bank"), and claims that contextualization errors are a result of violating these dependencies. Using a variety of techniques from mechanistic intepretability, we provide correlational and causal evidence in support of the hypothesis, and suggest inference-time interventions to address it.
1 Introduction
The paper addresses limited understanding of the algorithm-level factors behind successful or failed contextualization in LLMs. It proposes the LLM Race Conditions Hypothesis, arguing that out-of-order token contextualization can produce inconsistent generations, and supports it with mechanistic interpretability evidence and proposed interventions.
- LLMs can interpret the same occurrence of “bank” inconsistently, shifting from a geographical feature to a financial institution within the same context.
- The paper proposes the LLM Race Conditions Hypothesis, in which dependent token contextualization operations must occur in a particular layer order.For example, contextualizing “bank” must precede contextualizing the final question-mark token that reads from it.
- The authors use mechanistic interpretability techniques to provide qualitative and quantitative evidence for the hypothesis and propose inference-time interventions.
- The work identifies a robust contextualization failure mode affecting state-of-the-art models across three datasets.
2 Methods
The study constructs context-dependent question-answering tasks in which cues disambiguate subject entities and distractors systematically increase difficulty. It analyzes these tasks with multiple open-weight Gemma models and mechanistic tools including Patchscopes.
- The task combines a subject entity, a contextualizing cue, a cue-dependent yes/no question, and WikiText distractors that increase difficulty.
- Performance is measured on question pairs, counted correct only when both answers are accurate, controlling for bias toward affirmative responses.
- The datasets test contextual cue integration for polysemous words, overwritten facts, and gender bias across systematically varied distractor counts and cue positions.The polysemous-word dataset contains 120 question pairs per distractor count and cue position; the facts dataset contains 741.
- Patchscopes inspect intermediate transformer representations by copying a source residual-stream state into a target prompt and position at inference time.
- The primary model is gemma-2-9b-it, with replication using another similarly sized open-weight model and the smaller gemma-2-2b-it.
3 Behavioral Failure Modes
The paper stress-tests contextualization by inserting randomly selected distractors into context-dependent prompts. Across three domains, distractors substantially reduce models’ ability to integrate relevant cues, with performance also depending on cue position.
- Distractor text greatly harms contextualization across all three evaluated domains, producing sharp accuracy declines as distractors are added.
- The study evaluates six datasets per domain, containing between zero and five randomly selected distractors.
- Model performance varies depending on where the relevant cue is interleaved among distractor sentences.
4 The LLM Race Conditions Hypothesis
The LLM Race Conditions Hypothesis proposes that contextualization is a race between subject-entity processing and question-token processing, with errors arising when question tokens read unfinished representations. Evidence identifies a middle-layer critical window and shows that interventions to improve subject-entity contextualization can improve performance.
- 4 The LLM Race Conditions Hypothesis: The LLM Race Conditions Hypothesis states that question tokens can produce incorrect answers when they read the subject entity before its contextualization is complete.The hypothesis separates subject-entity contextualization with the cue from question-token contextualization with the subject entity.
- 4.1 A Critical Window: Attention to subject entities peaks in middle layers, suggesting that subsequent question tokens integrate subject information primarily during an intermediate processing window.Attention mass is computed from subsequent tokens to the final subject-entity token across heads and layers.
- 4.1 A Critical Window: Around layer 20, the model’s ultimate yes/no answer becomes identifiable, while success and failure show no difference in earlier layers.The logit-lens comparison controls for intrinsic affirmative or negative answer biases across layers.
- 4.1 A Critical Window: Cue ablation decreases performance and distractor ablation increases performance only in the first half of layers, after which contextualization interventions no longer affect outcomes.This causal pattern aligns the intervention boundary with the point at which the final question token’s decision becomes decipherable.
- 4.2 Subject Entity Contextualization: Open-ended interpretations show that distractors delay subject-entity contextualization and that incorrect answers have poorer contextualization than correct answers.The analysis evaluates semantic interpretations of polysemous subject entities over layers.
- 4.2 Subject Entity Contextualization: Cross-patching clean subject representations into distractor prompts and backpatching later-layer representations into earlier layers both improve performance across datasets.A modest improvement from a separate intervention was far smaller than the main experimental interventions.
5 Related Work
The paper situates its analysis within mechanistic interpretability, research on in-context learning and knowledge conflicts, and work identifying distinct processing stages in language models.
- Mechanistic interpretability: The work uses mechanistic interpretability and patching to study algorithms implemented in production-level large language models.It builds on prior mechanistic-interpretability and patching research.
- Context-dependent generation: The study complements research on in-context learning by examining contextualization in more natural context-dependent generation settings.The related work also connects contextualization to studies of knowledge conflicts.
- Processing stages: The paper contributes to research on distinct processing stages by identifying a contextualization process distributed across model layers.Related work has localized other mechanisms and proposed universal processing stages.
6 Conclusion
The paper tests the LLM Race Conditions Hypothesis and argues that contextualization failures arise from timing constraints in feedforward models. It identifies mechanistic interpretability and architectural changes as possible routes toward mitigation.
- 6 Conclusion: The authors test two predictions: a critical window for question-token contextualization and failures caused by unfinished subject-entity contextualization during that window.These predictions organize the paper’s evaluation of the proposed hypothesis.
- 6 Conclusion: The proposed account connects contextualization errors to the feedforward architecture, whose early layers cannot access later-layer representations.This architectural constraint is presented as a natural consequence of the hypothesis.
- 6 Conclusion: The paper suggests recurrent connections and more advanced inference-time interventions as potential ways to address the problem.The proposed interventions are framed as work in progress rather than established solutions.
7 Limitations
The study explains one form of contextualization error across diverse datasets but does not claim to cover all such errors. Its proposed interventions are not yet practical for production use.
- 7 Limitations: The hypothesis does not comprehensively explain all LLM contextualization errors, including errors caused by missing factual knowledge.The paper therefore bounds its claim to one mechanistically analyzed error type.
- 7 Limitations: The proposed inference-time interventions currently require knowing which entities and layers to patch in advance, limiting their practical deployment.The authors identify further development as necessary before production use.
8 Ethical Considerations
The ethical discussion highlights that contextualization errors can occur in gender-bias settings and cautions against using the presented interventions to address production gender bias. The evaluation uses distractor-based partitions and examples across several datasets.
- 8 Ethical Considerations: Contextualization errors occur in settings where models are likely to encode gender bias associated with a profession.The paper relates this observation to longstanding work on gender biases in language models.
- 8 Ethical Considerations: The authors explicitly caution that none of their interventions should be used to address gender bias in production language models.This is a direct scope boundary on the practical interpretation of the interventions.
- 8 Ethical Considerations: The study evaluates contextualization with fixed numbers and positions of distractors, choosing partitions to bring model accuracy close to 50% for intervention analyses.This design supports detecting both positive and negative intervention effects.
- 8 Ethical Considerations: The materials include examples involving factual overrides, gendered professions, and context-dependent questions, with answers shown for readers but excluded from model prompts.The supplied examples span the evaluated datasets and distinguish presentation from model input.
C Disaggregated Logit Lens Results
Disaggregated logit-lens results show changing yes/no preferences across layers and reveal when successful and failed contextualization cases begin to diverge. The analysis uses intermediate final-token representations and separates outcomes by correctness and ground-truth label.
- C Disaggregated Logit Lens Results: The analysis applies the logit lens to intermediate final-token representations and disaggregates results by correctness and ground-truth yes/no labels.Open-ended patchscopes separately generate descriptions of polysemous-word representations and use an autoscorer to assess word sense.
- C Disaggregated Logit Lens Results: The first half of layers favors “no” while the second half favors “yes”, so the final answer cannot necessarily be decoded until the last several layers.This layerwise bias makes early decoding unreliable despite later recoverability.
- C Disaggregated Logit Lens Results: Around the model midpoint, successful and failed cases diverge, with larger yes/no logit differences for questions ultimately answered “yes” and smaller differences for those answered “no”.The logit difference is uniformly positive from layers 20–30 or beyond, but its magnitude separates eventual outcomes.
- C Disaggregated Logit Lens Results: The accompanying examples report clear contextualization of subject entities over the course of layers.These examples are presented as qualitative support alongside the disaggregated logit-lens analysis.
F Pairwise Patching Results
Pairwise backpatching usually yields modest gains, but the best layer pairs depend on the question, and late-to-early patching can harm performance.
- Many layer pairs provide modest gains, while patching late representations into the earliest layers regularly harms performance, especially under frozen backpatching.
- Standard and frozen backpatching produce similar results for most layer pairs after layer 10, but frozen backpatching underperforms when patching layers 2 and 4.The extra processing of representations from layers 21-29 can increase gains over baseline more substantially.
G Replicating results on Llama-2
Llama-2 reproduces the main behavioral and intervention trends: distractors degrade performance, cue position matters, and similar interventions support the race-condition account across models.
- Llama-2 performance degrades as distractors increase and depends on cue position, with a more reliable U-shaped position curve than previously observed.
- Attention ablation and patching interventions closely replicate Gemma-2 results, supporting the presence of LLM Race Conditions across different LLMs.The replication also suggests that similar techniques can address these errors across models.
- The five-distractor analysis examines performance according to the cue’s position among distractor sentences.
H Replicating results on Gemma-2-2b
The smaller Gemma-2-2b model reproduces the larger model’s main patterns, including a middle-layer attention increase and a critical window ending near the model midpoint.
- Gemma-2-2b reproduces the larger model’s overall results on the polysemous-word dataset, with the 3-distractor, cue-index-2 partition reaching 49.2% accuracy.
- Attention to the subject entity increases across approximately layers 7-18, whether summed across heads or measured as the maximum from one head.
- Logit-lens and attention-intervention analyses place the critical-window endpoint around layer 10, while the larger Gemma-2-9b endpoint is around layer 20.This suggests the critical window may shrink with model depth rather than ending after a fixed number of processing steps.
- Patching interventions were also replicated in Gemma-2-2b, extending the replication beyond behavioral and attention analyses.