Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

Hyunjong Ok; Jaeho Lee

Source-linked AI summary

Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

Hyunjong Ok, Jaeho Lee

arXiv:2601.14152v2Jan 20, 2026cs.CLcs.AIcs.LG

TL;DR

The paper asks why language models are highly sensitive to prompt structure, focusing on the large MCQA difference between CQO and QOC despite unchanged semantic content. It tests competing explanations through controlled experiments and architectural analysis, identifying causal attention as the core mechanism. In QOC, causal masking blocks options from accessing the context, while interventions can improve QOC or degrade CQO; the paper remains diagnostic and does not provide a runtime-free training fix.

Problem
Prior research documents prompt-order sensitivity but provides limited mechanistic understanding of why certain MCQA component orderings are preferable.
Method
The paper evaluates competing hypotheses across models, datasets, architectures, and targeted interventions to explain CQO-QOC context-order sensitivity.
Results
14.72% gap separates CQO and QOC for decoder-only models, compared with 2.30% for encoder-decoder models and 0.02% for encoder-only models.
Takeaways & Limitations
Causal masking is the core mechanism: in QOC, options cannot attend to context, and targeted interventions improve QOC or degrade CQO.
Takeaways & Limitations
The paper is diagnostic in scope and does not propose a practical training-time fix that closes the gap without runtime overhead.

Abstract

from arXiv · show

Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets. Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options.

1 Introduction

The paper investigates why MCQA accuracy depends strongly on whether context precedes or follows the question and options. Across models and datasets, causal attention emerges as the mechanism behind CQO's advantage over QOC.

Motivation: LLM prompt sensitivity is well documented, but prior work largely describes effects without identifying their underlying mechanisms.This study focuses on component ordering in multiple-choice prompts as a consequential form of prompt sensitivity.
Prompt-order sensitivity: CQO places context before the question and options, whereas QOC places the question and options before context.The prompt consists of context (C), question (Q), and options (O), whose semantic content remains unchanged when reordered.
Prompt-order sensitivity: CQO consistently and substantially outperforms QOC across a wide range of models and datasets.Figure 1 compares average accuracies for 21 decoder-only LLMs across 4 datasets.
Hypotheses: The study tests biased training data, option recall failures, and causal attention as competing explanations for the ordering effect.Causal attention is identified as the winning hypothesis after systematic evaluation.
Causal-attention mechanism: In QOC, causal masking prevents option tokens from attending to the later context, creating an information bottleneck.Because decoder-only tokens attend only to preceding tokens, option representations are computed without context information.
Evidence and implications: Targeted interventions improve QOC prompts or degrade CQO prompts, providing additional evidence that causal attention drives context-order sensitivity.The experiments also identify context length and option positions as factors affecting the masking impact.

2 Experimental setup

The experiments evaluate context-order sensitivity across reading-comprehension datasets, model families, architectures, and prompting conditions. Accuracy is compared between CQO and QOC using an order-sensitivity gap.

Datasets: The evaluation uses four reading-comprehension benchmarks in multiple-choice format, each with four options.The listed benchmarks are LogiQA, RACE-H/M, and SciQ.
Controlled conditions: Instruction tuning and few-shot prompting are tested as factors that might explain the CQO-QOC gap.Figure 3 compares base and instruct models and evaluates increasing numbers of demonstrations.
Evaluation: The study measures accuracy under CQO and QOC and defines context-order sensitivity as ∆ = Acc_CQO − Acc_QOC.The performance gap quantifies the effect of prompt ordering.
Evaluation: Decoder-only, encoder-only, and encoder-decoder models are scored using protocols matched to their respective generative interfaces.The protocols include constrained option likelihood, masked-token prediction, and encoder-decoder option scoring.

3 Hypotheses and analysis

The study tests training-distribution bias, option-recall failure, and causal attention as explanations for the CQO–QOC performance gap. Controlled comparisons reject the first two and implicate causal masking, which blocks context-to-option information flow in QOC.

Hypothesis 1: Bias in training samples: Training-distribution tests found nearly identical CQO–QOC gaps across matched models, while five-shot QOC improved only 3.1%.These results rule out training distribution as the primary cause.
Hypothesis 2: Option recall: QOC achieved similar or higher option-recall accuracy than CQO, rejecting memory loss as the primary explanation for its accuracy drop.The recall test measured exact matches for each option after the prompt.
Hypothesis 3: Causal attention: Decoder-only models showed a 14.72% gap, compared with 2.30% for encoder–decoders and 0.02% for encoder-only models.The architecture pattern strongly implicates causal masking.
Hypothesis 3: Causal attention: QOC accuracy was nearly identical to QO accuracy, indicating that the context is effectively inaccessible under the causal mask.In QOC, options precede context, so their representations are computed without contextual information.
Hypothesis 3: Causal attention: QOC option-to-context attention is zero by construction, while context attribution is 0.797 in CQO versus 0.335 in QOC.Across layers, attention to options rises in QOC but decreases in CQO, consistent with differing context integration.
Modulating factors: Longer contexts increase the gap, and earlier answer positions suffer greater QOC accuracy drops than later positions.Later options are closer to the context and are therefore more robust.

4 Targeted interventions

The paper tests interventions that manipulate the option–context pathway. Repeating options, patching context-aware option states, and chain-of-thought prompting improve QOC, while pruning option-to-context attention degrades CQO.

Overall intervention evidence: Across 21 decoder-only models and four datasets, targeted pathway interventions changed performance in the predicted directions.The intervention suite was applied uniformly across the evaluated models and datasets.
Attention pruning: Blocking option-to-context attention reduced CQO accuracy from 69.26% to 42.46% (−26.8).The intervention applied the causal-mask constraint while leaving all other attention unchanged.
Activation patching: Activation patching increased QOC accuracy by 6.0 points on average.The method replaced middle-to-late-layer QOC option states with context-aware states computed under CQO.
Option repetition: Repeating options after the context improved QOC by 8.2 points without modifying model internals.The repeated options can attend to the context under the causal mask.
Chain-of-thought prompting: Chain-of-thought prompting shrank the CQO–QOC gap from 14.72 to 7.47 (−7.25) on average.The lower CoT block uses generative scoring, so the absolute accuracies are not directly comparable to the logit-scored block.

5 Conclusion

The paper attributes MCQA prompt-order sensitivity to causal attention rather than training-distribution bias or memory decay. Its architectural, attention, and intervention results provide mechanistic insight and practical inference-time guidance.

Conclusion: Causal attention is identified as the core mechanism behind the CQO–QOC gap in decoder-only LLMs.The causal mask prevents options from attending to context, rendering context effectively inaccessible in QOC.
Conclusion: Architecture comparisons, attention analysis, and targeted interventions jointly support the causal-attention explanation.The interventions both degrade CQO and improve QOC performance.

Limitations

The paper’s limitations concern both the minimal theoretical account and its diagnostic scope. It demonstrates inference-time mitigations but does not provide a training-time fix without runtime overhead.

Theoretical scope: The single-step bottleneck theory is deliberately minimal, establishing context-independence of QOC option states and intervention effects.It does not constitute a more extensive theoretical account.
Practical scope: The paper is diagnostic in scope and lacks a practical training-time fix that closes the gap without runtime overhead.Its demonstrated mitigations are inference-time methods including CoT, QOCO, and activation patching.

B.2 Dataset information

The evaluation covers context-based multiple-choice benchmarks, model architectures, prompt orderings, and likelihood-based scoring. Across 21 decoder-only models and four datasets, CQO exceeds QOC by 14.7%, with stronger context attribution.

Datasets: LogiQA contains 651 logical-reasoning test samples requiring multi-step inference over context.Its questions include categorical, conditional, and disjunctive reasoning.
Datasets: SciQ contains 1,000 crowdsourced science questions paired with short supporting passages.The questions span physics, chemistry, and biology, with passages averaging approximately 80 words.
Prompting: The study compares CQO and QOC, placing context before or after the question and options, respectively.Decoder-only prompts use open-ended completion with the answer label predicted as the next token.
Evaluation protocols: Models are scored through architecture-matched protocols, including constrained likelihood for decoder-only models and masked prediction for encoder-only models.Encoder-decoder models receive the full prompt in the encoder and generate the answer through the decoder.
Results: 14.7% is the average CQO–QOC accuracy gap across 21 decoder-only models and four datasets.Instruction tuning leaves the gap essentially unchanged: 14.70% for base models versus 14.12% for instruction-tuned models.
Results: 2.38× is the average context-attribution ratio favoring CQO across datasets.The figure reports that CQO uses context more effectively across all benchmarks.

C.7 Intervention results

Targeted interventions support the proposed attention-pathway explanation of the CQO–QOC gap. Blocking CQO option-to-context attention degrades CQO, while modifying QOC option representations or repeating options improves QOC.

Attention pruning: -26.8% is the average CQO accuracy drop when option-to-context attention is blocked.The result identifies this attention pathway as essential to CQO performance.
Activation patching: +6.0% is the average QOC improvement from replacing QOC option hidden states with CQO representations.This intervention directly targets the representations used by options in QOC prompts.
Option repetition: +8.2% is the average QOC accuracy improvement from repeating options after the context.In Q-O-C-O prompts, repeated option tokens can attend to the context without changing model internals.

D Results on other architectures

Results on alternative architectures generalize the ordering effect beyond standard softmax-attention Transformers. Strictly preceding-token access preserves the gap, whereas bidirectional access nearly removes it.

State-space models: +18.5% is FalconMamba-7B’s CQO–QOC gap, larger than the +14.7% gap for decoder-only Transformers.The competent FalconMamba-7B result implicates sequential left-to-right state compression rather than softmax attention specifically.
Hybrid attention: +20.1% is the largest observed gap, occurring on LogiQA for the left-to-right Qwen3-Next hybrid model.Qwen3-Next combines gated linear attention with softmax attention in hybrid blocks.
Diffusion language model: Dream shows a near-zero CQO–QOC gap while using bidirectional-within-block masking during diffusion-style denoising.Together, the comparisons associate the gap with strictly preceding-token access and its removal with bidirectional access.

E Closed-source model: Gemini

The closed-source Gemini evaluation tests whether the ordering effect persists at commercial scale and whether chain-of-thought reduces it. Gemini reproduces the gap, while CoT substantially narrows it.

Ordering gap: +11.84% is Gemini’s CQO–QOC ordering gap despite its higher overall accuracy.Gemini scores 87.4% with CQO and 75.5% with QOC.
Chain-of-thought: 83.4% is the relative reduction in Gemini’s ordering gap under chain-of-thought prompting.The gap decreases from +11.84 to +1.97.

F Formal analysis of the single-step bottleneck

Under QOC ordering, causal masking makes option representations context-blind, forcing evidence–option comparison into the final answer-scoring step; CQO avoids this structural bottleneck. The appendix further reports interventions and generative-task results that test and extend this account.

F Formal analysis of the single-step bottleneck: Causal masking imposes ht = fθ(x1:t), so each hidden state depends only on its preceding prefix.This is the formal left-to-right constraint used throughout the analysis.
F Formal analysis of the single-step bottleneck: Under QOC, option representations depend on Q and O alone, making them structurally independent of context information.The context appears after the options, so no context information is representable in their hidden states.
F Formal analysis of the single-step bottleneck: Under CQO, option representations can depend on C, Q, and O; attribution assigns 0.797 to context under CQO versus 0.335 under QOC.Attention to option tokens also changes in opposite directions across depth between the two orderings.
F Formal analysis of the single-step bottleneck: QOC therefore performs all evidence–option comparison through the final answer-scoring step using context-blind option representations.Later tokens can read earlier option states but cannot retroactively make them context-conditioned.
F Formal analysis of the single-step bottleneck: +8.2 points is achieved by repeating options after context in QOCO, restoring context-conditioned option states; activation patching adds +6.0 points.Chain-of-thought reduces the gap from 14.72 to 7.47, a −7.25-point change, by adding context-aware decoding steps.
F Formal analysis of the single-step bottleneck: The formal representation constraint does not itself constrain final prediction: an oracle matcher could compensate, but current decoder-only models leave a residual CoT gap of +7.47.This distinguishes the structural bottleneck from an absolute impossibility of accurate final prediction.
G Generalization beyond MCQA: open-domain QA and RAG: +15.14 F1 is observed for RAG when evidence precedes the question rather than following it, while open-domain QA compares CQ, QC, and question-only formats.The RAG setup includes one gold passage and four BM25 distractors, making the final comparison involve five competing passages.
F Formal analysis of the single-step bottleneck: Bidirectional encoder-only and encoder–decoder models have an essentially zero CQO–QOC gap because their option states can incorporate context under either template.These architectures remove the causal constraint that creates context-blind option representations.