Source-linked AI summary
On the Position Bias of Large Language Models: A Comprehensive Study
Yining Li, Huayang Li, Lemao Liu, Taro Watanabe
TL;DR
Position bias makes LLM performance fluctuate with input ordering, but a comprehensive picture across modern models and languages remains elusive. The study evaluates model scales across multiple-choice and multi-document tasks in five languages, finding that scaling and reasoning mitigate the bias while thinking models show nearly uniform performance across orderings.
Problem
Position bias makes LLM performance fluctuate with input ordering rather than semantic content, and its behavior across modern systems remains unclear.
Source: §1 · p. 1Method
The study evaluates models from small to frontier scale on multiple-choice and multi-document tasks across five languages.
Source: §1 · p. 2Results
Takeaways & Limitations
Small-scale model evaluations require care because severe answer-option bias may fail to reliably reflect model intelligence.
Source: §6 · p. 8Scaling and test-time computation help mitigate position bias, but efficient solutions remain appealing in computation-constrained scenarios.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Recent studies have shown that position bias remains a critical vulnerability in large language models (LLMs), undermining both the integrity of multiple-choice evaluations and the reliability of retrieval-augmented generation. However, existing understanding of this phenomenon is largely derived from studies with limited model coverage and English-centric benchmarks, leaving the behavior of frontier models and diverse languages underexplored. In this work, we conduct a comprehensive investigation of position bias across model scales ranging from billion-level to trillion-level parameters, covering both multiple-choice and multi-document tasks in five languages. Our experiments reveal that while position bias remains noticeable in small-scale models and Non-English languages, increasing model scale and employing standard Chain-of-Thought prompting significantly mitigate this issue. Moreover, we demonstrate that emerging \emph{thinking models} with extended reasoning capabilities are nearly immune to position bias, suggesting a costly but general approach toward resolving this vulnerability.
1 Introduction
Position bias makes LLM performance depend on input order rather than semantic content, threatening evaluation validity and real-world reliability. This study evaluates the issue across model scales, task types, and languages, finding that scaling and Chain-of-Thought prompting mitigate it while thinking models are nearly immune.
Introduction
- Position bias is a performance shift driven by input order rather than semantic content, threatening multiple-choice evaluation and retrieval-augmented generation reliability.The paper also identifies long-context systems as a practical concern.Source: §1 · p. 1
- Prior work leaves modern position bias incompletely characterized, with limited frontier-model coverage, little study of thinking models, and English-centric evaluation.Typologically diverse languages such as Arabic, Chinese, and Japanese remain underexplored.Source: §1 · pp. 1–2
- Position bias is pronounced in small models and non-English languages, but scaling and zero-shot Chain-of-Thought reduce it while thinking models are almost immune.Thinking models achieve near-uniform performance regardless of information ordering.Source: §1 · p. 2
- The evaluation spans multiple-choice and multi-document tasks, five languages, and models from 1.7B parameters to frontier systems.The languages are English, Chinese, Arabic, Spanish, and Japanese; the frontier models include DeepSeek-V4 and GPT-5.2.Source: §1 · p. 2
2 Problem Formulation
The section formalizes position bias by permuting information segments in a query context and treating stable performance across orders as the ideal. It specifies distinct procedures for multiple-choice and multi-document tasks and evaluates exact match alongside the largest across-position performance difference.
2.1 General Definition of Position Bias
- A query and K information segments are linearized in a permuted order; ideally, model performance remains stable across orders with the semantic content unchanged.The segments can be retrieved documents or answer options.Source: §2.1 · p. 2
- Position bias occurs when performance fluctuates significantly with the ground-truth segment’s position in the linearized sequence.Source: §2.1 · p. 2
2.2 Multi-Choice Tasks
- Severe position bias in multiple-choice benchmarks could make systematic reshaping of LLM evaluation necessary.These benchmarks are used to assess model performance.Source: §2.2 · p. 2
- The multiple-choice procedure shifts the ground option across all candidate positions to analyze sensitivity to answer ordering.Options are typically labeled “(A)” through “(D)”.Source: §2.2 · p. 3
2.3 Multi-Doc Tasks
- Multi-document tasks matter for applications such as retrieval-augmented generation and long-context modeling, and place long documents before the question with one answer-containing document.The procedure shifts that golden document across sampled positions.Source: §2.3 · p. 3
2.4 Evaluation Metrics
- Exact match measures performance under a specific permutation, while the largest performance difference across positions summarizes positional variation.P denotes the permutation set.Source: §2.4 · p. 3
3 Multi-Choice Question Answering
The section tests multiple-choice position bias by fixing the correct answer at each option label, then compares models, prompting settings, and languages. Bias is reduced by larger models and CoT prompting, but remains pronounced in smaller models and non-English settings; thinking models further reduce it.
3.1 Setup
- The evaluation fixes the correct answer at each of four labels, alphabetizes the options, and compares performance across runs on MMLU and four additional languages.The non-English evaluation samples 5,700 examples per language; the English MMLU runs cover all 57 subtasks.Source: §3.1 · p. 3
3.2 Main Results
- On MMLU English multiple-choice tasks using non-thinking models, Qwen3-1.7B’s EM score drops 56.9 points when the golden-answer position shifts from “(A)” to “(B)”; larger models show less position bias.Source: §3.2 · pp. 4–5, Figure 1 · p. 4
- 5-shot DA examples increase position bias for Qwen3-1.7B, Qwen3-30B-MoE, and Deepseek-V3.2, although their influence is marginal on Deepseek V4 and GPT-5.2.The findings suggest position bias can be influenced by prompt context, not solely by the model.Source: §3.2 · p. 5
- 5-shot CoT produces a more uniform performance distribution across option labels but does not improve overall performance.The authors report low-level position bias with 5-shot CoT examples.Source: §3.2 · p. 5
3.3 Results on Multilingual Data
- Even with 0-shot CoT, many models retain severe position bias on non-English data; Qwen3-30B-MoE fluctuates in Arabic, Japanese, and Spanish.Deepseek-V3.2 and GPT-5.2 also show more pronounced bias in non-English languages.Source: §3.3 · p. 5, Table 2 · p. 5
3.4 Results on the Thinking Model
- Deepseek-V3.2-Thinking exhibits less position bias than its standard counterpart, while DeepSeek-V4 has little room for further reduction and lower EM in thinking mode.The authors hypothesize that over-thinking on simple questions drives DeepSeek-V4’s lower EM score.Source: §3.4 · p. 5
4 Multi-Document Tasks
The section evaluates whether position bias persists in multi-document QA across model scales and languages. Across-position performance differences generally decrease with model size, non-English results vary more across models, and reasoning models show more uniform performance across document positions.
4.1 Setup
- The evaluation uses MLQA to test cross-linguistic effects, with 20 documents per question and the golden document placed at five positions.It samples 500 instances per language and uses Exact Match as the primary metric across models ranging from Qwen3-1.7B to GPT-5.2.Source: §4.1 · pp. 5–6
4.2 Main Results
- Across-position performance differences generally decrease as model parameter counts rise, although some outliers remain.The study reports the largest performance difference across positions for each model as Δθ.Source: §4.2 · p. 6, Table 3 · p. 7
- English performance curves overlap more across models, while non-English curves are more scattered; most models’ performance gaps remain within 8 percentage points, except DeepSeek-V3.2.The greater scatter indicates substantial variation in multilingual capabilities across models.Source: §4.2 · p. 7, Figure 4 · p. 7
4.3 Results on Reasoning Models
- Reasoning models show more uniform performance across document positions than non-reasoning models.The authors suggest that allocating more tokens for explicit reasoning is an effective general approach to mitigating position bias.Source: §4.3 · p. 7, Figure 5 · p. 8
5 Related Work
Prior work identifies position bias in both retrieval-augmented generation and multiple-choice answering, with consequences for evaluation and system reliability. These findings motivate a broader study across models and settings.
Related work
- In retrieval-augmented generation, retrieved-document order can bias outputs, and placing the golden document in the middle has been associated with a performance drop exceeding 20 points for GPT-3.5.The passage also notes efforts to understand and mitigate this phenomenon.Source: §5 · p. 8
- Prior studies report that answer-option permutations can affect multiple-choice performance, and that position bias also occurs in multimodal language models.The cited work spans multiple-choice and multimodal settings.Source: §5 · p. 8
- Because multiple-choice benchmarks are widely used and multi-document systems are widely adopted, position bias raises concerns about evaluation validity, accuracy, and trustworthiness.These concerns motivate studying a broader range of models rather than only settings such as LLaMA-2 or GPT-3.5.Source: §5 · p. 8
6 Conclusions
Position bias is more severe in models under 32B parameters, while scaling and longer reasoning mitigate it; thinking mode yields nearly position-independent performance. Few-shot examples can improve performance, but uniform option-label demonstrations may worsen bias in models below 1T parameters.
6 Conclusions
- Thinking mode produces nearly uniform performance largely unaffected by information position.The conclusion describes this as generating a long reasoning chain before answering.Source: §6 · p. 8
- Models below 32B parameters generally show more severe position bias across tested languages, while scaling and more reasoning tokens significantly alleviate it.Source: §6 · p. 8
- Uniform option-label distributions in few-shot demonstrations may exacerbate position bias in models below 1T parameters, despite few-shot examples improving overall performance.Source: §6 · p. 8
- Evaluation protocols for small models require care because answer-option bias may make their measured intelligence unreliable.The authors also note that scaling and test-time computation help mitigate position bias, while efficient solutions remain desirable when computation is constrained.Source: §6 · p. 8
A Additional Results
Most models in multi-document QA perform better when the golden document is at the beginning or end of the context than when it is in the middle, unlike the first-position advantage observed in multiple-choice tasks.
The U-Shape
- Most models show a U-shaped multi-document QA pattern: golden documents at the beginning or end generally yield better performance than in the middle.This contrasts with multiple-choice tasks, where the first position generally performs significantly better than all others.Source: A Additional Results · p. 11
- The multi-document U-shape is consistent with Liu et al. (2024a).Source: A Additional Results · p. 11
- In multiple-choice tasks, the first position generally achieves significantly higher performance than every other position.Source: A Additional Results · p. 11
B Dataset Construction Details
The multilingual multi-document evaluation adapts Liu et al. (2024b)'s setup using MLQA, with 20 documents and five tested positions for the answer-containing document.
B Dataset Construction Details
- Each question uses 20 documents, with the golden document placed at positions 0, 4, 9, 14, or 19.The setup follows LITM; each question has one answer-containing document and independent distractor documents.Source: B Dataset Construction Details · p. 11
- MLQA supplies question-answer-golden-document alignments, and 19 irrelevant documents are randomly sampled from the same dataset for each question.Source: B Dataset Construction Details · p. 11
- The evaluation constructs English, Arabic, Chinese, and Spanish datasets, sampling 500 examples per language and testing all five golden-document positions.Each experimental run therefore contains 2,500 examples.Source: B Dataset Construction Details · p. 11
C Additional Analysis of Thinking Models
A controlled DeepSeek-V3.2 comparison examines errors where thinking models outperform non-thinking models. In 66.1% of those cases, their advantage is attributed to handling multiple relevant or conflicting documents and identifying relevant documents.
C Additional Analysis of Thinking Models
- The analysis compares thinking and non-thinking versions of DeepSeek-V3.2 and classifies cases of thinking-model advantage into three error types made by the non-thinking baseline.Source: C Additional Analysis of Thinking Models · p. 11
- Table 4 reports the distribution of error types among cases where thinking models outperform non-thinking models.Source: Table 4 · p. 11
- In 66.1% of cases where thinking models outperform non-thinking models, the advantage is attributed to summarizing relevant documents, comparing conflicting information, and identifying relevant documents.Source: C Additional Analysis of Thinking Models · p. 11
C.1 Definition of Error Types
The section defines three error types for comparing non-thinking and thinking models, then selects questions where the thinking model outperformed the non-thinking model.
C.1 Definition of Error Types
- Conflicting Docs describes failure by the non-thinking model to compare or reconcile conflicting information in the context.Source: §C.1 · p. 11
- Relevance Identification describes outputs that cite supporting documents but omit the most relevant one.Source: §C.1 · p. 11
- Unknown applies when neither the model-generated answer nor the reference answer provides explicit clues explaining the thinking model’s better performance.Source: §C.1 · p. 11
C.2 Data Selection Criteria
- The selection identified 65 questions where the thinking model outperformed the non-thinking model by scoring responses across five golden-document positions.Completely incorrect responses scored 0; each additional correct response earned +1, up to 5.Source: §C.2 · p. 11
D Limitations
The study’s scope is constrained by language and prompt variety, and thinking models carry higher inference-time computation and latency. The paper also notes evaluation costs and the risk of misleading performance in high-stakes deployment.
D Limitations
- Five evaluated languages may not represent extremely low-resource languages with distinct syntactic structures, and limited prompt variety also constrains the study.Source: D Limitations · p. 12
- Thinking models mitigate bias at the cost of increased inference-time computation and latency, which may be impractical for real-time applications.Source: D Limitations · p. 12
E Ethical Considerations
- Evaluating extremely large models across permutations and reasoning modes has environmental and computational costs; down-sampled datasets were used where consistent trends were observed.Source: E Ethical Considerations · p. 12
- Position bias can produce misleading performance, risking unreliable systems in high-stakes domains; the authors encourage robust evaluation protocols rather than strategic information placement.Source: E Ethical Considerations · p. 12