Source-linked AI summary
Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG
Jorge Gabín, Anxo Perez, Javier Parapar
TL;DR
Findings on RAG context ordering and size remain inconsistent across models, datasets, and evaluation protocols, complicating interpretation of these design choices. This paper calibrates topic budgets and revisits position effects and retrieval-mediated RAG under controlled evaluation. It finds that topic sampling matters and that order and context-size effects vary with retrieval quality and dataset structure.
Problem
Prior studies report inconsistent effects of context ordering and size across topic sets, evaluation protocols, models, and retrieval pipelines.
Source: §1 · p. 1Method
The authors calibrate topic budgets through repeated subset sampling, then revisit position and RAG robustness claims and analyze retrieval quality, ordering, context size, and model variation.
Source: §1 · pp. 1–2Results
On single-hop NQ, performance generally improves as k increases and is comparatively order-insensitive; on multi-hop HotpotQA, larger contexts help mainly when high-value evidence is placed in favored positions.
Source: §6 · p. 9Takeaways & Limitations
The authors conclude that practitioners should prioritize retrieval quality and evidence placement over simply increasing context size.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) systems rely on retrieved documents being concatenated into a model's input context, making both document ordering and context size critical yet controversial design choices. Prior work reports position-based effects such as lost in the middle and related long-context phenomena. However, empirical findings remain inconsistent and hard to reproduce across models, datasets, and evaluation protocols. In this paper, we present a systematic reproducibility study that revisits these claims and examines how they evolve with contemporary LLMs under a controlled evaluation framework. We first show that topic sampling is a major source of variance: small topic sets can mask or exaggerate ordering effects. Based on repeated subset sampling across multiple topic budgets, we provide a practical calibration procedure that identifies topic counts yielding stable trends at feasible cost. Using these fixed topic sets, we then reproduce and extend results on position sensitivity, re-evaluating lost in the middle and positional biases in modern LLMs. Then, we also study a more realistic RAG scenario in which relevance is mediated by a retriever rather than oracle access to ground-truth documents. In this setting, we re-examine a recent industry study and identify discrepancies to evaluation choices such as limited topic coverage and reliance on LLM-based judges. Finally, we conduct an analysis of how retrieval order and context size affect downstream LLM performance under imperfect retrieval. Our results demonstrate that both factors interact strongly with retrieval quality and model choice, and that conclusions drawn from idealised setups do not always transfer to real-world RAG pipelines. We release all code and configurations to support reproducibility and future work on robust RAG evaluation.
1 Introduction
Prior findings on context size and passage order in RAG conflict, and differences in topic samples, evaluation protocols, models, and retrieval pipelines make them difficult to interpret. This study calibrates topic budgets before revisiting those effects and examining retrieval quality, ordering, context size, and model variation.
1 Introduction
- Prior studies report conflicting effects of passage order and retrieval depth, with results varying across models, datasets, topic sets, and evaluation protocols.This leaves practitioners uncertain about context-size and ordering choices.Source: §1 · p. 1
- Small, arbitrarily selected topic samples can yield unstable conclusions, while evaluating full topic sets is often impractical when comparing multiple LLMs.The study focuses on NQ, predominantly single-hop, and HotpotQA, which requires multi-hop evidence aggregation.Source: §1 · p. 1
- The authors calibrate topic budgets by repeatedly sampling subsets and identifying where variance in ordering-strategy ΔF1 is low enough to prevent ranking flips.The resulting stable topic sets are used for subsequent experiments.Source: §1 · pp. 1–2
- The study revisits long-context position and RAG robustness claims with contemporary LLMs, then tests how retrieval quality and strategy affect performance.It also examines model architectures and sizes under varying setups and releases code and configurations for reproducibility.Source: §1 · p. 2
2 Related Work
Related work documents position sensitivity, context-construction effects, and retrieval robustness, while noting that evaluation choices can shape reported findings. The paper distinguishes its study by controlling sampling and extending comparisons across ordering, context size, retrieval quality, and model families.
2 Related Work
- Prior work reports that answer-bearing evidence can perform differently by position, while passage composition and chunk count may also shape non-monotonic effects [12] [22].Other robustness studies perturb retrieval depth and ordering, finding average stability alongside instance-level trade-offs [2].Source: §2 · p. 2
- RAG findings can depend on dataset formatting, chunking, scoring methodology, and reduced topic subsets whose representativeness is rarely validated [21].Retrieved irrelevant or weakly relevant passages can distract generators, especially when prominent or placed in salient positions [1].Source: §2 · p. 2
- This study combines variance isolation with comparisons across ordering schemes, context sizes, retrieval quality, and model families and sizes.It also reproduces recent claims about weak order and depth effects under matched settings.Source: §2 · p. 2
3 Towards Stable Evaluation of Context Ordering and Size
The section develops a topic-sampling calibration procedure and applies it to establish stable topic sets for later comparisons of ordering and context size.
3.1 Investigating Sources of Variability
- Across 10 random subsets of 500 topics, the apparent ordering-strategy winner can change between samples, particularly at smaller context sizes.The analysis uses LLaMA-3.1:8B and F1-token, the official benchmark metric.Source: §3 · pp. 2–3, §3.1 · p. 3
3.2 Determining an Adequate Number of Topics
- The calibration varies sample size and measures ΔF1 variance across seeds, treating zero-crossings as evidence that sampling noise flips the winning strategy.It seeks the smallest topic budget that yields stable rankings while keeping evaluation cost manageable.Source: §3.2 · p. 4
- Using 1,000 HotpotQA topics and 2,000 NQ topics keeps standard deviation low enough to prevent most ΔF1 zero-crossings and preserve relative system rankings.The full-dataset differences serve as the reference, and these calibrated topic sets are used in subsequent experiments.Source: Figure 2 · p. 3, §3.2 · p. 4, §3.3 · p. 4
3.3 Controlled and Reproducible Experimental Setup
- The controlled topic sets support reproductions of positional and order/size claims and later comparisons across dataset structure, retrieval quality, and model family and scale.The calibration addresses the difficulty of interpreting order and context-size effects when full-topic evaluations are impractical.Source: §3 · p. 2, §3.3 · p. 4
4 Reproducibility Study of Context Position and Retrieval Robustness
The section first tests position effects with gold evidence in controlled contexts, then examines ordering and context size under imperfect retrieval. The reproduced findings are nearly flat across position in the gold-evidence setups, but show measurable order and context-size sensitivity in retrieved contexts.
4.1 Reproducing Context Position Effects
- On AmbigQA, accuracy remains comparatively flat across gold-document positions rather than forming a clear U-shaped curve, and absolute scores differ from those reported in [12].The authors note that dataset processing and document construction differ, including additional support used in the original study.Source: §4.1 · pp. 4–5
- On standard NQ and HotpotQA, performance is again nearly flat across relevant-evidence placements, with little evidence of position-specific degradation under this setup.For multi-evidence queries, relevant passages are grouped, and the first relevant passage defines the swept position.Source: §4.1 · p. 5
4 Reproducibility Study of Context Position and Retrieval Robustness
- The gold-evidence position tests are optimistic relative to RAG, so the study shifts to contexts formed by imperfect retrieval, where relevance is graded and gold documents may be missing.The reproduced retrieval setting uses the same top-k set across original, reversed, and random ordering conditions.Source: §4.2 · pp. 5–6
4.2 Reproducing Order and Size Robustness Under Retrieval
- The reproduction uses token-level F1 rather than an LLM judge; a 250-topic pilot found near-identical ordering gaps between the two scorers.The authors use F1 as the official benchmark metric and for reproducibility and comparability.Source: §4.2 · p. 6
- Under matched datasets, model family, 500-topic samples, and ordering schemes, the reproduction finds measurable F1 sensitivity to context size and order, more apparent at larger k than at low k.This differs from the strong average stability reported by Cao et al. [2].Source: §4.2 · p. 6, Figure 5 · p. 6
- The authors attribute the difficulty of reproducing average order robustness partly to small topic samples, since high per-query variance and modest order gaps make conclusions sample-dependent.The subsequent section uses calibrated topic sizes to reduce this sampling-noise concern.Source: §4.2 · p. 6
5 Controlled RAG Analysis: Ordering, Depth, Retrieval Quality, and Models
Using fixed topic sizes, the section tests ordering and context-size effects across retrieval-mediated QA settings and models. It finds that their effects vary with dataset, retrieval quality, and model.
5.1 Exploring Context Size and Ordering Effects (RQ1)
- HotpotQA increasingly favors reverse ordering as context grows, whereas NQ gains with k while remaining comparatively order-stable.The study plots F1 against context size; for HotpotQA, high-value passages later in the prompt help counter position sensitivity in multi-hop settings.Source: Figure 6 · p. 7, §5.1 · p. 7
5.2 Impact of Retrieval Quality and Reranking (RQ2 and RQ3)
- On HotpotQA, closed-book is weakest, while oracle passages and answer-bearing oracle sentences cluster just above 0.70 F1 across k.The oracle configurations estimate an upper bound attributable to evidence quality, rather than LLM capacity.Source: §5.2 · p. 7, Figure 7 · p. 8
- With gold passages present, adding standard-ranked BM25 noise lowers performance as k grows; Oracle+BM25 reverse beats standard for k≥10.The authors suggest that placing gold passages at the tail better aligns with the model’s reasoning.Source: §5.2 · p. 8
- Stronger retrieval favors shorter contexts and reduces ordering sensitivity: with BM25+E5 reranking, k∈{5, 10} performs noticeably better than BM25 alone on both datasets, while weaker BM25 benefits more from reverse ordering at higher k.Source: §5.2 · p. 8
5.3 Trends Across Model Sizes and Architectures (RQ4)
- Larger models generally show smoother, near-zero reverse-minus-standard ΔF1, but HotpotQA mid-sized models show stronger positive effects at larger k; NQ stays order-stable.The authors describe ordering sensitivity as an interaction between dataset properties and how each architecture manages long-context reasoning, rather than model size alone.Source: §5.3 · p. 9
- LLaMA-3.1:70B and Gemma-3:27B achieve the best F1 across both datasets and remain stable across topic sizes.The paper reports that increased capacity contributes to improved accuracy and greater robustness against context-size fluctuations.Source: §5.3 · p. 9
6 Conclusions
Across NQ and HotpotQA, context size and evidence ordering have dataset-dependent effects, while small topic samples can shift conclusions. The authors recommend prioritizing retrieval quality and evidence placement over simply increasing context size.
6 Conclusions
- Small topic subsets can shift conclusions about ordering and context size, making topic sampling a major source of variance.The study fixes topic sizes using repeated subset sampling to obtain stable trends at feasible cost.Source: §6 · p. 9
- On NQ, performance generally improves as k increases and is comparatively insensitive to order; on HotpotQA, larger contexts help mainly when high-value evidence occupies preferred positions.Dense reranking boosts small-k performance and narrows ordering sensitivity.Source: §6 · p. 9
- Larger models are typically more stable overall, but they do not eliminate ordering effects in multi-hop settings.Figure 9 plots reverse-versus-standard ΔF1 and separately marks each model's best F1 across orderings.Source: §6 · pp. 9–10, Figure 9 · p. 9
- The authors conclude that practitioners should prioritize retrieval quality and evidence placement over simply increasing context size.They release code and configurations for reproducible, order-aware RAG evaluation.Source: §6 · p. 10
Computational resources
The experiments used private infrastructure and A100 SXM4 80 GB hardware.
Computational resources
- 160 hours of computation on A100 SXM4 80 GB hardware generated an estimated 22.6 kgCO2eq, with 0% directly offset.The private infrastructure's carbon efficiency was 0.432 kgCO2eq/kWh.Source: Computational resources · p. 10