Source-linked AI summary
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A Huerta, Hao Peng
TL;DR
Long-context LLM performance is often attributed to retrieval failures, but it remains unclear whether perfect retrieval should make long inputs as effective as short ones. The paper uses controlled experiments across models and tasks to isolate input-length effects and tests a retrieve-then-reason mitigation. Performance still degrades with longer inputs despite perfect retrieval, while the mitigation improves GPT-4o on RULER by up to 4%.
Problem
Long-context research often separates retrieval from problem solving, leaving unclear whether perfect retrieval is sufficient for short-context-level performance.
Method
Controlled experiments make retrieval easy to verify using consecutive evidence, exact-match recitation, and varied long-context tasks, then prepend recited evidence before solving.
Results
Performance degrades substantially across five models and math, question answering, and coding tasks despite perfect retrieval; GPT-4o improves by up to 4% on RULER with retrieve-then-reason prompting.
Takeaways & Limitations
Input length alone can hurt performance independently of retrieval quality and distraction, so long-context ability should be evaluated beyond retrieval.
Takeaways & Limitations
The method requires perfect retrieval, which is difficult in many real-world retrieval settings, especially those harder than the paper’s synthetic data.
Abstract
from arXiv · showhide
Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. This gap is commonly attributed to retrieval failures -- the models' inability to identify relevant information in the long inputs. Accordingly, recent efforts often focus on evaluating and improving LLMs' retrieval performance: if retrieval is perfect, a model should, in principle, perform just as well on a long input as it does on a short one -- or should it? This paper presents findings that the answer to this question may be negative. Our systematic experiments across 5 open- and closed-source LLMs on math, question answering, and coding tasks reveal that, even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%--85%) as input length increases but remains well within the models' claimed lengths. This failure occurs even when the irrelevant tokens are replaced with minimally distracting whitespace, and, more surprisingly, when they are all masked and the models are forced to attend only to the relevant tokens. A similar performance drop is observed when all relevant evidence is placed immediately before the question. Our findings reveal a previously-unrealized limitation: the sheer length of the input alone can hurt LLM performance, independent of retrieval quality and without any distraction. They motivate our simple, model-agnostic mitigation strategy that transforms a long-context task into a short-context one by prompting the model to recite the retrieved evidence before attempting to solve the problem. On RULER, we observe a consistent improvement of GPT-4o up to 4% on an already strong baseline.
1 Introduction
Long-context performance may degrade even when models retrieve all relevant evidence, challenging retrieval-centered explanations and motivating holistic evaluation. Controlled experiments identify input length itself as a limitation and propose a simple retrieve-then-reason mitigation.
- Motivation: Models’ expanding context windows have not consistently produced comparable ability to solve problems over long inputs.Long-context tasks include integrating information across books, code repositories, and long-horizon conversations.
- Motivation: Retrieval is commonly treated as identifying and reciting relevant text from the context, distinct from search-engine retrieval in RAG.This paper uses retrieval to mean attending to and reciting specific context text.
- Findings: Whitespace can still produce wrong answers after correct evidence extraction, and similar degradation occurs when evidence is immediately before the question.The whitespace example inserts 25000 spaces with minimal distraction.
- Findings: Perfect retrieval does not guarantee short-context-level performance as input length increases.Experiments across five open- and closed-source models cover math, question answering, and code generation.
- Findings: 24.2% accuracy drop occurs for Llama-3.1-8B Instruct on MMLU problems extended to 30k tokens, despite exact retrieval on 970 of 1000 problems.The model’s retrieval performance matched that on the original shorter inputs.
- Mitigation: Prompting models to recite retrieved evidence before solving forms a shorter prompt and improves GPT-4o performance on RULER by up to 4%.The strategy is model-agnostic and converts a long-context task into a short-context one.
- Implications: Long-context evaluations should assess performance holistically because retrieval improvements alone may not translate into better long-context performance.The findings also offer a potential explanation for performance saturation or degradation as more RAG documents are added.
2 Background and Related Work
Prior work commonly separates long-context solving into retrieval and problem solving, motivating retrieval-focused benchmarks. Controlled analysis is needed to distinguish retrieval, distraction, context length, and task complexity.
- Prior framing: Long-context capability is commonly divided into retrieving relevant information and solving the task with that evidence.This decomposition motivates retrieval-based evaluations such as needle-in-a-haystack and passkey tests.
- Known challenges: Long-context failures can also arise when evidence must be aggregated, has low lexical overlap, appears in the middle, or is surrounded by distracting tokens.These factors complicate interpreting retrieval performance as a complete account of task success.
- Research gap: Carefully controlled experiments are needed to disentangle context length, token-level distraction, and task complexity.The paper frames this as essential for uncovering current models’ limits and addressing whether perfect information access suffices.
3 Measuring Long-context Performance under Perfect Retrieval
Controlled experiments test whether models can solve long-context tasks as effectively as short-context tasks when all required evidence is perfectly retrieved. Across diverse tasks and models, problem-solving accuracy declines substantially with longer inputs even while retrieval remains strong.
- Research question: The benchmark asks whether perfect retrieval preserves short-context problem-solving performance as context length increases.Long-context tasks are constructed from short-context problems by separating evidence and questions, then inserting distraction tokens.
- Main finding: Across five open- and closed-source models, performance drops substantially even when all evidence is retrieved with 100% exact match.The study measures retrieval on the evidence for the exact problems being solved, isolating retrieval failures from context-length effects.
- Benchmark design: Evidence is kept contiguous at the input beginning, while distraction tokens are inserted before the question to make retrieval comparatively easy to control.The design avoids scattered evidence and uses the beginning position, described as the easiest location to retrieve.
- Tasks: The synthetic benchmark covers math, question answering, code generation, and a simple variable-summation task.The evaluation uses GSM8K, MMLU, HumanEval, and a synthetic Variable Summation task.
- Measurement: Retrieval is measured by exact-match recitation of the evidence and question, separately from problem-solving evaluation.A retrieval failure occurs when the recited output differs from the original text; solving is evaluated without first requesting recitation.
- Main finding: Accuracy declines more sharply than retrieval, with much of the problem-solving drop occurring by 7K tokens, below the lengths where retrieval begins degrading.For Variable Summation, Llama is 59% below its 96% baseline and Mistral is 44% below its 68% zero-context score, while retrieval drops 8% and 2%.
- Interpretation: The results indicate that longer inputs can degrade reasoning, question answering, and coding even when retrieval is perfect.The findings separate the ability to retrieve relevant information from the ability to use it effectively in long-context settings.
4 Models Struggle Even Without Distraction
Controlled experiments show that increasing context length degrades performance even when irrelevant content is minimized or masked and evidence remains adjacent to the question. The effect appears across open-source models, while closed-source models generally show smaller declines.
- Minimum distraction: Whitespace tokens still produce substantial performance drops for Llama and Mistral across tasks.At 30K space tokens, the decline is at least 7%, including 48% for Llama on VarSum and 30% for Mistral on GSM8K.
- Closed-source models: Closed-source models generally experience smaller drops as context length increases than the tested smaller open-source models.GPT-4o and Gemini-2.0 achieve perfect VarSum performance throughout, while some closed-source results improve at longer lengths.
- Minimum distraction: Up to 17% for Mistral and 20% for Llama performance declines when evidence is moved immediately before the question.This controls the changing distance between evidence and question as context length increases.
- No distraction: Performance degradation persists when distraction tokens are masked and the model attends only to evidence and the question.The masked setup preserves the relevant content while increasing the distance between evidence and question.
- Conclusion: Llama and Mistral suffer common performance degradation with perfect retrieval and minimum or zero distraction.The results indicate that input length itself can hurt performance independently of retrieval quality and distraction.
5 Shortening Input Through Retrieval: A Simple Fix
The paper proposes Retrieve then Solve, which recites relevant evidence and uses it with the problem in a new shorter prompt. Experiments show improvements on synthetic GSM8K and RULER QA tasks.
- Strategy: Retrieve then Solve first recites relevant evidence, then concatenates it with the problem into a shorter prompt for solving.The model solves using the recited evidence without the original long context.
- Assumption: The strategy assumes accurate retrieval and does not itself address the retrieval problem.Its purpose is to reduce the context presented during reasoning after evidence has been retrieved.
- Synthetic benchmark: 30% boost is achieved for Mistral-v0.3-7B Instruct until the synthetic input reaches 26K tokens.The method keeps the performance gap below 10% under longer contexts without excessive prompt engineering.
- RULER: GPT-4o reaches 92.2% on RULER QA1 at 4K tokens, while achieving a maximum 4% improvement on QA2 at 32K.The QA1 baseline already ranges from 88.2–90.4% across the tested context lengths.
- Implication: The model-agnostic approach helps convert accurate retrieval from long contexts into gains on long-context tasks.The authors describe it as helping close the gap between retrieval performance and task performance.
6 Discussion
The discussion argues that context-length degradation remains after retrieval succeeds and distractions are minimized, challenging a simple retrieval-versus-reasoning decomposition. It connects the finding to RAG document overload and excessively long chains of thought.
- Implications: Performance can degrade despite perfect retrieval and minimum or zero distraction, including when evidence is immediately before the question.The discussion identifies longer input as a practical risk in settings such as chatbot dialogues.
- Implications: The results suggest that the two-part decomposition of long-context solving into retrieval and problem solving is inconclusive.The authors call for finer-grained evaluation of distinct failure modes.
- Related applications: The findings support prior observations that RAG performance can suffer when too many documents are retrieved.They also align with reports that excessively long chains of thought can hurt reasoning models.
7 Conclusion
The conclusion identifies input length itself as a previously underexplored source of language-model degradation, even with perfect retrieval and no distractions. It presents context reduction through retrieval as an initial mitigation.
- Finding: Input length itself may degrade language-model performance even when all relevant information is retrieved and distractions are removed.This limitation is presented as challenging the common decomposition of long-context solving into retrieval and problem solving.
- Mitigation: Reducing context length through retrieval can mitigate the degradation and begin bridging retrieval performance with long-context task performance.The authors characterize the strategy as simple yet effective and encourage broader model designs and evaluations.
Limitations
The study’s conclusions are limited to two open-source models, three closed-source models, and four tasks, with some setting combinations untested. The proposed mitigation also requires perfect retrieval and does not address retrieval failures in harder real-world or RULER settings.
- The conclusions cover two open-source models, three closed-source models, and four tasks, despite efforts to select representative settings.
- Some setting combinations were not tested, including retrieval on closed-source models because models sometimes refused to recite evidence under long inputs.
- The proposed mitigation requires perfect retrieval, which is difficult in many real-world tasks with harder retrieval settings than the synthetic data.
- Open-source results on RULER were not reported because retrieval failure directly causes performance drops and was outside this paper’s focus.
Declaration of generative AI and AI-assisted technologies
The authors used OpenAI tools to assist with coding and paper editing, then inspected and edited the generated content. They retain responsibility for the work’s content.
- OpenAI’s Codex, GPT-4o, o3, and ChatGPT 5 assisted with coding, spell and grammar checking, word selection, and paper rephrasing.
- The authors inspected the generated content, edited it as needed, and took full responsibility for the work’s content.
A.1 System Prompt
The appendix documents the prompts, detailed evaluation artifacts, datasets, computational resources, and potential risks used or reported in the work.
- A.1 System Prompt: The system-prompt materials include GSM8K, VarSum, and MMLU problem prompts, along with corresponding retrieval-task prompts.
- A.3 Scientific Artifacts Used in this Work: The datasets are listed in Table 10, are used only for research, and are not redistributed.
- A.3 Scientific Artifacts Used in this Work: The retrieval-before-reasoning strategy is illustrated through a RULER prompt for QA1 and QA2 and an accompanying example.
- A.2 Detailed Results: Detailed results report problem-solving accuracy and retrieval exact-match scores, including baseline values and deltas across inserted context lengths.
- A.2 Detailed Results: Additional tables report performance changes with whitespace inserted between evidence and questions, before evidence, and under masking conditions.
- A.3 Scientific Artifacts Used in this Work: The datasets contain no sensitive or offensive content.
- A.4 Computational Resources and Experiment Statistics of this Work: Experiments used GH200 GPUs and approximately 20,000 GPU hours, with every experiment run once for the reported results.
- A.5 Potential Risks: The study presents a mitigation strategy mainly for demonstration purposes, and its reliability is not guaranteed; the explored degradation is considered unlikely to create risks.