Source-linked AI summary
Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models
Mosh Levy, Alon Jacoby, Yoav Goldberg
TL;DR
The paper asks whether LLM reasoning remains consistent as inputs grow, especially when the relevant task and information stay fixed. It introduces FLenQA to vary input length through irrelevant padding while preserving the same two-span reasoning task. Reasoning performance drops well before technical input limits, and long-input next-word prediction does not track long-input reasoning performance.
Problem
Evidence was limited on whether LLM performance changes with input length itself, because prior studies varied input length and task requirements together.
Method
FLenQA creates multiple versions of the same two-span True/False reasoning sample by embedding its relevant context in longer irrelevant texts.
Results
Reasoning performance drops significantly before models reach their maximum input length; at 3000 tokens, average accuracy falls from 0.92 to 0.68, and next-word prediction is uncorrelated with long-input reasoning performance.
Takeaways & Limitations
Evaluating models at a single input length is insufficient; long-range capability requires maintaining performance across supported lengths.
Takeaways & Limitations
The behavioral tests leave the cause of performance degradation unexplained and use simple tasks that may overlook differences in more complex reasoning.
Abstract
from arXiv · showhide
This paper explores the impact of extending input lengths on the capabilities of Large Language Models (LLMs). Despite LLMs advancements in recent times, their performance consistency across different input lengths is not well understood. We investigate this aspect by introducing a novel QA reasoning framework, specifically designed to assess the impact of input length. We isolate the effect of input length using multiple versions of the same sample, each being extended with padding of different lengths, types and locations. Our findings show a notable degradation in LLMs' reasoning performance at much shorter input lengths than their technical maximum. We show that the degradation trend appears in every version of our dataset, although at different intensities. Additionally, our study reveals that the traditional metric of next word prediction correlates negatively with performance of LLMs' on our reasoning dataset. We analyse our results and identify failure modes that can serve as useful guides for future research, potentially informing strategies to address the limitations observed in LLMs.
1 Introduction
The paper isolates how increasing input length affects LLM reasoning while keeping the underlying task fixed. It finds substantial degradation well before models’ technical limits, alongside failure modes and a lack of correlation with next-word prediction.
- Motivation: The study asks whether long-input support transfers from short-input reasoning when the task and relevant information remain unchanged.Prior studies varied input length and task requirements together, making the source of degradation difficult to identify.
- Method: FLenQA creates multiple length versions of the same two-span True/False reasoning sample by embedding its relevant context in longer irrelevant texts.The tasks require jointly reasoning over both information pieces and are usually solved when presented without padding.
- Findings: At 3000 tokens, average accuracy drops from 0.92 to 0.68 across tested models, despite this being much shorter than their technical maximum.The dataset keeps the underlying task intact while varying irrelevant padding to isolate input length.
- Findings: Reasoning degradation appears across experimental settings that vary relevant-span locations and whether surrounding contexts are similar or dissimilar.Figure 1 reports performance across 600 samples for inputs containing relevant and incrementally added irrelevant text.
- Findings: Next-word prediction on long inputs is uncorrelated with reasoning performance on long inputs.The study measures exact next-word accuracy using the same samples without their questions.
- Findings: CoT prompting improves short-input performance in most models but generally does not mitigate the degradation caused by longer inputs.In GPT4, its effect increases with length; this exception still leaves it a limited mitigation technique.
- Failure modes: With longer inputs, models tend to ignore instructions, provide no answer, answer false, or incorporate less relevant information.Under CoT prompting, some responses present the final answer before outlining the reasoning steps.
2 Desired Data Properties
The desired dataset properties are designed to test reasoning over relevant text while isolating input length. They require jointly necessary evidence, novel facts, controlled placement, non-interfering padding, and natural-looking contexts.
- Evaluation design: The evaluation uses behavioral input intervention so it can apply to both open and closed models while keeping relevant information constant.Question answering tasks require models to reason over a given text.
- Ensuring reasoning: Each sample must contain several relevant spans that are jointly necessary and sufficient for solving the task.This excludes tasks solvable by separately identifying and combining independently usable spans.
- Ensuring reasoning: The relevant spans and questions should use novel facts to reduce reliance on memorized or contaminated parametric knowledge.The evaluation therefore targets reasoning over text that was not previously available.
- Isolating length: Length variations must preserve the relevant spans, prevent padding from contradicting or interfering with them, and allow span locations to be controlled.These constraints isolate length while preserving the reasoning requirement.
- Natural inputs: Contexts should remain natural-looking by using cohesive paragraphs rather than sequences of unrelated sentences.The intended setting resembles collecting relevant information from multiple sources.
3 FLenQA
FLenQA is a controlled QA reasoning dataset that varies input length, padding, and fact dispersion while preserving the underlying task. It contains three reasoning tasks built from two key facts and expanded into natural-looking contexts.
- Tasks: The dataset contains Monotone Relations, People In Rooms, and a simplified Ruletaker, with variations in length, background text, and fact dispersion.Each task has 100 base instances.
- Instance structure: Each base instance contains optional framing text, two thematically coherent key paragraphs, and an optional question-bearing suffix.Key paragraphs begin with sentences needed to solve the task.
- Instance construction: Key sentences are expanded into coherent paragraphs using GPT-4 and manually verified without adding new information.This expansion supports natural-looking inputs while preserving the underlying facts.
- Tasks: The three tasks require combining information across distinct reasoning formats, including transitive relations, room-property inference, and rule-based deduction.Monotone Relations asks about unstated relations between entities appearing in separate sentences.
- Length and padding: Inputs target roughly 250, 500, 1000, 2000, and 3000 tokens using duplicate, similar, or different padding and varied key-paragraph dispersion.Sample length is measured with the GPT-4 tokenizer within a ±70-token range.
4 Main Experiments
The experiments isolate input length while holding the task and prompting setup constant across five LLMs. Reasoning accuracy declines as contexts grow, with effects depending on padding relevance, key-paragraph location, and whether evidence is dispersed.
- Experimental setup: The study evaluates GPT4, GPT3.5, Gemini-Pro, Mistral Medium, and Mixtral 8x7B under the same prompt and temperature across input lengths.Outputs without an answer are counted as incorrect.
- Padding relevance: Both relevant and irrelevant duplicate padding affect performance, but GPT3.5 and GPT4 are less affected when added tokens remain relevant.Each plotted point represents 300 samples.
- Impact of length and location: Accuracy drops significantly beyond 500 tokens, while adjacent key paragraphs generally perform better and paragraphs placed last are often strongest.The latter pattern suggests recency bias, while middle placement can worsen some models’ performance.
- Impact of length and location: Reasoning performance shows a very large decline with longer contexts when relevant facts must be collected from two non-adjacent locations.This setting makes evidence integration across separated locations significantly harder.
- Kind of irrelevant material: Performance degrades with both similar and Books Corpus padding, and Books padding has a greater impact in most models.The different-padding condition was mostly worse than the similar-padding condition.
5 Correlation with Next Word Prediction
The paper compares next-word prediction with reasoning accuracy on extended inputs and finds that the two measures diverge. Next-word accuracy can increase with input length while reasoning accuracy declines, so it cannot substitute for downstream evaluation.
- Next-word accuracy is measured by exact matching the model’s predicted next word to the true next word because closed models lack full vocabulary probabilities.
- Next-word accuracy increases as input length grows, reproducing trends reported in prior work.
- Next-word accuracy correlates negatively with reasoning accuracy on FLenQA.Figure 6 reports this relationship across 300 samples.
- Perplexity and next-word prediction cannot substitute for downstream task evaluation on long inputs.
6 Does Chain of Thought Help?
The paper tests whether Chain-of-Thought prompting mitigates reasoning degradation as inputs lengthen. CoT generally improves short-input accuracy but does not eliminate the performance drop on longer inputs, with GPT4 as an exception.
- CoT prompting improves performance in most tested models but generally does not mitigate the drop caused by longer inputs.The reported models include GPT4, Mixtral 8x7B, Mistral Medium, GPT3.5, and Gemini-Pro.
- GPT4 is the exception, with the gap between CoT and normal prompting increasing as input length grows.
- Gemini-Pro improves with CoT at short lengths but declines as input length increases.
7 Length-induced Failure modes
The paper identifies length-dependent failures in instruction following, label selection, answer ordering, and Chain-of-Thought coverage. These failures become more frequent or severe as inputs grow longer and are associated with incorrect responses.
- Failure to answer: Longer inputs increase failures to answer with the instructed True/False choice, including refusals claiming insufficient information.
- Label bias: Some models increasingly favor the label “false” as input length grows, despite FLenQA’s balanced label distribution.
- The failure patterns include generating answers that contain neither of the instructed labels and producing the answer before reasoning.
- Answer first, reason later: Under CoT prompting, some models increasingly output the final answer before the reasoning steps as inputs grow longer.These incorrect responses are statistically dependent on answer-first outputs; the reported odds-ratio is 3.643 with p < 0.001.
8 Related Work
Prior work evaluates long-input models through downstream benchmarks or next-word prediction, but existing benchmark designs often use fixed lengths and vary tasks alongside inputs. This paper isolates input length to study its impact on performance.
- Long-input evaluation has followed downstream-task benchmarks and next-word prediction as two distinct pathways.
- Existing benchmark datasets use inputs of different but fixed lengths, limiting evaluation across continuously varying input lengths.
- The correlation between next-word prediction and downstream performance has been inconsistent in prior work, including this paper’s extended-length setting.
- This study isolates input length while examining prior interventions involving semantic content, prompting strategies, and QA-task properties.
9 Discussion
The study finds a strong decline in LLM reasoning performance as inputs lengthen, well before models reach their stated maximum capacity. It also identifies failure modes and argues that evaluations should test performance across supported input lengths.
- Input length substantially reduces LLM reasoning performance, even before reaching models’ maximum input-length capacity.
- The effect persists across different adjustments to the dataset samples, indicating that the length-related pattern is robust within the tested variants.
- The analysis identifies difficulties following extended instructions and biases toward less relevant information as specific failure modes.
- A single input length does not fully characterize model performance; long-range capability requires maintaining performance across technically supported lengths.
Limitations
The study’s limitations concern unexplained behavioral effects, restricted task coverage, generated padding text, and an untested aspect of input structure.
- The observed performance drop remains unexplained because the study relies on behavioral testing and lacks access to many models.
- Using tasks suited to the lowest common denominator may overlook nuanced differences in more complex reasoning tasks.
- The study focuses on a subset of reasoning task types, and GPT4-generated extensions may bias surrounding text relative to the original reasoning content.
- The experiments do not test the distance between key paragraphs, leaving that aspect of LLM performance unexplored.
C Full results
The full-results figures organize outcomes across three reasoning datasets and examine how key-paragraph position and answer-generation biases relate to model behavior.
- Figures 13–15 present the full results separately for the Ruletaker, MonoRel, and People In Rooms datasets.
- Figure 16 compares accuracy differences across positions of key paragraphs, averaging over similar and dissimilar irrelevant padding.
- Figure 17 reports each model’s response frequencies for True, False, and neither answers.