PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning

Shiwen Qin, Cameron Barker, Henry Gouk, Elliot J. Crowley, Shay B. Cohen, Mirella Lapata, Miao Li

arXiv:2610.22067v1cs.CL

TL;DR

Context length alone does not explain why long-context reasoning tasks of similar length can differ in difficulty. The paper defines evidence scope, dispersion, and reasoning depth, and builds LURCH to measure them. Across evaluations, each property relates to lower performance at fixed length, dispersed evidence becomes especially challenging as contexts grow, and demands across properties compound.

  • Problem

    Context length is commonly used as the main indicator of long-context difficulty, but it leaves open which properties beyond token count make reasoning difficult.

    Source: §1 · p. 2

  • Method

    The authors define evidence scope, evidence dispersion, and reasoning depth, then build LURCH, a 6,550-instance benchmark with supporting evidence and answer-conditioned reasoning graphs.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    At fixed length, broader, more dispersed, and deeper questions score lower; added length is harmful mainly when evidence is broad or dispersed, and demands on several properties compound even without an extreme value.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Evidence scope, dispersion, and reasoning depth complement context length as measures of long-context reasoning difficulty.

    Source: §1 · p. 3

Abstract

from arXiv · show

Long-context language models support ever longer context windows, and evaluations typically treat context length as the main indicator of difficulty. However, context length alone cannot explain why models fail on long-context tasks that require reasoning over complex information. Contexts of equal length can pose very different challenges: a task may be solvable from a single local passage, or it may require integrating evidence distributed across the full context. We quantitatively characterise this difficulty through three properties: evidence scope, how much necessary information must be found; evidence dispersion, how widely that information is scattered within the context; and reasoning depth, how many dependent steps are needed to derive an answer from it. We construct LURCH, a multi-domain question-answering benchmark of 6,550 instances with supporting evidence and reasoning graphs, enabling all three properties to be quantified. Evaluations of proprietary and open-weight models show that: (1) at fixed context length, all our three properties are strongly negatively correlated with model performance; (2) as context length increases, performance degrades sharply on samples with dispersed evidence, whereas it remains relatively stable on samples with localised evidence; and (3) difficulty compounds across properties, as questions with moderately high values on all three are harder than those with an extreme value on just one. These findings establish evidence scope, dispersion and reasoning depth as measures of long-context reasoning difficulty that complement context length.

1 INTRODUCTION

Context length alone does not explain variation in long-context reasoning difficulty. The paper quantifies evidence scope, dispersion, and reasoning depth, then finds that each relates to performance and that their demands can compound.

1 INTRODUCTION

  • Prior evaluations vary context length but leave open which properties beyond token count make long-context reasoning difficult.Earlier proposals for scope and dispersion were qualitative, while ETHIC quantified information coverage in a limited range of settings.Source: §1 · p. 2
  • LURCH uses quantitative evidence-scope and dispersion measures and reasoning depth from answer-conditioned evidence-to-answer graphs to characterise 6,550 multi-domain instances.Each instance includes an answer, supporting evidence, and a reasoning graph.Source: §1 · p. 2
  • At fixed context length, greater evidence scope, dispersion, and reasoning depth are each associated with lower model scores.Doubling reasoning depth is associated with a score drop comparable to doubling context length.Source: §1 · p. 2
  • As context length increases, performance degrades sharply with widely dispersed evidence but remains relatively stable with localised evidence.This pattern holds across all evaluated models.Source: §1 · p. 2
  • Questions moderately high on all three properties are harder than questions with an extreme value on only one.The pattern is broadly consistent with an additive combination of the properties.Source: §1 · p. 2
  • The three properties complement context length and provide a basis for diagnosing model failures and designing more informative evaluations.Source: §1 · p. 3

2 RELATED WORK

Long-context benchmarks increasingly test reasoning, but generally show where models fail rather than what makes individual instances difficult. Existing difficulty measures remain limited in quantification, task coverage, or conceptual scope.

2 RELATED WORK

  • The literature describes long-context failures and reasoning tasks, while factors determining those failures remain under-explored.Prior work reports middle-position performance degradation and effective context lengths that often fall short of claimed windows.Source: §2 · p. 3
  • Existing benchmarks span multi-document, domain-specific, and aggregation-heavy reasoning, but generally do not identify what makes individual instances difficult.This includes broad benchmarks such as LongBench v2 and Loong, alongside domain-specific DocFinQA and SciTrek.Source: §2 · p. 3
  • Goldman et al. (2024) conceptualise scope and dispersion but do not explicitly quantify them.Scope concerns the amount of information required; dispersion concerns how difficult it is to locate that information.Source: §2 · p. 3
  • ETHIC (Lee et al., 2025) quantifies information coverage using an evaluator model’s judgments, but covers a narrower range of reasoning settings and difficulty factors.Source: §2 · p. 3

3 REASONING CHARACTERISATION

The paper characterises long-context reasoning with evidence scope, evidence dispersion, and reasoning depth. These measure how much evidence is involved, how it is distributed, and how sequentially it must be combined.

3 REASONING CHARACTERISATION

  • Examples A and B require relatively little evidence and have depth two, but differ substantially in dispersion; example C combines broader coverage, high dispersion, and greater depth.Source: §3 · p. 3
  • Evidence scope measures the proportion of the context covered by the evidence set.It is based on the amount of supporting evidence relative to document length.Source: §3 · pp. 3–4
  • Evidence dispersion measures how much non-evidence text within the span between the first and last evidence intervals contributes to the context’s non-evidence text.Dispersion ranges from zero for a single interval to one when the evidence span covers the full context.Source: §3 · p. 4
  • Reasoning depth is the longest dependency path in an evidence-to-answer graph, counting sequential dependencies rather than total operations.Parallel evidence lookups do not increase depth unless later operations depend on them; directly reading an answer from evidence has depth one.Source: §3 · p. 4

4 BENCHMARK CONSTRUCTION

LURCH standardises long-context question-answering instances and adds supporting evidence and validated reasoning graphs so its difficulty properties can be measured. Its 6,550 instances combine long inputs with sparse evidence, often dispersed, and multi-step reasoning.

4 BENCHMARK CONSTRUCTION

  • The benchmark standardises instances from existing long-context benchmarks, using reference or automatically annotated evidence and validated reasoning graphs.Source: §4 · p. 4

4.1 DATA COLLECTION

  • LURCH retains 6,550 instances from 17 sources after filtering, validation, and subsampling 323,040 instances from 26 candidate sources.Source: §4.1 · p. 4

4.2 EVIDENCE ANNOTATION

  • Evidence annotation uses benchmark-provided evidence where available; otherwise, retrieved passages are expanded and verified, so scope and dispersion describe expanded supporting context rather than necessarily minimal evidence.Questions are retained only when three verifier answers using the expanded evidence each meet the task-specific correctness criterion.Source: §4.2 · p. 5

4.3 REASONING GRAPH GENERATION

  • Reasoning graphs are independently generated by Gemini 3.7 Flash and GPT-5.6 Terra, but graph-derived properties use Gemini generations after validation and comparison.The annotators assign identical depths in 65.6% of cases, with Spearman rank correlation of 0.48.Source: §4.3 · p. 5

4.4 DATASET PROPERTIES

  • Across LURCH, median context length is 103k tokens, median evidence coverage is 0.66%, 11% have evidence in one block, and 65% require at least two dependent steps.Among instances with evidence spanning multiple blocks, median dispersion is 0.39; similarly long tasks can have different evidence profiles, such as localised HotpotQA versus broader, highly dispersed SciTrek and CorpusQA.Source: §4.4 · p. 6

5 EXPERIMENTS

Across four LLMs and 6,550 questions, the experiments test how evidence scope, dispersion, and reasoning depth relate to scores beyond context length, and whether they modify length-associated score changes. All three properties are associated with lower scores, while length-related declines are sharper for broad or dispersed evidence and multiple moderate demands can coincide with low scores.

5.1 EXPERIMENTAL SETTINGS

  • The evaluation uses four LLMs on all 6,550 questions from 17 retained LURCH tasks, averaging three sampled responses per question–LLM pair.Each model is evaluated using the full context for each question.Source: §5.1 · p. 6

5.2 GENERALISED LINEAR MODEL

  • A fractional logistic GLM estimates score associations for four standardized log predictors with task- and LLM-specific intercepts; RQ2 adds length-by-attribute interactions.The analyses estimate conditional associations within LURCH rather than causal effects.Source: §5.1 · p. 6, §5.2 · p. 7

5.3 RQ1: DIFFICULTY AT THE SAME LENGTH

  • Evidence scope, dispersion, and depth each predict lower scores; interquartile contrasts are −2.14 pp, −1.10 pp, and −1.60 pp, versus −1.66 pp for context length.Scope has the largest coefficient magnitude (−0.21); scope is negative for three LLMs but positive for Gemini 3.8 Flash (+1.55 pp).Source: Table 1 · p. 7, §5.3 · p. 8

5.4 RQ2: LENGTH-DEPENDENT DIFFICULTY

  • Length-associated score drops are greater at high scope (−0.90 pp) and dispersion (−0.88 pp); the length-by-depth interaction is inconclusive.The scope and dispersion differences and corresponding logit interactions are significant after Holm adjustment; the direction persists across all four LLMs, although some per-LLM intervals include zero.Source: §5.4 · pp. 8–9, Table 2 · p. 9

5.5 RQ3: COMPOUNDING DIFFICULTY

  • Questions high on both scope and dispersion score 2.9 pp below their task–LLM mean, versus 3.6 pp above when low on both; the additive GLM tracks eight of nine cells within ±1.0 pp.The 164 questions above their task medians on all three properties, with none above its within-task 90th percentile, score 5.4 pp below average; the high-dispersion, high-depth pattern is sensitive to SciTrek.Source: §5.5 · p. 9

6 CONCLUSION

The paper argues that context length alone incompletely measures long-context difficulty and proposes evidence scope, dispersion, and reasoning depth as complementary properties. LURCH results show these properties relate to performance, with difficulty increasing when several are elevated together.

6 CONCLUSION

  • At fixed context length, broader, more dispersed, and deeper questions score lower; doubling reasoning depth is associated with a drop comparable to doubling context length.Added length is harmful mainly when evidence is broad or dispersed, and questions moderately high across several properties can be harder than those extreme on just one.Source: §6 · p. 10
  • LURCH contains 6,550 naturally occurring instances from 17 sources, with verified evidence and reasoning graphs produced by the annotation pipeline.Source: §6 · p. 10
  • The analyses are observational, and some annotations are automatic; future work could vary surrounding context while holding evidence and reasoning fixed to test effects causally.Source: §6 · p. 10

A EXISTING LONG-CONTEXT DATASETS

The listed long-context datasets cover broad real-world benchmarks as well as specialized tasks involving distributed evidence, reasoning, retrieval, memory, citations, and domain-specific analysis.

A EXISTING LONG-CONTEXT DATASETS

  • The candidate-source description excludes BABILong, ChatQA2, ETHIC, L-CiteEval, LOFT, LongCite, LongMIT, and LongReason.Source: A EXISTING LONG-CONTEXT DATASETS · p. 14
  • BABILong tests reasoning across facts in extremely long documents, while ∞Bench targets long dependencies and LongReason expands short-context reasoning questions into long-context ones.Source: A EXISTING LONG-CONTEXT DATASETS · pp. 14–15
  • Other benchmarks target retrieval and reasoning across modalities, long-term conversational memory, citation-supported QA, and evaluating use of the entire context.Examples include LOFT, LongMemEval, LongCite, L-CiteEval, and ETHIC.Source: A EXISTING LONG-CONTEXT DATASETS · pp. 14–15
  • LongBench v2 and LongBench-Pro cover real-world long-context reasoning, while Loong focuses on multi-document financial, legal, and academic scenarios.Source: A EXISTING LONG-CONTEXT DATASETS · pp. 14–15
  • Specialized datasets cover scientific aggregation and comparison, financial-document QA, computation-intensive corpus reasoning, and mathematical, logical, or multi-hop reasoning.These include SciTrek, DocFinQA, CorpusQA, and DocQA-RL, respectively.Source: A EXISTING LONG-CONTEXT DATASETS · pp. 14–15

B DATA FILTERING

LURCH selection filters target English, naturally occurring long contexts, and tasks suited to studying reasoning, while screening for context-free answerability and adequate task size.

B DATA FILTERING

  • Inputs are restricted to English-language instances containing 32k–256k tokens, measured with the Qwen3-0.6B tokenizer, to balance long-context coverage and computational feasibility.Source: B DATA FILTERING · p. 15
  • The collection retains naturally occurring long-form contexts, allows appended real documents, and excludes synthetic or synthetically expanded contexts.This restriction is intended to focus analysis on realistic long-context inputs.Source: B DATA FILTERING · p. 15
  • Source or subset definitions are audited to exclude tasks primarily targeting single-fact retrieval or span extraction and prioritize multi-hop reasoning, comparison, aggregation, or evidence integration.Retained instances are not required individually to involve multiple reasoning steps.Source: B DATA FILTERING · p. 15
  • Gemini 2.5 Pro answers each question without its context, and instances answered correctly under the task criterion are discarded; this does not guarantee context is required for every retained question or model.Source: B DATA FILTERING · pp. 15–16
  • Tasks with fewer than five annotated instances are dropped, including two ETHIC subsets with five instances total; SciTrek is subsampled to 1,000 instances to balance sources.Table 3 reports average token lengths for context, question, and answer, and reserves its threshold for screening and evidence validation rather than Section 5 analyses.Source: B DATA FILTERING · p. 16, Table 3 · p. 16

C LURCH STATISTICS AND METRICS

Table 3 reports LURCH statistics and metrics, while the evidence annotation procedure retrieves, expands, and validates supporting evidence before using it for measurements and reasoning graphs.

C LURCH STATISTICS AND METRICS

D EVIDENCE ANNOTATION PROCEDURE

  • Sources with reference evidence use those annotations directly; for other sources, evidence is identified through retrieval, expansion, and sufficiency validation.Source: D EVIDENCE ANNOTATION PROCEDURE · p. 16
  • Gemini 2.5 Pro retrieves verbatim passages as candidate evidence, then the procedure expands them to containing paragraphs or full tables and merges overlapping or adjacent intervals.Source: D EVIDENCE ANNOTATION PROCEDURE · pp. 16–17
  • Figure 7 illustrates depth 2: separate level-1 statements provide release years, and comparing them adds one dependent step.Source: Figure 7 · p. 17
  • The expanded evidence set is used for sufficiency validation, reasoning-graph annotation, and computing evidence scope and dispersion.These measures characterize expanded supporting context rather than only initially retrieved passages.Source: D EVIDENCE ANNOTATION PROCEDURE · p. 17
  • Gemini 3.7 Flash answers from the expanded evidence alone in three sampled responses; an instance is retained only if all three meet its task-specific correctness criterion.This repeated check does not establish that every retained passage is necessary or that the evidence set is minimal.Source: D EVIDENCE ANNOTATION PROCEDURE · p. 17

E REASONING GRAPH VALIDATION

The validation checks graph annotation agreement and uses a human audit of depth disagreements to select the Gemini graphs for the final measures. The annotation prompt requires minimal, atomic, evidence-grounded dependency graphs.

E REASONING GRAPH VALIDATION

  • The annotation prompt builds a minimal directed acyclic derivation from evidence to answer, with atomic lookup and inference steps linked by direct dependencies.It excludes unnecessary steps and requires every used fact to enter through cited evidence.Source: E REASONING GRAPH VALIDATION · p. 18
  • Depth matched exactly on about two-thirds of instances, and gaps of two or more were uncommon (2.8%); step count was least stable, with a 0.93-step mean absolute error.Rank correlations ranged from 0.42 for width to 0.56 for step count.Source: E REASONING GRAPH VALIDATION · p. 18
  • After auditing 40 instances, the authors dropped GPT graphs and used Gemini’s depth, width and step count for every instance.On matched-depth instances, both auditors accepted both graphs; side choices arose among the 20 disagreements.Source: E REASONING GRAPH VALIDATION · p. 19, Table 5 · p. 19

F EXTENDED DATA ANALYSIS

Across the 17 retained tasks, the appendix describes task-level distributions of context length and the three difficulty properties, and reports how those properties correlate with one another.

F EXTENDED DATA ANALYSIS

  • Scope and dispersion were the most strongly associated properties (r = 0.55 overall; r = 0.45 after task-mean centring), while logged context length was not positively correlated with any property.Length correlations were −0.24 with scope, −0.18 with dispersion and −0.01 with depth overall; task-centred values remained non-positive.Source: F EXTENDED DATA ANALYSIS · p. 19, Figure 9 · p. 20
  • Figure 8 displays context length and property distributions by task, using medians, percentile ranges and stacked proportions for reasoning depth.Numbers beside task names give question counts; scope is shown on a logarithmic axis.Source: F EXTENDED DATA ANALYSIS · p. 19, Figure 8 · p. 20

G.1 GLM FIT AND PREDICTIVE PERFORMANCE

The appendix tests the predictive fit of the additive GLM and supplements the primary analyses with attribute contrasts, interaction checks and exploratory configurations. The full model improves predictive loss modestly beyond task, LLM and length controls.

G.1 GLM FIT AND PREDICTIVE PERFORMANCE

  • The full GLM reduced cross-entropy by 6.38% versus the constant predictor and by 0.58% versus task, LLM and length alone.These are reductions in predictive loss; task identity gives the largest improvement among standalone predictor sets.Source: Table 6 · p. 21, §G.1 · p. 21
  • The full GLM broadly followed the calibration diagonal in well-populated bins, though predictions averaging 0.68 corresponded to an observed mean of 0.73.The constant predictor was approximately calibrated overall but gave nearly identical predictions across question–LLM pairs.Source: §G.1 · p. 21, Figure 10 · p. 22

G.2 RQ1: ATTRIBUTE CONTRASTS

  • Supplementary RQ1 checks report interquartile and doubling score contrasts, with doubling estimates covering 34% of questions for length, 91% for scope, 49% for dispersion and 60% for depth.The interquartile comparisons use the 3,471 questions from nine tasks with distinct depth quartiles.Source: Table 7 · p. 21

G.3 RQ2: INTERACTION AND SENSITIVITY CHECKS

  • For Gemini 3.8 Flash, length contrasts were positive at low scope (+1.21 pp) and low dispersion (+0.64 pp), versus −0.04 and −0.85 pp at their high values.Restricting the depth comparison to nine tasks scaled the estimate to −1.25 pp without changing the test.Source: §G.3 · pp. 22–23

G.4 RQ3: EXPLORATORY CONFIGURATIONS

  • The exploratory configuration analysis groups questions by how many properties exceed task medians and whether any exceeds the 90th percentile, then contrasts predictions for 357 all-three-high cases.Sensitivity checks omit one task at a time and also examine SciTrek alone.Source: §G.4 · p. 24, Table 10 · p. 24, Table 11 · p. 24

Related papers

Loading 2610.22067v1…