PaperlayerView exact revision on arXiv

Source-linked AI summary

You don't need long context

Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov

arXiv:2611.05306v1cs.AI

TL;DR

Long contexts raise generation costs, yet it is unclear whether long-horizon and long-input tasks require them. The paper tests frequent compaction and selective input access, finding that short contexts broadly maintain task accuracy, including with one action per context.

  • Problem

    Long contexts are costly, but whether long-horizon and long-input tasks need them to preserve task accuracy remains an open question.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates frequent context compaction, selective input reading, and single-action compaction across long-horizon and long-input tasks.

    Source: §3 · p. 9, §6 · p. 15

  • Results

    Across tested tasks, compaction usually matches or improves long-horizon accuracy, selective reading matches or improves full-input baselines in almost every case, and single-action compaction nearly preserves or improves baseline accuracy.

    Source: §6 · p. 15

  • Takeaways & Limitations

    The tested long-input and long-horizon workloads do not establish a need for long contexts when state is maintained and inputs are accessed selectively.

    Source: §6 · p. 15

    The evidence does not rule out tasks that benefit from long context or prove that every task can be solved with short context.

    Source: §6 · p. 15

Abstract

from arXiv · show

Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.

1. Introduction

Long contexts increase generation cost, latency, and memory use, yet their necessity for long-horizon and long-input tasks is uncertain. The section poses whether such tasks can be completed with short contexts, including one action per context.

1. Introduction

  • 3.6×, 5.6×, and 7× more per generated token is the cost of serving GLM 5.2, Kimi K2.6, and Qwen 3 Coder Next at 256K rather than 16K context windows.The cost comparison uses measured throughput and an on-demand H100 price; later tokens also take longer and require more memory in several measured models.Source: Figure 1 · p. 1, §1 · p. 1, Figure 2 · p. 2
  • For several locally run coding models, inter-token latency approximately doubles across measured context lengths, increasing response-generation time at longer contexts.The section also notes that longer contexts use more memory and make output generation more expensive.Source: §1 · p. 2
  • Context windows have grown far beyond the lengths of many influential writings, despite the cost advantages of keeping contexts short.The comparison is presented in Table 1; the section suggests model development has prioritized accommodating larger inputs over efficiency.Source: §1 · p. 2, Table 1 · p. 2
  • Long-horizon tasks accumulate interaction histories, while long-input tasks involve large source material; neither necessarily requires the entire history or input in context at every step.The paper notes that long-input tasks can be reframed as long-horizon tasks by storing source material as a resource and selectively reading it.Source: §1 · p. 3
  • The paper asks whether long-horizon and long-input tasks can work without long contexts, and whether long-horizon tasks can use at most one action per context.These questions follow from the cost of long contexts and uncertainty about whether they are necessary for task completion.Source: §1 · pp. 2–3

2. Methodology

The methodology tests context management across long-horizon and long-input benchmarks, using two model groups and agent harnesses. It compares full-history accumulation, compaction, selective input reading, and serving measurements as context length varies.

2.1. Definitions

  • Long-horizon tasks require sequential model actions whose later steps depend on earlier observations or decisions; short contexts are defined relative to contemporary windows, commonly around 16K–50K tokens.Long contexts approach a significant fraction of the roughly 256K-token-or-more windows common among frontier models; long-input refers to available source material size, whether or not it is ingested all at once.Source: §2.1 · pp. 3–4

2. Methodology

  • The study organizes benchmarks around long-horizon workflows with accumulating histories and long-input tasks with large source material, including software engineering, information seeking, and tool use.The benchmark suite includes realistic workloads and controlled long-input diagnostics.Source: §2.2 · p. 4

2.2. Benchmarks

  • The benchmark suite spans retrieval and reasoning over long inputs, including RULER at 256K and 500K tokens, LOONG’s multi-document tasks, and AA-LCR’s cross-document questions.Long-horizon evaluations cover sequential software and terminal work, information retrieval, user interaction, and tool use.Source: §2.2 · pp. 4–5

2.3. Models

  • Claude Opus 4.5 and GPT 5.6 Sol are the principal task-accuracy workers; open-weight models support throughput and complementary local experiments.The paper uses independently developed closed-weight model families and a diverse set of open-weight architectures.Source: §2.3 · pp. 5–6

2.4. Harnesses

  • The harnesses use a ReAct loop in which the model issues one Bash action at a time and receives its result; mini-swe-agent serves software and terminal tasks, while mini-agent handles the remaining benchmarks.Both harnesses provide task-specific environments and submission or evaluation procedures.Source: §2.4 · p. 6

2.4.1. Accommodations for long conversational context

  • The three regimes are full-history accumulation, end-of-conversation compaction, and single-action compaction, which carries compact semantic state between fresh conversations.FHA retains the whole history; EoCC compacts it at a threshold and resumes in a fresh conversation; SAC performs one action per conversation and discards that conversation afterward.Source: Figure 3 · p. 7, §2.4.1 · p. 7

2.4.2. Accommodations for long inputs

  • Long inputs are stored as environment documents and read selectively with Bash tools, turning input access into an explicit action in the agent workflow.This presentation reduces a long-input workload to a long-horizon workload.Source: §2.4.2 · p. 7

2.5. Latency and throughput measurement

  • Cloud throughput is measured under saturated serving load across context-window sizes, while local serving measurements record inter-token latency and device memory after increasing input prefixes.For cloud tests, request lengths are drawn from nested prefixes, each request generates 1,024 tokens, and the highest-throughput setting without KV-cache preemption is reported; local batch-one throughput is the reciprocal of mean inter-token latency.Source: §2.5 · p. 8, Figure 4 · p. 8

3. Long-horizon tasks don’t need long contexts

Frequent end-of-conversation compaction broadly maintains long-horizon task accuracy while reducing context size. Long-horizon tasks may need access to prior history, but not necessarily the entire history in the active context.

3. Long-horizon tasks don’t need long contexts

  • EoCC was evaluated against full-history baselines on six long-horizon benchmarks as context limits and compaction thresholds were progressively lowered.The tested context limits and thresholds varied by benchmark.Source: §3 · p. 9
  • Across long-horizon benchmarks, Opus 4.5 accuracy ranges from -3.7 to +5.6ppts and GPT 5.6 Sol from -3.3 to +4.4 relative to baseline, with few large losses.No more than 5 of 22 threshold configurations lose more than 2ppts; estimated cost savings range from 2.2x to 8.8x for Opus 4.5 and 2.4x to 5.0x for GPT 5.6 Sol.Source: Figure 5 · p. 9, §3 · p. 10

3.1. Long-horizon tasks may sometimes need the history

  • Compaction is lossy, and repeated compactions can discard still more history; tasks requiring exact retrieval of every coin-toss outcome can therefore fail.The paper describes an agent that must later inspect a particular toss outcome as an example of a task guaranteed to eventually lose accuracy under a fixed compaction budget.Source: §3.1 · p. 10
  • A harness can retain every conversation message outside the active context and expose history through selective reading when needed.This turns querying prior conversation history into a long-input task.Source: §3.1 · p. 10
  • The findings support keeping the entire history out of context, not eliminating access to conversational history.The authors distinguish information that must be in context from information that could be retrieved selectively.Source: §3.1 · p. 10

4. Long-input tasks don’t need long contexts

Selective input reading, with compaction as a fallback, keeps long-input task contexts short while usually matching or improving baseline performance. Conversation lengths remain broadly bounded as document lengths grow.

4. Long-input tasks don’t need long contexts

  • The evaluation compares selective-reading harnesses using Bash tools against full-input baselines, with context limits and compaction thresholds lowered for some tasks.For many tasks, especially needle-based tasks, one restrictive context limit suffices to demonstrate on-par or better performance.Source: §4 · p. 11
  • Across long-input benchmarks, Opus 4.5 relative performance ranges from -0.4 to +40.0ppts and GPT 5.6 Sol from +1.0 to +30.9; estimated cost savings reach 54.6x and 233.3x, respectively.Savings range from 6.5x to 54.6x for Opus 4.5 and 4.4x to 233.3x for GPT 5.6 Sol.Source: Figure 6 · p. 11, §4 · p. 12
  • Conversation lengths stay in broadly similar bands across document lengths spanning orders of magnitude, rather than growing proportionally with document size.The figures plot conversation length on a linear axis against document length on a logarithmic axis.Source: §4 · p. 12, Figure 7 · p. 12, Figure 8 · p. 13
  • Selective input reading with compaction as a fallback matches or improves baseline performance on almost all long-input benchmarks and stays close in the remaining case.Peak live context remains far below baseline conversation lengths.Source: §4 · p. 13

5. Compaction can be performed after every action

Single-Action Compaction (SAC) preserves or improves accuracy for Claude Opus 4.5 while GPT 5.6 Sol incurs small reductions, despite each interaction taking only one action. The authors propose Semantic State Space Models to maintain a language-described conversation state between actions.

5. Compaction can be performed after every action

  • SAC preserves or improves baseline accuracy for Claude Opus 4.5, while GPT 5.6 Sol incurs small reductions despite one action per context and much shorter live contexts.The comparison is against full-history accumulation; Table 2 also reports best EoCC and peak-context statistics.Source: §5 · p. 14, Table 2 · p. 14

5.1. Semantic State Space Models (SSSMs).

  • An SSSM is proposed as a post-trained model that interprets a language description of conversation state, acts, registers outcomes, and updates that description for a later inference instance.Unlike a state-space model’s black-box neural state, an SSSM maintains an explainable natural-language state between turns.Source: §5.1 · p. 14
  • The proposed SSSM’s stated utility is increased token efficiency from maintaining short contexts; the authors also surmise training it may be more data-efficient.They state that matching full-history models remains a possibility, not an experimental result established here.Source: §5.1 · pp. 14–15

6. Conclusion

Across the tested long-context workloads, compaction and selective access preserve task performance with short contexts. The results do not establish that short contexts suffice for every possible task.

6. Conclusion

  • Long-horizon compaction usually matches or slightly improves full-history accuracy, while selective reading with fallback compaction matches or improves full-input baselines in almost every tested case.Single-action compaction nearly preserves or improves baseline accuracy while carrying only a compact semantic state between actions.Source: §6 · p. 15
  • The experiments support the conclusion that long contexts are unnecessary for completing the tested “long-context” tasks.The authors connect this conclusion to the bounded-input, finite-state premise underlying computability.Source: §6 · p. 15
  • The evidence does not rule out tasks that benefit from long context or prove that every task can be solved with short context.It shows only that arguments based on long inputs and long conversational horizons do not establish necessity in the tested settings.Source: §6 · p. 15

A. What characterizes a good compaction?

Useful compactions for autonomous software engineering preserve information needed for future work, rather than merely narrating the agent’s history. In human-in-the-loop workflows, they must also retain information relevant to responding to later human feedback.

A. What characterizes a good compaction?

  • Compaction can remove crucial information or user preferences, requiring retrieval again and potentially slowing or confusing human-in-the-loop software engineering.Compactions must account for information that may matter to autonomous work following human steering feedback.Source: A. What characterizes a good compaction? · p. 20
  • Treating compaction as ordinary summarization is suboptimal because summaries can weight facts evenly and overemphasize repeated trace segments irrelevant to the remaining work.The study compares summarization and message pruning, and different choices of which trace material to prune.Source: A. What characterizes a good compaction? · p. 20
  • The most accuracy-preserving compactions retain the task, the agent’s current occupation and next steps, and artifacts relevant to future execution.They also include a decomposed plan, completion requirements, and relevant information or pointers; the structure was used across the workflows considered, including non-SWE workflows.Source: A. What characterizes a good compaction? · p. 20

B. Estimated optimized retail cost

The appendix estimates optimized retail cost using model-specific token costs and measured decode throughput. It combines estimated prices with aggregate uncached-input, cached-input, and output token totals.

B. Estimated optimized retail cost

  • Fixed GPU allocation and hourly rate make the 16K-to-256K output-cost ratio depend on measured decode-throughput scaling.The throughput measure is peak aggregate output tokens per second after input-context processing, assuming KV-cached input.Source: B. Estimated optimized retail cost · p. 20
  • The rounded mean of the reported measurements is r = 0.20.The listed measurements are 0.2772 for GLM 5.2, 0.1777 for Kimi K2.6, and 0.1439 for Qwen 3 Coder Next.Source: B. Estimated optimized retail cost · p. 20
  • The assumed reference context windows are 200,000 tokens for Opus 4.5 and 272K for GPT 5.6 Sol.The GPT reference is its standard-price ceiling, although the model supports 1,050,000 tokens; experimental compaction does not exceed that ceiling.Source: B. Estimated optimized retail cost · p. 20
  • The cost estimate combines uncached-input, cached-input, and generated-output token totals using their estimated per-token costs.The estimates preserve the frontier models’ original 5:1 output-to-uncached-input and 10:1 uncached-to-cached-input price ratios.Source: B. Estimated optimized retail cost · pp. 20–21

C. Detailed long-horizon results

Across the detailed long-horizon results, compaction generally performs close to or better than corresponding baselines, with some task- and setting-specific accuracy losses. The reported comparisons span several benchmarks and both Claude Opus 4.5 and GPT 5.6 Sol.

C. Detailed long-horizon results

  • Compaction accuracy is generally close to or slightly better than the corresponding baseline, while new input tokens tend to rise and cached tokens fall as context limits shrink.Output-token counts are generally elevated when compacting.Source: C. Detailed long-horizon results · p. 22
  • GPT 5.6 Sol matches or improves on its SWE-bench Verified baseline at every evaluated setting; Claude Opus 4.5 improves except at 16K / 12K, where accuracy falls by 1.2 percentage points.Best accuracy gains are 4.8 percentage points for Claude Opus 4.5 and 4.4 points for GPT 5.6 Sol.Source: C. Detailed long-horizon results · p. 22, Table 3 · p. 22
  • On SWE-bench Multilingual, gains peak at 48K / 36K—1.7 percentage points for Claude Opus 4.5 and 0.7 points for GPT 5.6 Sol—while both fall below baseline at 16K / 12K.At the tightest setting, the decreases are 2.0 and 1.6 percentage points, respectively.Source: C. Detailed long-horizon results · p. 22, Table 4 · p. 22
  • On Terminal-Bench, Claude Opus 4.5 improves at every evaluated setting; GPT 5.6 Sol gains 1.2 points at 48K / 36K but loses 1.1–2.2 points at the other settings.Claude’s largest gain is 5.6 percentage points at 32K / 24K.Source: C. Detailed long-horizon results · p. 22, Table 5 · p. 23
  • On FRAMES, accuracy falls by 1.3–3.7 points for Claude Opus 4.5 and 1.5–3.3 points for GPT 5.6 Sol; at 16K / 12K, estimated optimized costs fall 7.7× and 3.8×, respectively.The corresponding accuracy reductions at that setting are 1.4 and 3.1 points.Source: C. Detailed long-horizon results · p. 23, Table 6 · p. 23
  • The BFCLv3 evaluation covers 200 long-context multi-turn episodes, while on τ2-bench Claude Opus 4.5 stays within 1.1 points of its 79.3% baseline across the reported settings.Its accuracy is 78.2% at 12K / 8K, matches baseline at 16K / 12K, and reaches 78.8% at 8K / 6K; GPT 5.6 Sol improves at 12K / 8K and remains near baseline elsewhere.Source: C. Detailed long-horizon results · p. 23, Table 8 · p. 23, Table 7 · p. 23

D. Open-weight model results

Compaction robustness varies substantially across the five open-weight models: all lose SWE-bench accuracy at the tightest setting, but all retain some workflow utility. Qwen 3 Coder Next suffers a particularly large SWE-bench collapse, while Qwen 3.6 27B and MiniMax M2.7 retain Terminal-Bench baseline scores.

D. Open-weight model results

  • 53.0% to 3.6%: Qwen 3 Coder Next’s SWE-bench Verified accuracy collapses at 16K / 12K, mostly because verbose trajectories leave too little room for progress between compactions.At the tightest setting, every model loses SWE-bench accuracy relative to baseline, although Qwen 3.6 27B and MiniMax M2.7 retain their Terminal-Bench baseline scores.Source: D. Open-weight model results · p. 24, Table 9 · p. 24, Table 10 · p. 24
  • Despite tighter compaction thresholds, all five models still complete a significant fraction of benchmark workflows.The evaluated settings are 48K / 40K, 32K / 24K, and 16K / 12K, each expressed as total context limit / compaction threshold.Source: D. Open-weight model results · p. 24

E. Selective reading of long inputs

The mini-agent exposes long inputs as files and retrieves only task-relevant excerpts into the model’s context. Its instructions favor targeted searches and bounded reads over dumping whole documents.

E. Selective reading of long inputs

  • Mini-agent exposes long inputs as text files and lets the model retrieve selected excerpts through Bash; only returned matches enter context.LongBench v2 uses one document file, while LOONG and AA-LCR expose multiple source files.Source: E. Selective reading of long inputs · p. 25
  • A search identifies relevant entities, quantities, or claims, then reads a small surrounding window; follow-up searches can broaden the evidence.The process uses commands such as grep and sed, with convenience tools for bounded reads and search-term variants.Source: E. Selective reading of long inputs · p. 25

E.1. Instructions supplied to the agent.

  • The instructions favor searching for relevant entities or claims, reading nearby passages, and keeping each read small rather than dumping the document.The bounded-read tool defaults to 6000 characters and is hard-capped at 8000; targeted searches are described as faster than sequential reading.Source: §E.1 · p. 25

E.2. Examples from recorded runs.

Recorded runs illustrate selective retrieval across single-document grammar lookup, multi-paper search, and multi-document reasoning with an exclusion. They show how searches and bounded follow-up reads locate and verify evidence without reading entire inputs.

E.2. Examples from recorded runs.

  • The traces are illustrative examples, not estimates of typical reading cost.The examples retain only commands and selected tool output.Source: §E.2 · p. 25

1. LongBench v2: locate a word, then read its grammatical context.

  • For a roughly 1M-character Kalamang grammar, the agent searches a sentence word, reads a 36-line passage, then retrieves surrounding grammar and examples.The follow-up read supplies context beyond the initial lexical match.Source: §1 · pp. 25–26

2. LOONG: search across papers, then verify a paper’s identity.

  • For a citation-relationship question across roughly 138K characters of papers, the agent searches for a paper’s name and reads the matching paper’s beginning to verify its identity.The example combines a cross-file grep with a surgical 1K-character follow-up read.Source: §2 · p. 26

3. AA-LCR: read the context needed to apply an exclusion.

  • For an exclusion question across four documents totaling roughly 404K characters, the agent narrows the search to one report and retrieves a short passage with industry names, counts, and time span.The question excludes broadcasting; the example shows repeated searches and reads until relevant evidence is identified.Source: §3 · p. 26

F. Detailed long-input results

Selective reading with compaction matches or improves on full-input baselines across several long-input benchmarks, while keeping live context bounded. The section also outlines the two agent harnesses and their compaction handoffs.

F. Detailed long-input results

  • For long inputs, agents read task-relevant document portions and compact when working context reaches the threshold; settings are reported as total context limit / compaction threshold.The section compares these agents with baselines that paste long inputs into work conversations.Source: F. Detailed long-input results · p. 27
  • On LongBench v2, GPT 5.6 Sol improves by 1.8 percentage points and Claude Opus 4.5 stays within 0.4 points of baseline; every LOONG setting improves both models’ average score and perfect-answer rate.LongBench v2 uses a 16K total context limit; LOONG settings bound peak live context at 16–48K tokens.Source: F. Detailed long-input results · p. 27, Table 11 · p. 27, Table 12 · p. 27
  • At 16K / 12K on AA-LCR, Claude Opus 4.5 reaches 84.0% accuracy versus 65.0% for baseline, while GPT 5.6 Sol reaches 83.0% versus 82.0%.The comparison is against the full-input baseline.Source: F. Detailed long-input results · p. 28, Table 13 · p. 28
  • Selective reading reaches 100% accuracy on NIAH and raises RULER mean recall from 62.5% to 95.7% for Claude Opus 4.5 and from 65.5% to 96.4% for GPT 5.6 Sol.NIAH uses no compactions; RULER has a 16K total context limit and median zero compactions for both models.Source: F. Detailed long-input results · p. 28, Table 14 · p. 28, Table 15 · p. 28
  • On Counting-Stars, both models reach a 99.7% mean score and 81.2% exact match, improving exact match from 65.1% for Claude Opus 4.5 and 76.6% for GPT 5.6 Sol.Neither model performs any compaction on this benchmark.Source: F. Detailed long-input results · p. 29, Table 16 · p. 29
  • On BABILong, accuracy improves by 24.9 percentage points for Claude Opus 4.5 and 23.3 points for GPT 5.6 Sol.Peak live context is bounded at 16K tokens; GPT 5.6 Sol never compacts and Claude Opus 4.5 performs at most one compaction per task.Source: F. Detailed long-input results · p. 29, Table 17 · p. 29

G. Agent outlines

  • The outlines retain mini-swe-agent’s task-solving loop and specify handoff recaps for both coding tasks and the generalized mini-agent workflow.Mini-agent removes SWE-specific instructions and adds instructions for additional agentic workflows.Source: §G.1 · p. 30, §G.2 · pp. 30–31

Related papers

Loading 2611.05306v1…