PaperlayerView exact revision on arXiv

Source-linked AI summary

Agents Don't Paginate: First-Chunk Selection for LLM Tool Responses

Tatiana Petrova, Andrei Mazniak, Radu State

arXiv:2608.26130v1cs.CLcs.IR

TL;DR

Coding-agent tool responses often exceed per-turn budgets, yet agents do not request pagination, motivating first-chunk selection. The paper evaluates six budgeted selection functions and tests whether higher p1 improves downstream file-localisation accuracy. It finds that rank-1 gains do not systematically improve accuracy, while metadata additions can reduce p1.

  • Problem

    Tool responses exceed agents’ per-turn token budgets, but evidence is limited on how first-chunk selection and gold-item rank affect downstream accuracy.

  • Method

    The paper formulates first-chunk selection as a 0/1 knapsack, compares six value functions on 500 SWE-bench Verified tasks, and probes downstream localisation across five language models.

  • Results

    Raising p1 does not systematically raise downstream accuracy; the paper reports keyword-scorer and metadata findings as its main results.

  • Takeaways & Limitations

    Downstream outcomes depend on whether the gold reaches the first chunk at all, not on whether it ranks first within that chunk.

  • Takeaways & Limitations

    The grep-based candidate generator leaves 44.2% of tasks without a matching gold file and caps the absolute p1 ceiling at 55.8%.

Abstract

from arXiv · show

Coding agents built on large language models (LLMs), such as Claude Code, Cursor, OpenAI Codex, GitHub Copilot, and Aider, receive tool responses that routinely exceed the agent's per-turn token budget. The standard remedy, pagination, is available in every protocol that produced these responses; yet across the corpus of session logs from a public Model Context Protocol middleware we observed no agent-initiated requests for a second chunk. The first chunk is what the agent reads, so we ask how often the gold item (the one the agent needs) is placed first in it: the precision-at-1 rate $p_1$. In a controlled offline benchmark we treat first-chunk selection as a 0/1 knapsack and compare six value functions on 500 SWE-bench Verified tasks, then test whether $p_1$ matters with a single-turn file-localisation probe on five language models (4,800 LLM calls; not an end-to-end resolve-rate test). Two pre-registered hypotheses did not hold and are our main findings. The central one is negative: raising $p_1$ does not systematically raise downstream accuracy. Per-model deltas stay under three percentage points (p.p.), are not consistently signed, and no model is significant; the agent recovers the gold from anywhere in the chunk, so what reaches its answer is first-chunk inclusion, not the gold's rank within it. The second: adding four file-metadata signals to a keyword scorer hurts $p_1$ by 4.8 p.p. (paired significance test, $p = 0.001$). A parameter-free keyword scorer does raise $p_1$, from a 24.2% baseline to 35.0% (+10.8 p.p., far beyond chance; $p = 3.9 \times 10^{-8}$), and to 35.8% with a fallback to the tool's native ordering when no keyword matches. But by our central finding this is a rank-1 gain, and rank-1 is the part that does not reach the agent's answer: downstream accuracy does not move.

1 Introduction

Tool responses often exceed per-turn budgets, but production agents do not request later pages, making first-chunk selection central. The paper measures whether the needed item is ranked first and tests selection strategies against downstream accuracy.

  • Motivation: 37% of get_epics and 28% of get_merge_request_diffs calls exceed an 8 K-token budget.These responses can contain dozens or hundreds of concrete items, while pagination is available.
  • Motivation: Zero agent-initiated second-chunk requests were observed across the full session-log corpus.After truncation, agents moved directly to their next reasoning or editing step.
  • Research question: The paper reframes compression as selecting a first chunk that contains the item the agent will use next.It distinguishes first-chunk rank from merely including the gold item anywhere in the chunk.
  • Approach: The study casts selection as a 0/1 knapsack and evaluates six value functions on 500 SWE-bench Verified tasks before a downstream probe across five language models.The two-stage design separates algorithmic p1 measurement from downstream testing.
  • Findings: Raising p1 does not propagate systematically to downstream accuracy because gold inclusion, rather than rank within the chunk, binds.A keyword ranker improves p1, while adding file metadata reverses that gain.

2 Setting and Baselines

The deployed middleware sits between tool results and coding-agent clients, rewriting oversized responses to fit token budgets. Production telemetry shows pagination is unused, while FIFO remains the file-search baseline with p1 = 24.2%.

  • Deployment setting: The middleware rewrites tool-result bodies to fit agents’ per-turn token budgets.It is deployed as an MCP server, npm CLI, and agent-skill bundle across seven SaaS providers.
  • Pagination behavior: Zero pagination requests occurred across all over-budget production responses.This held across coding-agent frontends and the seven federated SaaS providers.
  • Benchmark setting: 57.6% of SWE-bench Verified tasks have ≤5 candidates, 31.4% sit at or near the 50-candidate cap, and 11.0% fall between.The candidate distribution is bimodal.
  • Baseline: 24.2% is the FIFO baseline p1 for file-search responses.FIFO preserves filesystem-traversal order from grep -rln or rg -l without relevance ranking.

3 Method

The method assigns values to candidate items and greedily solves a budgeted 0/1 knapsack, using p1 to measure whether the gold item is surfaced first. Six value functions span native ordering, controls, keyword overlap, metadata composites, and a safety fallback.

  • Knapsack formulation: Each candidate item has token cost c_i and nonnegative value v_i, and selection is constrained by response budget B.For file search, an item is a file path and cost is tokenizer-dependent.
  • Knapsack formulation: Greedy density sorting by v_i/c_i solves the 0/1 knapsack in O(N log N) and is 1/2-optimal.Exact dynamic programming yields no measurable improvement at the observed candidate-list sizes.
  • Metric: p1 is the probability that the gold item ultimately cited by the agent is the top-ranked item surfaced by the value function.The deeper cutoffs p3, p5, and p10 are budget-invariant in this regime.
  • Value functions: The six value functions include FIFO, Random, Reversed, Priority-KW, Priority-ALL, and Priority-KW+.They respectively provide baseline, lower-bound, adversarial, keyword, composite-metadata, and fallback variants.
  • Value functions: Priority-KW+ falls back to FIFO when every candidate keyword score is zero, preventing an empty candidate set.This corner case occurs in approximately 14% of SWE-bench Verified tasks.
  • Value functions: The design space is intentionally broad rather than exhaustive, so changes in p1 can be traced to individual choices.FIFO anchors production behavior, while the other functions isolate signal, adversarial ordering, composite metadata, and safety fallback effects.

4 Experimental Setup

The study evaluates candidate selection for oversized coding-agent tool responses using deterministic file-search candidates, multiple budgets, and algorithmic and downstream experiments.

  • Benchmark: 500 SWE-bench Verified tasks provide gold files and noisy issue-derived queries for evaluating candidate selection.The benchmark covers 12 popular Python repositories.
  • Candidate generation: A deterministic grep-based generator tokenizes issue text and emits matching file paths, truncating each candidate list at 50 files.Candidates are cached per task and reused across strategies.
  • Budgets: Four budgets—1,000, 2,000, 4,000, and 8,000 tokens—bracket practical per-tool-response operating points.
  • Experiments: E1 computes p1 across 500 tasks, six strategies, and four budgets without an LLM, totaling 12,000 cells.The CPU evaluation takes approximately two minutes.
  • Experiments: E2 probes downstream file localisation on 100 winnable tasks across five language models using 4,800 deterministic calls.The subset contains the gold file in every candidate set.
  • Metrics: Table 1 reports p1 with task-level bootstrap 95% confidence intervals, and p1 is constant across budgets within each strategy.

5 Results

Keyword-based ordering substantially improves first-position placement, especially where FIFO is weak, while downstream accuracy changes little despite the p1 increase.

  • Algorithmic first-position rate: The keyword scorer’s gain concentrates at rank 1: top-1 improves by 11.6 p.p., while top-10 improves by 4.0 p.p.
  • Algorithmic first-position rate: Figure 2 compares FIFO with Priority-KW+ across small, medium, and large candidate-list buckets containing 288, 55, and 157 tasks.The lift is largest in medium and large buckets, where FIFO rarely puts the gold first.
  • Downstream LLM accuracy: A roughly 20 p.p. p1 increase produces accuracy shifts between −2.8 and +2.2 p.p., with no model showing a significant change.Table 2 uses paired task-level McNemar tests on budget-averaged accuracy.
  • Downstream LLM accuracy: When the gold is ranked first, the gemma4–Opus accuracy gap contracts from 7.8 p.p. to 1.9 p.p.
  • Replication: The FIFO operating point and keyword dominance replicate across two independent production corpora, but the benchmark lift and pagination-zero claim do not.

6 Discussion

The discussion separates retrieval from selection: rank-1 improvements help only when the gold is retrievable, while composite metadata signals can undermine keyword ordering.

  • 6.1 Composite scorers degrade p1 relative to a single signal: H2 reverses: Priority-ALL falls 4.8 p.p. short of Priority-KW, with McNemar p = 0.001 on identical candidate lists.Depth, extension, recency, and filename signals can down-weight correct candidates in atypical workflow locations.
  • 6.2 Top-1 placement does not predict LLM accuracy: The paired tests reject H3: accuracy changes as p1 rises are insignificant for every model and inconsistently signed.The pre-registered correlation is uninformative because p1 has only two or three distinct values per model.
  • 6.2 Top-1 placement does not predict LLM accuracy: Downstream accuracy depends on first-chunk inclusion rather than the gold’s rank within that chunk.The local models remain at 69–76% accuracy on the large bucket, so the null is not attributed to a ceiling.
  • 6.3 Retrieval ceiling versus selection lift: The grep generator misses the gold in 221 of 500 tasks, capping first-chunk inclusion at 55.8%.This is a retrieval ceiling rather than a selection ceiling.
  • 6.3 Retrieval ceiling versus selection lift: FIFO already places the gold in the first chunk’s top-10 for 86.0% of winnable tasks but ranks it first for only 43.4%.Thus keyword scoring often reorders a chunk that already contains the gold.

7 Related Work

The paper situates first-chunk selection as structural-item compression before rendering, distinct from token-level compression after responses become flat text. It also relates its keyword scorer to bag-of-words retrieval and places the approach within the tool-use protocol layer.

  • Pre-LLM compression: First-chunk selection chooses structural items before rendering, whereas LLMLingua, LLMLingua-2, RECOMP, and ACON compress already merged text streams.The paper measures first-position placement with p1 rather than reconstruction fidelity.
  • Retrieval and ranking for code: The keyword-overlap scorer is a parameter-free bag-of-words retrieval method that could be replaced by learned re-rankers or dense retrieval.The deployed pipeline does not include a learned component.
  • Tool-use protocol layer: First-chunk selection operates in the tool-use protocol layer across MCP ToolAnnotations, Writing Tools for Agents, OpenAI function calling, and ResourceLink.The paper presents p1-targeted compression as orthogonal to these protocol mechanisms.
  • Benchmark: SWE-bench Verified is used as a file-localisation probe rather than an end-to-end resolve-rate test.This distinguishes the benchmark role from full software-engineering task evaluation.

Limitations

The study is bounded by a deterministic grep-based retrieval ceiling, a Python-only benchmark, a single-turn localisation probe, cache-TTL conditions, and limited cross-corpus comparability.

  • Retrieval ceiling: 221 of 500 tasks (44.2%) lack a candidate matching the gold file, capping the present generator’s absolute p1 ceiling at 55.8%.Learned retrieval is proposed but not evaluated; selection is separately measured on winnable tasks.
  • Benchmark scope: SWE-bench Verified covers Python-only tasks from 12 repositories, with Django comprising approximately 46%; non-Python languages and non-repository corpora are untested.
  • Single-turn evaluation: Downstream accuracy is measured with a single-turn file-localisation probe, not an end-to-end SWE-bench resolve-rate evaluation.Patching and test execution remain a follow-up.
  • Cache TTL artefact: The Opus 4.7 batch run had a 12.6% KV-cache hit rate under the default 5-minute TTL, and the study did not rerun it with an extended TTL.A longer TTL is expected to bring performance closer to Sonnet 4.5’s 66.5% hit rate on the same workload.
  • Cross-corpus availability: External replication corroborates the regularity rather than a precise effect size because intent labels came from different, uncontrolled per-corpus LLM judges.The side-by-side replication report cannot be redistributed.

Ethical Considerations

The paper describes privacy-preserving telemetry extraction and reports no misuse scenarios specific to its compression layer. Implementation details include cached prompt prefixes and a deterministic ranking probe.

  • Data handling: The telemetry pipeline retains aggregate per-event features while discarding raw response bodies, project identifiers, and user-facing strings at source.The production telemetry comes from session logs of a public open-source Model Context Protocol middleware.
  • Risk assessment: The authors report no plausible misuse scenarios specific to this work and distinguish broader LLM coding-agent risks as independent of the compression layer.
  • Evaluation implementation: The E2 evaluation uses a stable cached system prefix plus a per-task template shipped with the released code.
  • Evaluation implementation: The localisation probe asks a code-navigation assistant to return one JSON line for a truncated candidate-path list containing the gold.
  • Evaluation implementation: The ranking rules prioritize keyword matches, then source files, shallow source paths, named submodules, exact filenames, and caution around generic names.The probe explicitly ignores ordering as a signal.

B Pre-Registration of Hypotheses

The preregistered evaluation tests keyword ranking, composite metadata signals, the relationship between p1 and downstream accuracy, caching cost, and local-model parity. Results support keyword gains and local-model parity, but reject the composite-scoring and p1–accuracy hypotheses.

  • Hypotheses and outcomes: H2 reversed: Priority-ALL changed p1 by −4.8 p.p. relative to Priority-KW instead of improving it by at least 3 p.p.
  • Hypotheses and outcomes: H3 was unsupported: per-model accuracy shifts ranged from −2.8 to +2.2 p.p. when p1 rose, and none of five models was significant.The preregistered correlation was uninformative because p1 took only 2–3 distinct values across cells.
  • Hypotheses and outcomes: H4 was not directly tested as stated; reported cache-hit rates were 66.5% for Sonnet 4.5 and 12.6% for Opus 4.7 under a default 5-minute TTL.Cache-hit rate upper-bounds rather than equals input-cost reduction because cached input tokens remain billed at a reduced rate.
  • Hypotheses and outcomes: H5 passed: gemma4:26b came within 1.9 p.p. of Claude Opus 4.7 conditional on the gold being ranked first.
  • Depth and downstream accuracy: The keyword advantage is concentrated at rank 1, with p1 +11.6 p.p. but top-10 inclusion only +4.0 p.p.This pattern is consistent with reranking the chunk’s top rather than changing whether the gold enters the multi-item chunk.
  • Selection lift conditional on retrieval: Conditional on retrieval, Priority-KW+ raises p1 from 43.4 to 64.2, a +20.8 p.p. lift over FIFO on 279 winnable tasks.The full-set ceiling remains dominated by candidate-generator recall.
  • Safety fallback: Approximately 14% of tasks have all-zero keyword scores, so Priority-KW+ falls back to FIFO to avoid returning an empty candidate set.For django-12713, the fallback restores the gold’s rank-1 selection.

G Reproducibility Kit

The reproducibility kit documents the benchmark’s hardware, software, costs, released aggregates, and analysis workflow, alongside experiment design, candidate distributions, and replication scope.

  • Environment: AMD Ryzen 5950X, NVIDIA RTX 3090, 128 GB RAM, and Windows 10 define the reported hardware environment.The software environment includes Python 3.12.10, Rust 1.90, Ollama 0.21.0, and uv 0.11.7.
  • Execution and cost: $25.09 was the total cloud billed cost, with approximately 7 hours of total wall time including local inference and aggregation.The algorithmic sweep cost $0, while cloud model runs accounted for the billed total.
  • Selection and accuracy: Table 6 reports budget-invariant top-K inclusion, with the keyword scorer’s gain over FIFO shrinking from p1 +11.6 to p10 +4.0 p.p.Table 7 similarly reports a conditional p1 gain of +20.8 p.p. over FIFO on 279 winnable tasks, while downstream accuracy remains flat.
  • Reproduction workflow: A single make all target reproduces data generation, algorithmic and LLM evaluations, aggregation, and plots from a clean repository.The workflow separates candidate generation, algorithmic sweeps, local and cloud evaluations, and final aggregation.
  • Benchmark structure: The benchmark has 500 tasks, with 136 tasks (27.2%) at the 50-candidate cap and buckets of 288 small, 55 medium, and 157 large tasks.The candidate-list median is 4, while the distribution has a long tail to 50 candidates.
  • Replication scope: External corpora replicate the FIFO baseline and dominant keyword signal, but do not verify the pagination-zero finding or extend benchmark effect sizes to another deployment population.Corpus B’s candidate-list tail differs, and the corpora used different LLM judges for workflow-intent labelling.
Loading 2608.26130v1…