Source-linked AI summary
Coding Agents are Effective Long-Context Processors
Weili Cao, Xunjian Yin, Bhuwan Dhingra, Shuyan Zhou
TL;DR
Long-context models can access massive inputs but often process them ineffectively and opaquely as context grows. The paper instead delegates processing to off-the-shelf coding agents that navigate text organized as files and manipulate it with executable tools. These agents achieve state-of-the-art results on four of five benchmarks spanning 188K to three trillion tokens, with effectiveness attributed to tool proficiency and file-system familiarity.
Problem
As context length increases, LLM performance can degrade, reasoning remains latent and difficult to interpret, and fixed RAG retrieval limits iterative multi-hop processing.
Method
The paper organizes large text corpora as file systems and delegates processing to off-the-shelf coding agents using terminal commands, programmatic search, file manipulation, and iterative execution.
Results
Coding agents achieve state-of-the-art results on four of five benchmarks spanning context lengths from 188K to three trillion tokens.
Takeaways & Limitations
Native tool proficiency and file-system familiarity support task-specific strategies, suggesting structured text aligned with code can enable effective extended-context reasoning without specialized architectures.
Takeaways & Limitations
Naively adding retrieval tools may degrade performance, and off-the-shelf coding agents are optimized for coding rather than long-context reasoning.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated remarkable progress in scaling to access massive contexts. However, the access is via the latent and uninterpretable attention mechanisms, and LLMs fail to effective process long context, exhibiting significant performance degradation as context length increases. In this work, we study whether long-context processing can be externalized from latent attention into explicit, executable interactions, by allowing coding agents to organize text in file systems and manipulate it using its native tools. We evaluate off-the-shelf frontier coding agents as the general interface for tasks that require processing long contexts, including long-context reasoning, retrieval-augmented generation, and open-domain question answering with large-scale corpus contains up to three trillion tokens. Across multiple benchmarks, these agents outperform published state-of-the-art by 17.3% on average. We attribute this efficacy to two key factors: native tool proficiency, which enables agents to leverage executable code and terminal commands rather than passive semantic queries, and file system familiarity, which allows them to navigate massive text corpora as directory structures. These findings suggest that delegating long-context processing to coding agents offers an effective alternative to semantic search or context window scaling, opening new directions for long-context processing in LLMs.
1. Introduction
Long-context scaling expands access but does not ensure effective, interpretable processing. This work proposes using coding agents to externalize processing through executable tools and file-system organization, and reports strong performance across diverse long-context tasks.
- Long-context models often degrade as context length increases, while their latent reasoning provides limited transparency into which context informs generation.
- Standard RAG pipelines use fixed, shallow retrieval mechanisms that limit iterative, multi-hop reasoning for complex long-context questions.
- Coding agents explicitly organize, filter, and transform text with executable programs rather than relying solely on latent attention or fixed retrieval.
- In a 385K-token transcript example, an agent iteratively refined a Python script after inspecting failures and discovering domain-specific spell references.
- Coding agents consistently outperform strong baselines across settings, reaching state-of-the-art on four of five benchmarks across context scales from hundreds of thousands to trillions of tokens.
- The analysis attributes effectiveness to native tool proficiency and file-system familiarity, alongside emergent strategies such as query refinement and programmatic aggregation.
2. Text Processing as File System Operation
The approach reformulates long-context processing as file-system navigation: documents become files in directory structures, and coding agents autonomously explore and manipulate them with native tools.
- The task is defined as producing an answer from a query by reasoning over either a large document corpus or a single long document.
- Large corpora are represented as document files organized within a corpus directory, while single long documents are placed in one text file.
- The file-system representation supports navigation, programmatic aggregation, saved intermediate results, and iterative query refinement for multi-hop reasoning.
- The agent receives a file or directory path and query, then uses terminal commands, Python scripts, intermediate files, and iterative exploration.
- The system imposes no fixed processing strategy, allowing agents to scan files, construct searches, write parsers, or combine methods autonomously.
3. Experiments
The experiments evaluate coding agents across benchmarks spanning multi-document retrieval, long-document reasoning, and diverse long-context tasks, against full-context, RAG, ReAct, and recursive-language baselines. The study also examines file-system structure and alternative coding-agent implementations.
- Benchmarks: The evaluation covers BrowseComp-Plus, Natural Questions, LongBench-v2, and Oolong variants, spanning multi-document QA, long-document reasoning, and diverse real-world tasks.BrowseComp-Plus and Natural Questions require synthesizing information across massive corpora, while LongBench-v2 and Oolong test reasoning over long documents.
- Benchmarks: BrowseComp-Plus uses a fixed corpus of 100K web documents containing the gold documents and evaluates multi-hop answers with GPT-5 judging.The reported metric is accuracy.
- Benchmarks: LongBench-v2 evaluates six categories of long-context understanding with multiple-choice questions and reports accuracy.Its categories include single- and multi-document QA, long in-context learning, dialogue history, code repositories, and structured data.
- Baselines: The comparison includes full-context GPT-5, standard RAG, ReAct-style search agents, recursive language models, and coding agents with native, BM25, or dense retrieval configurations.RLM is excluded from BrowseComp-Plus because the full 100K-document evaluation is prohibitively time-consuming.
- Ablations and agent implementations: The file-system ablation compares folder structures with a single-file configuration on a 100-example BrowseComp-Plus subset.The study also evaluates Claude Code on Oolong-Real and LongBench, but budget constraints limit that evaluation to two benchmarks.
4. Main Results
Coding agents outperform the evaluated baselines across diverse benchmarks and context scales, while the comparisons include important evaluation caveats for full-context GPT-5 and RLM.
- Main results: Coding agents significantly outperform all baselines across diverse benchmarks, with gains spanning average 188K-token contexts to corpora exceeding three trillion tokens.The reported conclusion is that off-the-shelf agents provide a general-purpose solution without task-specific training or architectural modifications.
- Evaluation caveats: GPT-5 full-context accuracy is 20.0% on BrowseComp-Plus and 27.0% on NQ, with the authors attributing these non-trivial scores likely to data contamination.On Oolong, the comparison includes longer contexts than the original evaluation, where performance degrades significantly.
5. Ablations and Analysis
The ablations show that navigable folder structures improve agent performance and shape more selective exploration, while retrieval tools can displace native search. Agents also adapt their processing strategies to task demands, using iterative search for multi-hop retrieval and code for analytical aggregation.
- 5.1. File System Structure Matters: Folder structure outperforms a single-file corpus configuration across retriever settings.The comparison tests repository-like directories against a single JSON dictionary.
- 5.1. File System Structure Matters: With folders, agents use coordinate-based reading that indexes files and slices relevant line ranges.The reported strategy combines nl for line indexing with sed for selective extraction; sed usage increases by over seven times in the folder setting.
- 5.2. Retrieval Tools Do Not Uniformly Improve Performance: Retrieval tools do not consistently improve performance and can reduce agents’ use of native search commands.Agents without retrieval tools issue substantially more native searches, including grep, than retriever-augmented variants.
- 5.3. Emergent Task-Specific Processing Strategies: On BrowseComp-Plus, agents iteratively refine searches by extracting entities and relationships from intermediate documents.This emergent multi-hop strategy is illustrated by a six-hop chain across linked entities.
- 5.3. Emergent Task-Specific Processing Strategies: On Oolong analytical tasks, agents shift from search toward code generation, with lower reading and higher code volume.The programmatic strategy parses documents, applies regex-based rules, and aggregates results, refining patterns when edge cases appear.
- 5.3. Emergent Task-Specific Processing Strategies: Coding agents dynamically vary tool use across tasks rather than following a fixed processing strategy.The broader analysis characterizes this flexibility as a distinction from fixed-action or uniform recursive approaches; costs remain competitive with strong methods despite exceeding lightweight RAG baselines.
6. Related Work
Related work identifies limitations in long-context models and fixed RAG pipelines, while positioning external-environment approaches and coding agents as adjacent alternatives. The paper distinguishes its use of off-the-shelf agents and native file-system tools from recursive language models and task-specific coding systems.
- Long-Context Language Models: Long-context models can process long documents but often show substantial performance degradation as context length increases.Prior work reports that models may lose much of their short-context capability before reaching advertised context limits.
- Agentic RAG: Traditional RAG uses fixed retrieval followed by generation, limiting iterative refinement and multi-hop reasoning.Agentic RAG addresses this limitation by dynamically reformulating queries based on intermediate findings.
- Agent with Long-Term Memory: Memory-centric agent architectures manipulate stored context, whereas this work places context in the environment as files.The distinction concerns where context is externalized: agent memory in prior work versus an interactive file environment here.
- Recursive Language Models: Recursive Language Models and this work both treat long text as an external environment that models actively explore through actions.RLMs use recursive sub-calls over text segments, while this work uses off-the-shelf coding agents with native file-system tools and custom scripts.
- Coding Agents: Coding agents incur higher costs than lightweight RAG baselines but remain competitive with or cheaper than other strong methods.The cost comparison is reported across benchmarks alongside substantially superior performance.
- Coding Agents: Prior coding-agent work largely targets reasoning or software engineering, while one cited study reports poor long-context processing performance.The paper frames its setting as general text processing rather than long-horizon coding tasks over codebases.
7. Conclusion and Future Work
Off-the-shelf coding agents provide an effective paradigm for long-context processing, while the paper identifies native tool proficiency and file-system familiarity as key factors. The authors also describe limitations involving retrieval integration and agents’ coding-oriented optimization.
- Conclusion: State-of-the-art results were achieved on four of five benchmarks spanning context lengths from 188K to three trillion tokens.The approach uses file-system navigation, terminal commands, programmatic search, and iterative script refinement without task-specific training or architectural modifications.
- Conclusion: Native tool proficiency and file-system familiarity underlie the agents’ effectiveness in navigating hierarchically organized text.These capabilities support executable interactions beyond natural-language retrieval queries and provide inductive priors for corpus navigation.
- Conclusion: Coding agents autonomously develop task-appropriate strategies, including iterative query refinement and programmatic aggregation.These strategies are presented as emergent behaviors associated with the agents’ native capabilities.
- Conclusion: Structuring text as files aligned with code can support effective reasoning over extended contexts without specialized long-context architectures.The conclusion connects file-oriented organization with reduced reliance on specialized architectures.
- Future Work: Naively providing retrieval tools may degrade performance, and future work should improve integration without suppressing native exploration.The paper hypothesizes that default reliance on imperfect retrieval ranking can displace broader filesystem exploration, while leaving the precise mechanism open.
- Future Work: Future work should specialize coding agents for navigating and reasoning over massive text corpora.This direction follows the reported transfer from coding to text-processing tasks.
B. Case Studies: Agent-Generated Scripts
The case studies show coding agents converting long-document tasks into structured scripts that parse boundaries, filter relevant content, extract patterns, and aggregate results. The examples cover dice-roll statistics and character-specific spell tracking across episodes.
- Example 1: Counting Dice Rolls: The dice-roll case study asks an agent to count rolls with a specific value and compute their percentage in a long transcript.The agent-generated script reads the transcript, identifies episode boundaries, processes player dialogue, and computes the requested statistic.
- Example 1: Counting Dice Rolls: The script computes the target-roll percentage from the total number of extracted rolls.The reported calculation divides the count of rolls matching the target value by the total roll count.
- Example 2: Tracking Character Actions Across Episodes: The spell-tracking case study identifies the last spell cast by Vax’ildan in each episode of a multi-episode transcript.The approach separates episodes, filters character-related lines, matches spell-related content, and extracts the last occurrence.
- Example 2: Tracking Character Actions Across Episodes: The spell-tracking script splits the transcript into episodes, filters lines involving the target character, and records matching spell lines.Episode boundaries are detected from transcript markers, while speaker names, character mentions, keywords, and a predefined spell list guide extraction.
B.3. Key Observations
The case studies reveal structured parsing, robust pattern matching, programmatic aggregation, and domain adaptation as recurring features of the agents’ processing. These behaviors emerge without explicit instruction and reflect transferred software-engineering skills.
- Key Observations: Agents leverage document structure, such as episode markers and speaker prefixes, instead of treating long text as unstructured.This structured parsing supports both episode segmentation and speaker- or character-specific filtering.
- Key Observations: Multiple regex patterns accommodate variation in how information is expressed, such as “rolled 15” versus “rolled a fifteen.”The observation emphasizes robustness to surface-form variation during extraction.
- Key Observations: Programmatic aggregation processes entire documents systematically rather than retrieving only a few relevant passages.This strategy is presented as a way to support complete coverage of distributed information.
- Key Observations: Agents incorporate domain knowledge, including player names, spell lists, and D&D conventions, into parsing logic.Domain-specific information is used to guide extraction from the long documents.
- Key Observations: These behaviors emerge without explicit instruction, demonstrating transfer of software-engineering skills to text-processing tasks.The conclusion connects the observed strategies to the agents’ coding capabilities.
C.1. BrowseComp-Plus: Iterative Query Refinement
The agent solves a six-hop entity-chain question by progressively refining searches from discovered entities and verifying each link. It autonomously moves from identifying Riot Games and its founders to tracing the chain to Max Mazanov.
- Task setup: The task asks for an American professional gamer connected through game, company, founder, spouse, and doctorate-university constraints.The target must satisfy six linked conditions spanning identity, achievement timing, game release, company founding, marriage, expertise, and education.
- Initial decomposition: The agent begins with broad question keywords, then pivots to Riot Games and its co-founders after finding no direct matches.This decomposition changes the search target from the full question to identifiable entities.
- Iterative refinement: It refines queries across Brandon Beck, Natasha Beck, and Pepperdine University to verify the spouse relationship, expertise, and doctorate.The agent confirms Natasha Beck as a parenting and public health expert and identifies her Psy.D. from Pepperdine University, founded in 1937.
- Final verification: The final search identifies Max Mazanov as an American Valorant player born in 2002, and verification confirms his relevant achievement and game connection.The verification links Valorant to Riot Games and confirms Mazanov’s Valorant Champions 2023 MVP.
- Observed strategy: 15+ search operations progressively chain Riot Games → Brandon Beck → Natasha Beck → Pepperdine → Valorant → Demon1 → Max Mazanov.This six-hop reasoning process emerges without explicit instruction.
C.2. Oolong-Synthetic: Programmatic Aggregation
For NLI aggregation, the agent abandons search and processes all 1,772 sentence pairs programmatically with a rule-based contradiction classifier. It reports User 23934 as having the most contradiction instances.
- Task setup: The task is to identify which user has the most contradiction-labeled instances among 1,772 sentence pairs.The labels are not provided, so the agent must infer contradiction from each pair.
- Programmatic parsing: The agent parses the structured records with Python, extracting user IDs and sentence pairs for systematic processing.The records contain dates, users, and paired sentences whose relations must be classified.
- Rule-based classification: Its rule-based NLI classifier detects negation, quantity mismatch, numerical disagreement, and antonym pairs as contradiction patterns.These patterns provide the linguistic rules used by the custom contradict() function.
- Aggregation and limitation: The classifier runs over all 1,772 pairs and aggregates contradiction counts per user with a Counter object.An attempted PyTorch and Transformers refinement fails because of environment conflicts, so the rule-based results are retained.
- Result: User 23934 is identified as having the most contradiction instances.The result follows complete abandonment of retrieval in favor of programmatic processing, with zero search commands and approximately 300 lines of Python.
C.3. LongBench: Hybrid Search-Read Strategy
The glacier-mouse question is answered through a hybrid strategy that alternates targeted search with focused reading across scattered document sections. The agent concludes that rolling helps obtain nutrients rather than primarily providing thermal protection.
- Task setup: The task asks why glacier mice roll during the warm season, with nutrient acquisition and thermal protection among the choices.The options distinguish getting nutrients from water discharge, hiding from the sun, and preserving body heat.
- Initial search: The agent first identifies the scientific document and searches for motion-related terms such as roll, stick, and creep.These searches locate sections discussing the mechanics and contexts of glacier-mouse movement.
- Evidence gathering: It examines thermal and nutrient-related passages, including the thermally ameliorated environment and the nutrient base necessary for plant life.The search also finds that positioning can provide a nutrient supply during summer.
- Evidence synthesis: Focused reading confirms the habitat is moist, relatively warm, and supplied with food, while thermal protection is secondary to nutrient redistribution.The evidence supports interpreting rolling primarily through nutrient movement rather than solar or heat avoidance.
- Result: 17 operations alternating between rg searches and sed reading yield the final answer: (B) Get nutrients.The answer requires synthesizing information scattered across multiple sections.