PaperlayerView exact revision on arXiv

Source-linked AI summary

LOCA-bench: Benchmarking Language Agents Under Controllable and Extreme Context Growth

Weihao Zeng, Yuzhen Huang, Junxian He

arXiv:2602.07962v1cs.AI

TL;DR

Long-context benchmarks often miss the dynamic exploration, instruction following, and action selection required by realistic agents. LOCA-bench addresses this by automatically expanding environment states while preserving task semantics, finding that performance generally degrades with context growth while context engineering can improve success rates.

  • Problem

    Existing long-context benchmarks mainly test static retrieval or single-step aggregation, whereas realistic agents must explore environments and act under dynamically growing context.

  • Method

    LOCA-bench varies environment description length automatically and controllably while keeping task semantics fixed, evaluating models together with context-management scaffolds.

  • Results

    Most models show a sharp accuracy drop as context length increases, while context engineering strategies can substantially improve performance.

  • Takeaways & Limitations

    LOCA-bench provides a platform for measuring long-context behavior in realistic agentic settings and comparing context engineering across models and scaffolds.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly capable of carrying out long-running, real-world tasks. However, as the amount of context grows, their reliability often deteriorates, a phenomenon known as "context rot". Existing long-context benchmarks primarily focus on single-step settings that evaluate a model's ability to retrieve information from a long snippet. In realistic scenarios, however, LLMs often need to act as agents that explore environments, follow instructions and plans, extract useful information, and predict correct actions under a dynamically growing context. To assess language agents in such settings, we introduce LOCA-bench (a benchmark for LOng-Context Agents). Given a task prompt, LOCA-bench leverages automated and scalable control of environment states to regulate the agent's context length. This design enables LOCA-bench to extend the context length potentially to infinity in a controlled way while keeping the underlying task semantics fixed. LOCA-bench evaluates language agents as a combination of models and scaffolds, including various context management strategies. While agent performance generally degrades as the environment states grow more complex, advanced context management techniques can substantially improve the overall success rate. We open-source LOCA-bench to provide a platform for evaluating models and scaffolds in long-context, agentic scenarios: https://github.com/hkust-nlp/LOCA-bench

1. Introduction

LOCA-bench addresses the gap between static long-context retrieval benchmarks and dynamic agentic tasks whose contexts grow through exploration. It controls context growth by expanding environment descriptions while preserving task semantics and evaluates models together with context-management scaffolds.

  • Motivation: Existing long-context benchmarks mostly provide information upfront or reduce evaluation to retrieval and single-step aggregation.They do not capture agents that begin with limited environmental knowledge and progressively gather observations through interaction.
  • Benchmark goal: LOCA-bench evaluates long-context agents that explore tool-grounded environments, follow instructions, extract information, and take correct actions.Its tasks are drawn from realistic scenarios rather than isolated retrieval settings.
  • Controllable context growth: Environment description length increases the information available in the initial state while the underlying task prompt and semantics remain fixed.This creates controlled growth in the context agents must handle during exploration.
  • Agentic challenges: The benchmark combines complex retrieval and reasoning, instruction following, and environment exploration challenges that emerge as context grows.Agents must jointly process multiple tool outputs and preserve constraints across interactions.
  • Scaffolds and context engineering: LOCA-bench evaluates models as combinations of language models and scaffolds, including context editing, memory, context awareness, and programmatic tool calling.The framework is designed to compare context-management strategies across agentic systems.
  • Scaffolds and context engineering: Context engineering strategies can substantially improve performance, with frontier models generally benefiting more than open-source models.Programmatic tool calling is also reported to reduce intermediate exploration cost while improving tool orchestration and control flow.

2. LOCA-Bench

LOCA-bench constructs realistic, verifiable agentic tasks with automatically scalable environment states and measurable description lengths. Its modular implementation separates environments, models, and scaffolds so users can extend tasks and context-management evaluations.

  • Design principles: LOCA-bench targets realistic agentic settings where models explore environments through tools, combine information from multiple sources, and reason through edge cases.This design goes beyond single-step retrieval from long text.
  • Design principles: The benchmark controls context growth by systematically expanding environment states while keeping task semantics fixed.Environment complexity can therefore vary without changing what the task fundamentally requires.
  • Adjustable environment state: Users configure environment parameters such as the number of courses, source proportions, exceptions, and distracting content, which generators instantiate from predefined templates.The resulting states can include Canvas pages, announcements, email messages, exams, and other task-specific content.
  • Environment description length: Environment description length is measured by tokenizing concatenated tool outputs that an agent would need to read during scripted interaction.The metric treats the token count of aggregated environment information as a complexity measure.
  • Implementation: LOCA-bench contains 15 seed tasks, seven target lengths from 8K to 256K tokens, five random seeds per length, and 525 total samples.The benchmark spans 280 tools across diverse services and environments.
  • Implementation: The implementation decouples environments, models, and agentic scaffolds and uses execution-based binary evaluation for reproducible success measurement.The toolkit also supports extending tasks, context lengths, and context-engineering strategies.

3. Experiment

Experiments evaluate frontier and open-source models across environment descriptions from 8K to 256K tokens using accuracy, efficiency, and trajectory analyses. Accuracy generally falls sharply with longer contexts, while models exhibit declining reasoning, weaker instruction following, insufficient exploration, and evidence-consistency failures.

  • 3.2. Main Results: Accuracy drops quickly across models as environment description length increases, with frontier models reaching roughly two to three times open-source accuracy in longer settings.Most models exceed 70% accuracy at 8K, while the performance gap begins around 32K and widens thereafter.
  • 3.2. Main Results: Claude-4.5-Opus reaches 96% accuracy at 8K, while GPT-5.2-Medium remains relatively strong through 256K and DeepSeek-V3.2-Thinking stays competitive with frontier models up to 64K.These are model-specific patterns within the broader context-length degradation.
  • 3.2. Main Results: After 96K, trajectory length and tool-call counts largely plateau even as environment state size continues growing linearly.The plateau indicates limited exploration rather than proportionally increased probing of the environment.
  • 3.2. Main Results: Models retrieving more tool-output tokens tend to perform better, while frontier models retrieve substantially more tool output than open-source models.Shorter maximum context windows can force repeated tool calls when earlier content is truncated.
  • 3.3. Failure Mode Analysis: Longer contexts produce declining complex reasoning, weaker instruction following, insufficient exploration, and hallucination-like inconsistencies.These failures involve missed cross-source mappings, violated output schemas, premature stopping, and distorted values after correct retrieval.
  • 3.3. Failure Mode Analysis: In one exploration failure, the agent checks only the first 100 products, stops after finding no matches, and produces an incorrect empty CSV despite a larger catalog.The example illustrates how partial evidence can be mistaken for complete environment review.

4. Context Engineering for Agents

The section evaluates context-engineering strategies as part of language-agent scaffolds, finding that advanced tool-use methods can improve accuracy and reduce context-related costs, although benefits vary by model and scaffold.

  • Context engineering setup: Context engineering is evaluated as a combination of model and scaffold strategies for managing limited context during long agent interactions.The evaluation includes context editing, context awareness, memory tools, and programmatic tool calling.
  • Evaluation setup: At 128K environment-description length, the study compares frontier and open-source models under different context-engineering strategies using accuracy and trajectory length.Claude-4.5-Opus is evaluated only with programmatic tool calling because of higher testing cost.
  • Model differences: Frontier models benefit more than DeepSeek-V3.2-thinking from context-awareness and memory tools, which can hurt the open-source model.The reported comparison shows model-dependent effectiveness rather than uniform gains across models.
  • Strategy effectiveness: Advanced tool-use methods outperform crude deletion-based editing, with memory tools and context awareness producing larger improvements for Gemini-3-Flash and GPT-5.2-Medium.Deletion-based approaches include tool-result clearing, thinking-block clearing, and related methods that mainly reduce context by removing content.
  • Programmatic tool calling: Programmatic tool calling consistently improves accuracy while reducing trajectory length across models.It avoids long intermediate tool outputs and converts verbose tool interactions into compact, code-driven workflows that support systematic handling of edge cases.
  • Scaffold effects: Scaffolds can reduce performance when their built-in features encourage inappropriate behavior, as Claude Agent led Claude-4.5-Opus to launch many irrelevant parallel subcalls.The framework’s advanced features mainly accelerated accumulation of irrelevant context rather than task progress in the reported trajectories.

5. Conclusion

LOCA-bench evaluates long-horizon language agents in realistic tool-using environments while scaling context in a controlled manner. The authors report that longer contexts sharply reduce performance, whereas context-engineering strategies can mitigate this pressure and improve success rates.

  • Conclusion: LOCA-bench evaluates agents that explore environments, follow instructions and plans, extract information, and take correct actions over long horizons.The benchmark targets realistic agentic settings rather than only single-step retrieval.
  • Conclusion: The benchmark automatically expands environment states while keeping task semantics fixed, enabling scalable context growth without changing the underlying task.This design supports evaluation across increasingly long contexts.
  • Conclusion: Most frontier and open-source models show a sharp performance drop as context length increases, with the gap widening at longer contexts.The conclusion reports this pattern across the evaluated models.
  • Conclusion: Context engineering strategies can reduce effective context length, mitigate growing-environment pressure, and sometimes improve overall success rates.The toolkit includes clearing tool outputs and intermediate reasoning, context awareness, memory tools, and programmatic tool calling.

A. Statistics across environment description lengths

The benchmark evaluates model performance across environment-description lengths from 8K to 256K tokens and reports accuracy alongside trajectory and tool-use statistics.

  • Statistics across environment description lengths: Environment description lengths range from 8K to 256K tokens across seven evaluation settings.The settings are 8K, 16K, 32K, 64K, 96K, 128K, and 256K.
  • Statistics across environment description lengths: Table 1 reports model accuracy, while Tables 4–6 report trajectory length, tool-call count, and tool-output length.These measurements characterize both success and interaction behavior across context lengths.

B. Tool Sets Used in Tasks

The task suite uses servers that provide collections of tools; the Canvas server contains nearly 70 tools, including assignment and quiz retrieval.

  • Tool Sets Used in Tasks: Each task requires servers that provide collections of tools for agent interaction.Table 7 lists the servers required by each task.
  • Tool Sets Used in Tasks: The Canvas server contains nearly 70 tools, including get assignment and get quiz.This illustrates the breadth of available tools in one task environment.

C. Failure Mode Examples

Figures 5–7 illustrate three failure modes under long-context agentic tasks: declining complex reasoning, weaker instruction following, and hallucination.

  • Figures 5, 6, and 7 demonstrate declining complex reasoning, weaker instruction following, and hallucination, respectively.

D. Programmatic Tool Calling Examples

The section illustrates how agents handle varied tool-based tasks, including programmatic tool calling, while showing failures in reasoning, instruction following, and data transcription. The examples span Canvas, Email, Excel, BigQuery, CSV, WooCommerce, and Google Sheets workflows.

  • Environment Details: The section also identifies tool-output length under different environment-description lengths and lists the servers and tools used by each task.These descriptions provide context for the environments supporting the illustrated workflows.
  • Failure Examples: A complex-reasoning failure omits exam information from emails and Canvas course identifiers, so the Excel file includes only announcement-based exams.The task required gathering exam details from both Canvas announcements and email notifications, then linking each exam to its course.
  • Failure Examples: An instruction-following failure uses “A conversion pct” and “B conversion pct” instead of the required “A conversion %” and “B conversion %,” causing evaluation failure.The task involved analyzing BigQuery data and recording the conversion rate in CSV format.
  • Failure Examples: A hallucination failure queries the correct M006 sensor data but writes incorrect values into Python code, producing an anomaly report with in-range data.The workflow queries BigQuery’s live_sensor table for readings outside their normal range and compiles the result into anomaly_report.csv.
  • Programmatic Tool Calling: Programmatic tool calling invokes WooCommerce programmatically to find products below a stock threshold and explicitly handles pagination.The example follows inspection of the tool output format and uses code to perform the threshold check.
Loading 2602.07962v1…