PaperlayerView exact revision on arXiv

Source-linked AI summary

ACON: Optimizing Context Compression for Long-horizon LLM Agents

Minki Kang, Wei-Ning Chen, Dongge Han, Huseyin A. Inan, Lukas Wutschitz, Yanzhi Chen, Robert Sim, Saravan Rajmohan

arXiv:2510.00615v3cs.AIcs.CL

TL;DR

Long-horizon agents face unbounded context growth that raises inference memory costs and can degrade reasoning, while existing compression methods remain brittle or require impractical parameter updates. ACON optimizes natural-language compression guidelines through failure analysis for histories and observations, then distills the compressor into smaller models. Across three benchmarks, it reduces peak token usage by 26–54% while improving task success over existing baselines and improving small-LM performance by up to 46%.

  • Problem

    Long-horizon agents need to retain precise, changing state information, but unbounded contexts create memory costs and reasoning degradation that existing compression methods do not fully solve.

  • Method

    ACON iteratively optimizes natural-language compression guidelines using agent failure analysis for observations and histories, without updating agent weights, and distills optimized compressors into smaller models.

  • Results

    26–54% lower peak token usage with improved task success over existing compression baselines was observed across AppWorld, OfficeBench, and Multi-objective QA.

  • Takeaways & Limitations

    ACON supports model-agnostic context compression, including proprietary API-based LLMs, while enabling smaller LMs to function more effectively as long-horizon agents.

  • Takeaways & Limitations

    Evaluation primarily focuses on GPT models, and compression itself adds computational overhead and latency.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed as agents in dynamic real-world environments, where success depends on maintaining precise records of actions and observations. However, the resulting unbounded context growth in long-horizon agentic tasks makes two critical bottlenecks: prohibitive inference memory costs and reasoning degradation due to irrelevant information. Existing compression methods fail to fully address this, often relying on brittle heuristics or requiring parameter updates impractical for proprietary or large-scale LLMs. We introduce Agent Context Optimization (ACON), a unified framework that optimally compresses both observations and history into concise, informative representations. Distinct from prior works, ACON employs an optimization in natural language space: it iteratively refines compression guidelines based on failure analysis of the agent, ensuring critical state information is preserved without model fine-tuning. To further minimize computational overhead, we distill the optimized compressor into smaller models. Experiments on AppWorld, OfficeBench, and Multi-objective QA demonstrate that ACON reduces peak token usage by 26-54% while improving task success over existing compression baselines. Notably, it enables smaller LMs to function effectively as long-horizon agents, achieving up to 46% performance improvement by mitigating context distraction. Our code is available at https://github.com/microsoft/acon.

1. Introduction

Long-horizon agents require compression that reduces unbounded context costs while preserving task-relevant state. ACON addresses this with model-agnostic guideline optimization, compressor distillation, and evaluation across three multi-step benchmarks.

  • Motivation: Long-horizon agents must preserve observations, tool outputs, evolving states, and task-relevant details as interactions accumulate.Losing details such as file paths or API parameters can derail workflows, while excessively long contexts increase inference costs.
  • Limitations of prior work: Existing compression methods are insufficient because they target conversational coherence, single-step document use, or brittle specialized heuristics.These approaches do not fully address dynamic state tracking in long-horizon agentic tasks.
  • ACON: ACON optimizes natural-language compression guidelines through failure analysis for both environment observations and interaction histories without updating agent weights.The framework is model-agnostic and applicable to proprietary API-based LLMs.
  • ACON: ACON distills optimized compressors into smaller models, reducing compression overhead while preserving over 95% of the teacher’s performance.This supports deployment of the compression module with minimal computational overhead.
  • Evaluation: 26–54% lower peak token usage accompanies improved task success over existing compression baselines across AppWorld, OfficeBench, and Multi-objective QA.Each benchmark requires more than 15 interaction steps.
  • Evaluation: Small LMs improve by 32% on AppWorld, 20% on OfficeBench, and 46% on Multi-objective QA when long-context distraction is mitigated.These results indicate more effective agent operation for smaller models across the evaluated benchmarks.

2. Related Works

Prior context-compression work spans document or retrieval methods and agent-focused approaches, but dynamic long-horizon agents require broader state-aware compression. ACON targets this gap with a universal, model-agnostic framework for compressing both histories and observations.

  • Long-horizon agents: Long-horizon agents iteratively observe environments, select tools, execute actions, and revise plans, creating continuously changing context.This differs from chatbots or single-pass solvers.
  • Context compression for LLMs: Context-compression research includes document- or retrieval-based approaches, but relevance shifts as agents progress through dynamic, heterogeneous contexts.These settings remain insufficiently addressed by existing research directions.
  • Agent context compression: Recent agent-focused methods rely on naive prompting, narrow domains, or reinforcement learning to optimize context compression.These choices limit broader applicability or introduce additional optimization demands.
  • ACON: ACON provides universal optimization for arbitrary LLMs, supporting both history and observation compression through a generalizable process.Its model-agnostic design applies to open-source and proprietary API-based models.

3. Agent Context Optimization (ACON)

ACON formulates context compression as a joint optimization of interaction histories and observations for long-horizon agents. It refines natural-language compression guidelines using trajectory-level failure feedback, then selects and distills compressors while balancing task reward against context cost.

  • 3.1. Problem Formulation: ACON models long-horizon agent context as histories and latest observations whose accumulated encoding cost can grow without bound.The framework treats context cost as the per-step cost of encoding dynamic context during action generation.
  • 3.2. History & Observation Compression with LLMs: The compressor can selectively replace oversized histories or observations with compressed representations while leaving shorter inputs unchanged.History compression is triggered above T_hist, and observation compression above T_obs, limiting unnecessary compressor overhead.
  • 3.2. History & Observation Compression with LLMs: ACON optimizes compressor parameters to maximize task reward while minimizing context cost, with the agent model and task prompt fixed.The compressor may involve pretrained weights and natural-language guidelines, while the acting agent remains fixed during the formulation.
  • 3.3. Optimizing Compression Guidelines: The optimization is difficult because compression lacks gold supervision, rewards are sparse and terminal, costs are discrete, and reinforcement-learning alternatives can be expensive and high-variance.Updating LLM parameters and executing multi-step agent-compressor rollouts are identified as major practical obstacles.
  • 3.3. Optimizing Compression Guidelines: The method refines natural-language compression guidelines rather than fine-tuning compressor parameters, using failures under compression as feedback about lost information.Tasks where the agent succeeds without compression but fails with compression form a contrastive subset for feedback generation.
  • 3.3. Optimizing Compression Guidelines: An LLM aggregates trajectory feedback into textual optimization steps, generates candidate prompts, evaluates them on contrastive tasks, and selects the best-performing guideline.The process is illustrated as guideline optimization driven by contrastive task feedback and utility maximization.

4. Experiments

ACON is evaluated on three multi-step benchmarks using accuracy and token-efficiency metrics, with experiments covering compression quality, distillation, small agents, thresholds, optimizers, and costs.

  • Experimental setup: ACON is evaluated on AppWorld, OfficeBench, and 8-objective QA using task performance, steps, peak tokens, and dependency.The experiments use ReAct agents and compare against no compression, FIFO, retrieval, LLMLingua, and prompting baselines.
  • Overall performance and token efficiency: 26–54% lower peak token usage accompanies improved task success over existing compression baselines across the evaluated benchmarks.On AppWorld, ACON reduces peak tokens by over 25% while preserving no-compression accuracy; on OfficeBench, it reduces peak context by nearly 30% while maintaining accuracy above 74%.
  • Overall performance and token efficiency: 54.5% lower peak tokens and 61.5% lower dependency are reported for 8-objective QA, while ACON surpasses the no-compression baseline in EM/F1.This result is reported alongside the OfficeBench and AppWorld efficiency findings.
  • Overall performance and token efficiency: UT improves performance while reducing token cost, whereas CO further reduces cost but may slightly hurt accuracy depending on the environment.UTCO is often preferred for verbose, noisy AppWorld observations, while UT is generally safer for high-fidelity OfficeBench and 8-objective QA tasks.
  • Compressor distillation: Distilled compressors retain over 95% of the gpt-4.1 teacher’s performance while reducing computational overhead.Qwen3-14B, Qwen3-8B, and Phi-4 are distilled using optimized guidelines after UT; gpt-4.1-mini also serves as an effective compressor without distillation on OfficeBench and QA.
  • ACON for distilled small agents: ACON improves Qwen3-14B agent performance by 32.4% on AppWorld and 45.6% on 8-objective QA.On AppWorld, performance rises from 25.6% to 33.9%; on 8-objective QA, EM rises from 0.158 to 0.23.
  • Threshold ablations: Moderate compression thresholds provide the best trade-off: 4096 for history and 1024 for observations.Smaller thresholds reduce tokens but increase compression frequency and degrade accuracy, while larger thresholds preserve accuracy at higher cost.
  • Prompt optimizer ablation: The o3 optimizer with contrastive feedback yields the best guideline performance in the AppWorld ablation.Removing contrastive feedback or switching optimizer models results in lower accuracy, although weaker optimizers can still produce sufficiently fine guidelines relative to baseline.

5. Conclusion

ACON compresses interaction histories and environment observations for long-horizon agents, reducing peak tokens while improving task success and enabling smaller compressor models. The authors note remaining evaluation and latency limitations.

  • ACON systematically compresses interaction histories and environment observations for long-horizon LLM agents.
  • 26–54% lower peak token usage accompanies improved task success over existing compression baselines.
  • Distilling optimized compressors into smaller models substantially lowers compression overhead without sacrificing performance.
  • The empirical evaluation primarily focuses on GPT models, and compression introduces computational overhead and increased latency.

Impact Statement

The work targets the computational costs of long-horizon autonomous agents by optimizing context compression and distilling compressors into smaller models. Its stated impact is more resource-efficient and accessible agentic capability.

  • ACON addresses computational costs in long-horizon autonomous agents through optimized context compression and distillation into smaller models.
  • The approach primarily contributes to resource-efficient and more accessible advanced agentic capabilities.
  • The work has positive implications for reducing AI systems’ computational footprint and lowering research barriers.

A. Limitations & Future Works

The paper identifies scope, deployment, optimization, distillation, and latency boundaries that constrain the current evidence and efficiency of ACON. Future work targets broader validation, stronger optimization foundations, more scalable distillation, and improved cache handling.

  • Scope of empirical evaluation: The empirical evaluation primarily uses GPT models, leaving generalizability to other foundation models insufficiently verified.
  • Real-world deployability: The benchmark environments are controlled, so in-the-wild deployment and live multi-agent production integration remain open challenges.
  • Convergence of natural language space optimization: Natural-language guideline optimization lacks a formal convergence guarantee despite candidate sampling and selection.
  • Distillation gap and data scalability: A performance gap remains between distilled and teacher models, potentially reflecting only 100 training examples per domain.
  • Latency and KV-cache dynamics: Compression can invalidate the KV-cache and require recomputation, while generation overhead adds latency and offsets some efficiency gains.

B.1. Datasets

The evaluation uses AppWorld, OfficeBench, and 8-Objective QA to test long-horizon agents across application coordination, office automation, and deep research-style tasks. The supplied passages also reference example tasks in Table 5.

  • AppWorld: AppWorld integrates nine everyday applications through 457 APIs and averages 42.5 API calls per task.
  • OfficeBench: OfficeBench evaluates office automation across Word, Excel, PDF, Calendar, Email, Shell, and Calculator applications.
  • OfficeBench: OfficeBench experiments use text-related tasks, with 92 training tasks and 95 test tasks after a 1:1 random partition.
  • 8-Objective QA: 8-Objective QA poses eight distinct questions within one task and requires answers to all of them at the end.
  • Table 5 contains example tasks across the benchmarks.

B.2. Evaluation Metrics

The evaluation uses peak tokens and dependency to characterize token efficiency and computational cost, alongside API pricing for cost analysis.

  • Peak Tokens: Peak tokens measure the maximum number of tokens in any single trajectory sequence, excluding system prompts.This serves as a proxy for inference-time memory requirements.
  • Dependency: Dependency is defined as the area under the curve in Figure 2 across the agent trajectory.It approximates cumulative computational cost for action generation.
  • Dependency: Dependency uses input and output token counts at each step t to quantify cumulative generation cost.
  • API Cost: API cost analysis uses official OpenAI prices for gpt-4.1 and gpt-4.1-mini, and approximated OpenRouter pricing for Qwen3-14B.The listed prices distinguish input and output tokens for each model.

B.3. Implementation Details & Hyperparameters

The implementation fixes inference settings, uses benchmark-specific compression thresholds, optimizes prompts through candidate selection, and applies standardized distillation and baseline configurations.

  • API Inference: Inference uses temperature 0.0, seed 42, and Azure OpenAI snapshots to reduce instability.The paper notes that fixing temperature and seed does not eliminate non-determinism.
  • Compression: Compression thresholds are benchmark-specific for history and observation contexts.History thresholds are 4096 for AppWorld and OfficeBench and 2048 for 8-objective QA; observation thresholds are 1024, 512, and 400, respectively.
  • Prompt Optimization: Prompt optimization samples five candidate prompts and selects the best-performing candidate on a training subset.The OpenAI o3 model performs both prompt analysis and updates.
  • Prompt Optimization: The optimization procedure uses separate analysis and update prompts for UT and CO steps, with detailed prompts and procedures provided in the appendix.
  • Guideline Optimization: One optimization round consists of a single UT step followed by a single CO step unless otherwise noted.
  • Baselines and Distillation: Baselines use fixed configurations for FIFO, retrieval, LLMLingua, and naive prompting, while distillation uses standardized LoRA settings without hyperparameter tuning.The LoRA setup includes rank 16, α = 32, learning rate 10^-4, three epochs, batch size 4, and maximum sequence length 10,000.

C. Additional Results

Additional quantitative results are provided to complement the paper’s main experiments in Section 4.

  • Additional Results: Additional quantitative results complement the main experiments in Section 4.
  • Additional Results: The additional results are presented separately from the main experiments.
  • Additional Results: Section 4 is supplemented with further quantitative evaluation.

C.1. Results with different agent models

ACON’s reported trends extend across smaller and stronger proprietary agent models. History compression improves gpt-4.1-mini performance, while AppWorld improvements with gpt-5-chat follow the gpt-4.1 trend.

  • gpt-4.1-mini: ACON’s trends remain consistent for gpt-4.1-mini across AppWorld, OfficeBench, and 8-objective QA.The results highlight robustness under resource-constrained settings.
  • gpt-4.1-mini: History compression improves gpt-4.1-mini performance compared with the baseline.This complements the paper’s finding that ACON enhances smaller language-model agents.
  • gpt-5-chat: ACON’s improvements on AppWorld with gpt-5-chat follow the same trend as with gpt-4.1.The paper reports this as evidence that ACON generalizes to stronger proprietary models.

C.2. Detailed results and analyses

ACON extends evaluation across benchmarks, model configurations, compression settings, and qualitative cases. Results show gains in challenging tasks, effective distillation, and practical flexibility, while some combined or extra optimization settings introduce trade-offs.

  • Benchmark analysis: ACON yields the largest OfficeBench gains on the most challenging Level 3 tasks.
  • Distillation: Optimized guidelines consistently produce stronger performance than unoptimized guidelines when distilled into smaller models.
  • Ablations: Combining history and observation compression achieves larger reductions in peak token usage and dependency but substantially degrades performance relative to either compression alone.
  • Ablations: A single optimization round is sufficient for effective guideline learning, because an additional utility maximization step causes a performance drop.
  • Distillation: Distilled observation compressors largely preserve performance, confirming transfer of optimized compressors to smaller models.
  • Generalization: On WebVoyager, ACON improves accuracy over prompting while reducing dependency for observation compression from 0.85 to 0.80.

D. Qualitative Examples

Qualitative examples show how optimized compression preserves task-critical state and structure while shortening context. The examples also expose implementation guidance, optimization stages, and boundaries where compression can remove necessary information.

  • Compressed histories: Optimized history preserves variable states, environment guardrails, and essential information required for future decisions while becoming shorter after compression maximization.
  • Compressed observations: Optimized observation compression preserves JSON structure and the crucial play_music API that naive prompting omits.
  • Compressor distillation: Observation-compressor distillation uses Qwen3-14B, Qwen3-8B, and Phi-4 students distilled from a gpt-4.1 compressor.
  • Guideline optimization: The optimization procedure collects baseline and compressed trajectories, compares failures, updates guidelines, and selects candidates using reward or reward-cost evaluation.
  • Prompt requirements: Compression prompts require preserving authentication values, behavior-affecting defaults, endpoints, and identifiers needed for comparisons, loops, or API calls.
  • History compression: In an OfficeBench case, retaining credentials, access tokens, and API dependencies turns repeated authorization failures into successful file deletion.
Loading 2510.00615v3…