ACON: Optimizing Context Compression for Long-horizon LLM Agents

Problem

• Long-horizon agents face prohibitive inference memory costs and degraded decision quality because growing contexts require massive KV caches and dilute relevant information.Losing details such as file paths or API parameters can derail workflows.Source: §1 · p. 1
• Productivity agents must preserve factual history, action-outcome relationships, success preconditions, and future decision cues across heterogeneous tools, which token truncation and generic summarization can discard.Effective compression must balance aggressive reduction with precise retention of task-relevant state information.Source: §1 · pp. 1–2

Limitations

• Existing methods are limited because heuristics are brittle and specialized, while model-optimization approaches entangle compression with agent weights and cannot directly update proprietary API-based LLMs.Dialogue-oriented and document-centric methods also emphasize conversational coherence or single-step reasoning rather than persistent state tracking.Source: §1 · p. 2

Method

• ACON jointly compresses environment observations and interaction histories by refining natural-language compressor guidelines through failure analysis, without altering the agent model weights.This makes the framework model-agnostic and applicable to proprietary, API-based LLMs.Source: §1 · pp. 2–3

Advantages

• The optimized guidelines derive environment-specific rules across tasks, preserve essential information to reduce distraction, and can be distilled into smaller models for cost-efficient compression.This enables smaller models to act more effectively while reducing the compression module’s computational overhead.Source: §1 · pp. 2–3

Empirical validation

• ACON lowers peak token usage by 26–54% while improving task success, preserves 95% of the teacher compressor’s accuracy after distillation, and improves small-LM performance by 32% on AppWorld, 20% on OfficeBench, and 46% on Multi-objective QA.These results were obtained on AppWorld, OfficeBench, and Multi-objective QA, each requiring 15+ interaction steps.Source: §1 · pp. 2–3

Long-horizon LLM agents

• Long-horizon agents iteratively observe environments, use tools, execute actions, and revise plans across dozens to hundreds of steps, creating dynamic context-management challenges.They must retain multi-step histories and process heterogeneous observations as relevance shifts during execution.Source: §2 · p. 3

Context compression for LLMs

• Prior LLM context-compression methods include document or retrieval compression, dialogue-memory summarization, and low-level KV-cache compression, but remain insufficient for shifting long-horizon agent contexts.These approaches have shown benefits in their respective settings but do not fully address dynamic, heterogeneous relevance.Source: §2 · p. 3

Context compression for LLM agents

• Agent-specific compression methods rely on naive prompting, narrow domains, or reinforcement-learning-based model updates that couple compression with policy reasoning and require internal-weight access.ReSum instead optimizes the policy model to use summaries more effectively.Source: §2 · p. 3
• ACON provides universal, model-agnostic compression for both agent histories and observations across open-source and proprietary API-based LLMs.Its optimization process is designed to generalize across arbitrary LLMs.Source: §2 · p. 3

3. Agent Context Optimization (ACON)

• ACON jointly compresses interaction history and latest observations to reduce dynamic context cost while preserving information needed for long-horizon task success.The framework treats history and observations as complementary compression targets and defines cost over the dynamic context while keeping the agent and system prompt fixed.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 4

3.2. History & Observation Compression with LLMs

• History compression activates only above Thist, while observation compression activates only above Tobs, avoiding unnecessary compressor overhead for short inputs.Compressed history replaces raw history, and compressed observations replace raw observations in the agent’s stored interaction history.Source: §3.2 · p. 4

3.3. Optimizing Compression Guidelines

• ACON optimizes natural-language guidelines instead of compressor parameters, using failures under compression versus successful uncompressed trajectories as richer feedback than scalar rewards.This avoids the computational cost, expensive rollouts, and high-variance estimates associated with reinforcement-learning updates to large language models.Source: §3.3 · p. 5, §3.2 · p. 5
• The guideline optimizer alternates utility maximization for task reward with compression maximization, refining prompts toward shorter contexts that remain sufficient for successful execution.The second stage uses successful compressed-context tasks to identify information actually used during execution and reduce context cost.Source: §3.3 · p. 5

3.4. Distilling Context Compression into Small Models

• ACON distills the optimized teacher compressor into a smaller student trained on teacher-generated outputs, replacing the teacher at inference to reduce compression overhead.The two-stage pipeline decouples decision making from compression, with the student trained by cross-entropy on input-output pairs from successful teacher-compressed tasks.Source: §3.4 · pp. 5–6
• ACON consistently improves AppWorld accuracy while reducing peak input tokens and dependency, with ACON UTCO achieving the best overall performance across difficulty levels.Table 1 reports accuracy, steps, peak input tokens, and dependency on the test-normal AppWorld benchmark.Source: Table 1 · p. 6

4.1. Experimental Setup

• ACON is evaluated with ReAct on long-horizon benchmarks requiring 10+ interaction steps: AppWorld, OfficeBench, and 8-objective QA, using task performance, Steps, Peak Tokens, and Dependency metrics.AppWorld uses task completion score, OfficeBench uses benchmark-defined accuracy functions, and 8-objective QA reports average EM and F1 across eight questions.Source: §4.1 · p. 6

4.2. Overall performance and token efficiency

• 54.5% and 61.5%: On 8-objective QA, ACON reduces peak tokens and dependency while surpassing no compression in EM/F1; it also preserves accuracy with over 25% fewer peak tokens on AppWorld and nearly 30% fewer on OfficeBench.On OfficeBench, accuracy remains above 74%. For observation compression, ACON consistently outperforms all baselines.Source: §4.2 · p. 7
• UT improves performance while reducing token cost across benchmarks, whereas UTCO further lowers cost but can slightly reduce accuracy; UTCO suits redundant observations, while UT better preserves precise facts.The experiments apply one optimization step per interaction step; verbose AppWorld observations favor aggressive pruning, whereas OfficeBench and 8-objective QA favor fidelity.Source: §4.2 · pp. 7–8, §4.1 · p. 7

4.3. Compressor distillation

• Over 95%: LoRA-distilled Qwen3-14B, Qwen3-8B, and Phi-4 compressors retain the gpt-4.1 teacher’s performance while reducing computational overhead.gpt-4.1-mini also serves as an effective compressor without distillation on OfficeBench and QA, allowing the expensive model to be reserved for the agent.Source: Figure 4 · p. 7, §4.3 · p. 8

4.4. ACON for distilled small agents

• 45.6%: With ACON, Qwen3-14B improves from 0.158 to 0.23 EM on 8-objective QA, while AppWorld performance improves 32.4%, from 25.6% to 33.9%.Compression mitigates distracting long contexts, enabling smaller agents to approach larger-model performance.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5. Analysis

• Moderate thresholds—4096 for history and 1024 for observations—best balance accuracy, peak-token reduction, and compression frequency.The o3 optimizer with contrastive feedback performs best, guideline optimization costs less than $2 per benchmark, and distilled Qwen3-14B reduces compressor cost to $0.0004, or 99.1% below gpt-4.1.Source: §4.5 · pp. 8–9
• ACON reduces end-to-end API cost but adds wall-clock latency because compression requires an additional compressor call.Latency is measured locally on one A100 GPU, while API cost estimates exclude input caching.Source: §4.5 · p. 9, Table 4 · p. 9

Conclusion

• ACON reduces peak tokens by 26–54% while improving task success over existing compression baselines, with small degradation relative to full-context baseline.Experiments cover AppWorld, OfficeBench, and Multi-objective QA.Source: §5 · p. 9
• ACON systematically compresses interaction histories and environment observations through adaptive, model-agnostic guideline optimization in natural language space.This distinguishes ACON from approaches based on naive prompting or narrow domains.Source: §5 · p. 9
• Optimized compressors can be distilled into smaller models, lowering overhead without sacrificing performance and helping Qwen3-14B approach much larger models.Concise, informative contexts support smaller agents in long-horizon tasks.Source: §5 · p. 9

Limitations and Future Work

• Evaluation primarily focuses on GPT models due to resource constraints, limiting the demonstrated empirical scope.The authors identify broader evaluation as future work.Source: §5 · p. 9
• Compression reduces context costs but introduces computational overhead and increased latency.This limitation is shared by many context management frameworks.Source: §5 · p. 9

Impact Statement

• Context compression and distillation into smaller models reduce the computational footprint of AI systems and lower barriers to entry for resource-constrained researchers.The authors foresee no specific negative societal consequences beyond general considerations for deploying autonomous systems.Source: Impact Statement · p. 10

Overview

• The framework reduces token costs with minimal performance degradation, but its limitations motivate several future research directions.These include empirical scope, deployability, optimization convergence, distillation, and computational overhead.Source: A. Limitations & Future Works · p. 16

Scope of empirical evaluation

• Empirical generalizability beyond GPT models remains insufficiently verified, especially for Gemini, Claude, DeepSeek-R1, and Qwen3-235B.GPU availability and computational constraints restricted evaluation; broader testing would strengthen robustness and applicability.Source: A. Limitations & Future Works · p. 16

Real-world deployability

• Benchmark validation remains controlled, leaving in-the-wild deployment and integration into live multi-agent production systems open.Future environments may involve stochastic task complexity and less predictable constraints.Source: A. Limitations & Future Works · p. 16

Convergence of natural language space optimization

• Natural-language optimization lacks a formal convergence guarantee despite sampling multiple candidates and selecting the best performer.Future work should analyze this process more deeply and develop principled methods for optimizing the objective in 3.3.Source: A. Limitations & Future Works · p. 16

Distillation gap and data scalability

• A performance gap remains between distilled and teacher models, potentially because 100 examples per domain insufficiently capture complete behavioral cloning.Increasing training-data diversity and volume may reduce or eliminate the residual gap.Source: A. Limitations & Future Works · p. 16

Latency and KV-cache dynamics

• Compression can increase latency because history compression invalidates the KV-cache and requires recomputing the compressed sequence.Observation compression reduces context before caching, but compression generation still adds overhead; dynamic KV-cache eviction or compression is a proposed direction.Source: A. Limitations & Future Works · p. 16

B.1. Datasets

• AppWorld integrates nine everyday applications through 457 APIs and evaluates multi-step productivity tasks averaging 42.5 API calls.It includes approximately 100 simulated users and uses 90 training tasks for optimization and distillation plus 168 test-normal tasks.Source: §B.1 · p. 16
• Table 5 provides an example task from each benchmark to illustrate the evaluated task formats.The table covers example tasks across AppWorld, OfficeBench, and 8-objective QA.Source: Table 5 · p. 17, §B.1 · p. 17
• OfficeBench tests cross-application office automation across 1-app, 2-app, and 3-app tasks using Word, Excel, PDF, Calendar, Email, Shell, and Calculator.Experiments restrict evaluation to text-related tasks, excluding OCR-dependent tasks to avoid confounding from OCR quality.Source: §B.1 · p. 17
• OfficeBench is randomly split into 92 training and 95 test tasks, with ambiguous tasks removed and synthetic testbeds kept separate across splits.No official split is available, so the partition uses a 1:1 ratio before these refinements.Source: §B.1 · p. 17
• The 8-objective QA benchmark groups eight NaturalQuestions questions into each deep-research task, requiring answers to all questions at the end.This creates a more challenging long-horizon setting than conventional multi-hop QA; the study uses 100 training and 100 test tasks with BM25 retrieval over 2018 Wikipedia.Source: §B.1 · p. 17
• The 8-objective QA retrieval setup follows Jin et al. (2025) by applying BM25 to the 2018 Wikipedia knowledge base.Tasks are constructed following Zhou et al. (2025), with questions drawn from NaturalQuestions.Source: §B.1 · p. 17

• The efficiency evaluation adopts peak tokens and dependency, metrics introduced in LightThinker (Zhang et al., 2025) and MEM1 (Zhou et al., 2025).Source: §B.2 · p. 17
• Peak tokens measures the maximum tokens in any trajectory sequence, excluding system prompts, as a proxy for inference-time memory requirements.It corresponds to the maximum peak shown in Figure 2.Source: §B.2 · p. 17
• Dependency is defined as the area under the Figure 2 curve and approximates cumulative computational cost for action generation.Source: §B.2 · p. 18
• API-cost analysis uses September 2025 OpenAI pricing for gpt-4.1 and gpt-4.1-mini, and OpenRouter-based estimates for Qwen3-14B (Yang et al., 2025).gpt-4.1 costs $3.00 per 1M input tokens and $12.00 per 1M output tokens; gpt-4.1-mini costs $0.80 and $3.20, while Qwen3-14B is estimated at $0.06 and $0.24.Source: §B.2 · p. 18

API Inference

• API inference uses temperature 0.0, seed 42, and Azure OpenAI snapshot endpoints to reduce residual nondeterminism.The snapshots are gpt-4.1-2025-04-14 and gpt-4.1-mini-2025-04-14.Source: §B.3 · p. 18

Compression

• Compression retains the last action–observation pair, with history budgets of 4096, 4096, and 2048 and observation budgets of 1024, 512, and 400 for AppWorld, OfficeBench, and 8-objective QA, respectively.These settings apply to ACON and all baselines.Source: §B.3 · p. 18

Prompt Optimization

• Prompt optimization uses OpenAI o3 for analysis and updates, samples five candidate prompts, and selects the best on a training subset.One experimental round contains one UT step and one CO step; subsets are chosen from compression-failure or compression-success tasks as specified.Source: §B.3 · p. 18

Compressor & Agent Distillation

• Compressor and agent distillation share LoRA rank 16, α = 32, learning rate 10^-4, three epochs, batch size 4, and maximum sequence length 10,000 without hyperparameter tuning.Training uses linear warmup, weight decay 0.01, AdamW, one teacher sample, and one A100 80GB GPU; fine-tuned inference uses greedy decoding.Source: §B.3 · p. 18

• Additional quantitative results complement the main experiments in Section 4.Source: C. Additional Results · p. 19

Experiments with gpt-4.1-mini

• History compression improves gpt-4.1-mini’s performance over the baseline across AppWorld, OfficeBench, and 8-objective QA, supporting ACON’s robustness under resource constraints.Results are reported in Tables 9–11 and complement findings that ACON enhances smaller language-model agents.Source: §C.1 · p. 19

Experiments with gpt-5-chat

• On AppWorld, ACON shows the same improvement trend with gpt-5-chat as with gpt-4.1, indicating generalization to stronger proprietary models.The gpt-5-chat results are reported in Table 12.Source: §C.1 · p. 19

C.2. Detailed results and analyses

• ACON yields its largest OfficeBench gains on the most challenging Level 3 tasks.The difficulty breakdown uses gpt-4.1 and is reported in Table 8.Source: §C.2 · p. 19
• Optimized guidelines consistently outperform unoptimized guidelines when distilled into smaller AppWorld compressor models.Additional distilled-optimizer results are reported in Table 13.Source: §C.2 · p. 19
• Combining history and observation compression reduces peak tokens and dependency more, but substantially degrades performance compared with either compression alone.The joint-compression ablation uses gpt-4.1 and is reported in Table 14.Source: §C.2 · p. 19
• An extra utility-maximization step after the standard UT–CO sequence reduces performance, indicating that one optimization round suffices.The additional-iteration ablation is reported in Table 14.Source: §C.2 · p. 19
• Distillation largely preserves observation-compression performance, demonstrating transfer of optimized observation compressors to smaller models.These results are reported in Figure 7, paralleling the history-compression analysis.Source: §C.2 · p. 19
• ACON improves WebVoyager accuracy over no compression and prompting while reducing peak tokens and dependency versus no compression.For history compression, accuracy rises from 42.9 to 48.6 over prompting; observation compression raises accuracy from 45.7 to 47.1 and lowers dependency from 0.85 to 0.80.Source: §C.2 · p. 20, Table 6 · p. 20
• History compression converts a gpt-4.1-mini failure into successful file removal by retaining authentication dependencies and eliminating redundant exploration.The compressed context preserves credentials, access-token propagation, and the correct API sequence, enabling removal of all .pdf files in /downloads.Source: §C.2 · p. 20

C.3. Comparison with MEM1

• Unlike MEM1 (Zhou et al., 2025) and concurrent works (Ye et al., 2025), ACON optimizes at the prompt level without weight updates and decouples agent and compressor models.This supports proprietary or open-source API models and pairings such as gpt-4.1 agents with distilled Qwen3-14B compressors, which MEM1-like unified training cannot support.Source: §C.3 · pp. 20–21, Table 7 · p. 20

Compressed histories

• Optimized history guidelines retain task progress, variable states, and environmental guardrails before CO shortens histories while preserving essential decision-making information.This illustrates improved compression efficiency and effectiveness in AppWorld with gpt-4.1.Source: D. Qualitative Examples · p. 21
• Guideline optimization consistently improves compressed-history quality across AppWorld, 8-objective QA, and OfficeBench examples.Compression Examples D.2 and D.3 extend the qualitative evidence beyond AppWorld.Source: D. Qualitative Examples · p. 21

Compressed observations

• Optimized observation compression preserves both JSON structure and the crucial “play_music” API that naive prompting omits in a lengthy Spotify-app observation.The structured representation retains functionality necessary to complete the task.Source: D. Qualitative Examples · p. 21

History compression after optimization (UT)

• The optimized history prompt preserves progress, rationale, runtime variables, pending and completed actions, and concise guardrails in an exact five-part HISTORY_SUMMARY structure.It requires retaining essential facts, parameters, artifacts, and errors that help avoid future mistakes while returning no commentary.Source: D. Qualitative Examples · p. 31

History compression after optimization (UTCO)

• The UTCO history prompt targets summaries well under 1500 characters by collapsing repetition, pruning unused variables and logs, and preserving factual continuity and evidence-driven essentials.It retains items such as tokens, IDs, emails, amounts, lists, paths, descriptions, and task status, while outputting bounded REASONING, VARS, TODO, COMPLETED, and GUARDRAILS sections.Source: D. Qualitative Examples · pp. 31–32

Observation compression after optimization (UT)

• The UT observation prompt keeps only the minimal sufficient information for the next step, including callable endpoints, full parameters and defaults, required schema fields, and raw rows needed for comparisons or loops.It explicitly forbids omitting behavior-affecting defaults, declaring parameters noncritical without proof, hallucinating fields, or replacing machine-readable data with vague prose.Source: D. Qualitative Examples · p. 33

Observation compression after optimization (UTCO)

• The UTCO observation prompt further minimizes current observations by retaining authentication values, behavior-affecting defaults, endpoint names, and identifiers needed for comparisons, loops, or API calls.It deletes unused fields and narrative text, compresses structured data, summarizes uniform lists when safe, and excludes history, prior summaries, and meta-comments without altering required literals.Source: D. Qualitative Examples · p. 34

AppWorld trajectory example

• The AppWorld Spotify trajectory illustrates preserving API-relevant state: the agent identifies show_recommendations, then retains its endpoint, authentication requirement, and pagination parameters.The example task asks for the artist least recommended to the user on Spotify.Source: D. Qualitative Examples · p. 35

Qualitative Spotify trajectory

• A separate Spotify trajectory retrieves an access token, obtains personalized recommendations, counts artist appearances, and submits Evelyn Rose as the least-recommended artist.The trajectory uses the login specification and supervisor credentials before requesting recommendations and completing the task.Source: D. Qualitative Examples · pp. 35–36

Example D.3: AppWorld trajectory example - gpt-4.1-mini agent successes with history compression

• With history compression, the agent preserves the task, authentication failures, obtained token, missing token-use mechanism, and guardrails against redundant unauthorized calls.The summary explicitly records successful login, repeated failures without token usage, and the need to pass tokens explicitly while retaining credentials securely.Source: D. Qualitative Examples · p. 42
• The compressed trajectory identifies that show_directory requires an access_token parameter and uses it to list files in ~/downloads, including multiple .pdf files.The resulting directory listing contains paths such as bakeware.pdf, bowls_for_snacks.pdf, and travel_insurance.pdf.Source: D. Qualitative Examples · p. 43

Compression D.1: AppWorld History Compression Example

• ACON’s AppWorld histories retain the relevant APIs, credentials, dates, message identifiers, sender information, completed actions, TODOs, and guardrails needed to continue Splitwise-invitation processing.The utility-step example records the phone-app search state and next contact-check actions, while the utility-plus-compression example preserves a shorter voice-message state with the contact-only constraint.Source: D. Qualitative Examples · pp. 45–46

Compression D.2: 8-objective QA History Compression Example

• In the 8-objective QA example, ACON records confirmed answers such as 1955 for the last Dodgers–Yankees World Series and Paris, 2024 for the next Olympics, while marking unresolved questions as pending or blocked after failed searches.The compressed history also applies a loop guard after three failed Olympics-location searches rather than repeating the same query.Source: D. Qualitative Examples · pp. 47–48

Compression D.3: OfficeBench History Compression Example

• The OfficeBench example preserves the source workbook, top revenue years 2014 ($4,983,791) and 2015 ($4,611,210), output-file state, and remaining spreadsheet and report tasks.The compressed state records that headers and the first year were written while the revenue cells, second row, and reports.docx remained pending.Source: D. Qualitative Examples · pp. 48–49

• The agent located company_revenues.xlsx, created top2.xlsx, and wrote headers plus the 2014 year.Source: D. Qualitative Examples · p. 50

Compression D.4: AppWorld Observation Compression Example

• The AppWorld example presents a 15-minute offline-drive request to play an already downloaded Spotify album without repetition.Source: D. Qualitative Examples · p. 50

• The prompting baseline compresses Spotify observations into three API capabilities: album library, downloaded songs, and specific-album details.Source: D. Qualitative Examples · p. 56
• ACON’s utility step retains those Spotify capabilities and adds a constrained music-playback operation specifying allowable inputs and queue behavior.The playback description permits at most one of queue_position, song_id, album_id, or playlist_id, with distinct behavior depending on which is provided.Source: D. Qualitative Examples · p. 56
• ACON’s utility-plus-compression history summary reduces the record to four terse entries, including album-library, downloaded-song, private-album, and album-based playback actions.Source: D. Qualitative Examples · pp. 56–57