What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

Shuyu Liu

Source-linked AI summary

What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

Shuyu Liu

arXiv:2608.16370v1Aug 17, 2026cs.AI

TL;DR

Task completion may miss the interaction cost of context compression when agents must reacquire dropped execution-relevant state. This paper measures that cost directly and finds retrieval increases broadly even when completion remains statistically unchanged.

Problem
Existing evaluations primarily test whether task completion survives compression, leaving runtime reacquisition cost insufficiently measured.
Method
The study uses controlled interventions in a fixed-horizon deterministic planning environment to vary compression, retention, and tool-call type across models and task regimes.
Results
Retrieval increased in all six model–regime comparisons while 5× compression left completion nonsignificantly changed; GPT-5.5 retrieval rose from 21.0 to 63.9 calls.
Takeaways & Limitations
Compression can impose environment-dependent hidden interaction costs when execution-relevant state must be reacquired, which completion metrics may not expose.
Takeaways & Limitations
The study measures cost with turns and tool calls rather than provider-dependent wall-clock latency or dollar cost.

Abstract

from arXiv · show

Task completion is the standard metric for evaluating context compression, yet it is incomplete: compression can increase an agent's interaction cost by forcing it to reacquire dropped state while leaving completion statistically unchanged. We introduce a controlled runtime measurement protocol for reacquisition cost in a bounded-horizon tool-using agent. The agent acts in a deterministic planning environment under a fixed 24-turn horizon. We vary compression severity, compare a dropping operator with a fact-preserving operator, restore dropped state through controlled oracle interventions, and decompose tool calls into retrieval and execution. We evaluate three models across two task regimes. Retrieval calls increase in all six model-regime comparisons and account for almost all added interaction; five of six remain significant after Holm correction. At the prespecified 5x comparison point, completion changes are not significant in any cell. DeepSeek shows a significant completion drop only at 10x compression. GPT-5.5 is the clearest case: completion changes from 80% to 85% (p = 1.0) while retrieval increases from 21.0 to 63.9 calls (p = .002). Retention interventions further separate state quantity, state type, and content validity. Random selection is comparable to an offline hindsight oracle, while replacing retained D-state with semantically irrelevant content increases retrieval by 57% (p < .001) without a significant completion change. In a second environment, ALFWorld, sliding compression produces no retrieval surge, showing that the reacquisition signature is environment-dependent rather than intrinsic to shortening context. Overall, compression can impose hidden interaction costs when execution-relevant state becomes absent and must be reacquired, while completion alone may not expose those costs.

1 Introduction

Context compression can preserve task completion while increasing the tool calls required to reacquire execution-relevant state that was dropped. This section introduces a controlled protocol showing that reacquisition cost is measurable, decomposable, model- and environment-dependent, and often invisible to completion metrics.

Motivation: Compression is commonly judged by preserved task performance and reduced context cost, but near-lossless retention does not measure the cost of reacquiring absent execution-relevant state.Tool calls provide a direct measure of interaction cost spent on reacquisition within a bounded interaction horizon.
Measurement protocol: The protocol fixes a 24-turn horizon, varies compression severity, spans two task regimes, restores dropped state through oracle interventions, and decomposes tool calls into retrieval and execution.These controls isolate how missing state affects interaction cost rather than evaluating completion alone.
Core result: 80% →85% completion (p = 1.0) coexists with retrieval rising from 21.0 to 63.9 calls (p = .002) for GPT-5.5, demonstrating evaluation non-identifiability.The large interaction-cost change is not detected as a completion change.
Retention interventions: Random retention matches an offline hindsight oracle, whereas semantically irrelevant retained content sharply increases retrieval without necessarily changing completion.These interventions separate state quantity, state type, and content validity.
Boundary conditions: Retrieval responds at 5× compression before completion responds, DeepSeek degrades only at 10×, and ALFWorld shows no retrieval surge, making the cost signal conditional and environment-dependent.Across models and regimes, reacquisition cost increases more consistently than completion loss.

2 Related Work

Prior work evaluates what context compression preserves, recovers, uses, or achieves, but does not diagnose the interaction cost of reacquiring dropped state. This paper addresses that gap by measuring reacquisition under a bounded interaction budget and testing whether completion exposes it.

Category map: Prior work spans compression methods, recoverability, long-context utilization, and aggregate efficiency evaluation.These lines ask what to keep, whether information survives or is used, and which strategy performs best on completion and aggregate cost.
Evaluation metrics for agents, and the insufficiency of completion: Completion-centric evaluation can hide process-level variance because agents may re-query tools or re-fetch already-computed state without changing the final outcome.The related work frames completion as insufficient for measuring the trajectory behind an agent’s result.
Closest evaluation-perspective contrasts: None of the closest evaluation-perspective works measures the interaction budget spent reacquiring state dropped by compression or manipulates state availability causally.The paper’s protocol directly targets both the budget and causal state-availability questions.
Closest evaluation-perspective contrasts: The paper distinguishes reacquisition from attention bottlenecks, representation-level round-trip recovery, latent-state probing, and within-inference KV-cache compression.Its black-box focus is environment state across tool calls, where genuinely absent state can be re-fetched under a bounded horizon.
Contribution: Compression becomes costly when dropped state must be reacquired through additional interaction under a bounded budget, even when completion remains unchanged.This contribution asks whether completion metrics expose the cost of recovering execution-relevant state.

3 Method

The method measures task completion and tool-level interaction cost separately in a deterministic planning environment under a fixed 24-turn horizon. It manipulates context compression, state restoration, and retention while distinguishing retrieval from execution to expose reacquisition costs.

Measurement protocol: The protocol compares full or fact-preserving contexts with compressed sliding contexts, separating missing-state retrieval from execution within a fixed 24-turn horizon.An oracle branch injects dropped D* or R* state, making recoverability a controlled intervention rather than a passive observation.
Outcome measures: The evaluation records Q for task completion, CR for retrieval calls, and CE for execution calls, with C = CR + CE as tool-level interaction cost.CR represents state reacquisition effort, whereas CE represents task work; C is not a wall-clock or monetary-cost claim.
Environment and agent: IRBench is a deterministic project-planning environment with 10 tasks, capacity-one resources, hidden execution constraints, and four tools divided into retrieval and execution.Retrieval tools obtain environment state, while execute performs tasks and reveals constraints, enabling direct measurement of reacquisition cost.
Compression operators: Sliding deletes dropped state, whereas extractive Summary preserves observed state facts under the same token budget, isolating absent-state reacquisition from summarizer quality.The extractive summary discards trajectory reasoning text while retaining represented state facts, making the operator contrast a designed control.
Experimental design: Each condition uses 10 seeds and 100 task-level observations, with retention interventions using 20 paired seeds and per-seed paired Wilcoxon tests plus bootstrap 95% confidence intervals.The primary comparisons are prespecified and use the same seed under both conditions.

4 Experiments · 4.1 Setup · 4.2 Compression raises reacquisition cost before completion may degrade (Figure 2)

The experiments measure completion and tool-use costs within a fixed 24-turn horizon, decomposing tools into retrieval and execution calls. Across compression sweeps, retrieval-heavy interaction costs rise before completion reliably declines, with degradation emerging only under the most aggressive settings.

4.1 Setup: Completion is the fraction of ten required tasks finished within 24 turns, while tools are per-run calls decomposed into retrieval and execution.Each condition uses 10 seeds with per-seed paired Wilcoxon tests and bootstrap 95% confidence intervals; primary comparisons are Full versus Sliding.
4.2 Compression raises reacquisition cost before completion may degrade (Figure 2): The sweep varies sliding-window compression ratios r ∈{1, 1.7, 2.5, 5, 10} for DeepSeek in the High-IR regime.Figure 2 reports completion against compression ratio and decomposes tool calls into retrieval and execution.
4.2 Compression raises reacquisition cost before completion may degrade (Figure 2): Completion remains flat through moderate compression, then declines by 11 percentage points at 5× and 17 percentage points at 10×.The 5× change is not significant (p = .25), whereas the 10× change is significant (p = .016) and is the study’s only significant completion change.
4.2 Compression raises reacquisition cost before completion may degrade (Figure 2): At Low-IR, completion stays at 100% through 5× and reaches 97% at 10×, while tools increase from 32.7 to 59.7 calls (+83%).The increase is retrieval-dominated, and the public graph makes Low-IR state re-derivable within the evaluated horizon.
4.2 Compression raises reacquisition cost before completion may degrade (Figure 2): The retrieval-heavy tool increase can remain small enough to preserve completion within the fixed horizon when state is re-derivable from the public graph.This explains why interaction cost can rise without an immediate completion penalty in Low-IR.
4.2 Compression raises reacquisition cost before completion may degrade (Figure 2): Sliding completion is 72% versus 83% for Summary, exceeding the few-point batch spread despite Full-context completion varying from 80% to 86% across batches.Sliding is the 5× canonical-sweep condition, whereas Full and Summary come from a separate operator-contrast batch.

4.3 What is retained matters more than the ratio (operator contrast) · 4.4 Restoring dropped state reduces reacquisition (oracle intervention)

At the same 5× compression ratio, extractive summaries preserve execution-relevant state and remain near-lossless, whereas sliding compression increases retrieval and harms completion. Restoring dropped task state reverses most retrieval overhead and recovers most of the completion gap, while history-only restoration leaves retrieval costly.

4.3 What is retained matters more than the ratio (operator contrast): The operator contrast indicates that retained content can matter more than compression ratio: preserving observed state facts avoids the sliding operator’s retrieval surge.This reproduces the finding that compression need not hurt when execution-relevant state survives.
4.3 What is retained matters more than the ratio (operator contrast): At identical 5× compression, extractive summary yields 83% completion versus 80% full and 37.4 →39.0 tools, while sliding triples retrieval and degrades completion.The summary preserves observed state facts but discards trajectory reasoning text; sliding drops both.
4.4 Restoring dropped state reduces reacquisition (oracle intervention): R∗ restoration improves completion by +12pp but leaves retrieval at 69.0, because history-only constraints and fail-counts do not replace missing D.R∗ helps completion without removing the defensive re-querying caused by absent execution-relevant state.
4.4 Restoring dropped state reduces reacquisition (oracle intervention): 72.9 →35.8 tools after restoring D∗ removes most compensatory retrieval, while completion recovers from 66% →80%.D∗ is the queryable task graph, representing execution-relevant state dropped by sliding compression.
4.4 Restoring dropped state reduces reacquisition (oracle intervention): The full + R∗ sanity condition is harmless, with 85% completion versus 86% full, confirming that oracle injection does not distort the baseline.The intervention therefore isolates the effect of restoring dropped state rather than introducing a baseline artifact.
4.4 Restoring dropped state reduces reacquisition (oracle intervention): Together, the intervention results provide causal evidence that removing execution-relevant state drives reacquisition, whereas restoring that state substantially reduces the interaction cost.The comparison separates state type from merely restoring history-only information.

4.5 The cost generalizes; the outcome does not (three models)

Across six model–regime comparisons, compression consistently increased retrieval interactions, but completion outcomes remained statistically unchanged and varied in direction. GPT-5.5 High most clearly separated reacquisition cost from completion, with retrieval roughly tripling while completion did not change detectably.

Cross-model comparison: At the pre-specified 5× comparison point, five of six retrieval increases remained significant after Holm–Bonferroni correction, while execution calls stayed approximately stable in five comparisons.The six-cell matrix used a single 5× compression ratio fixed before the severity sweep.
Cross-model comparison: Retrieval calls increased in all six model–regime comparisons, establishing a consistent reacquisition pattern across the evaluated models.The reported pattern did not imply uniform completion degradation.
Outcome heterogeneity: Completion changes were not significant in any cell (all p ≥0.125), whereas cells occupied distinct combinations of retrieval growth and completion change.DeepSeek High combined high retrieval growth with completion loss; GPT-5.5 High and both Low cells had stable completion; Qwen showed little retrieval growth with a mild completion drop.
GPT-5.5 High: GPT-5.5 High showed no detected completion change (p = 1.0) while retrieval roughly tripled (+42.9, p = .002), with added calls almost entirely retrieval.The completion confidence interval was [−6, +21] pp, and execution remained approximately flat.

4.6 Retention interventions reveal two distinct levels of state dependence (Figure 4)

Retention interventions distinguish state selection from content validity: fine-grained selection among real, task-relevant atoms has little effect, whereas semantically irrelevant retained content can sharply increase retrieval at looser budgets without changing completion. The content effect depends on budget and partially replicates across models.

Selection versus content validity: −22.1% vs. −22.4%: Random selection matches the offline hindsight oracle on retrieval cost, with no reliable advantage for TypeAware over Random or Recent.Random’s paired difference from Hindsight is +0.10 percentage points, with 95% CI [−2.3, +2.3] and d ≈0.02; all TypeAware pairwise comparisons are nonsignificant.
Selection versus content validity: +18%, p = 0.002: At B = 265, replacing retained D-state with semantically irrelevant content increased retrieval in all 20 seeds while leaving execute unchanged.The retrieval increase ranged from +3 to +23 calls and exceeded even the no-digest sliding baseline.
Budget dependence: ∆+0.6, p = 0.60: At B = 100, real versus irrelevant D-content was behaviorally null, whereas at B = 265 the ∼209-token digest became load-bearing.At the tighter budget, the agent relied more on recent turns; at the looser budget, valid content reduced retrieval relative to irrelevant content while completion remained statistically unchanged.
Cross-model replication: −9.5, p = 0.002: GPT-5.5’s real-D digest reduced retrieval relative to sliding at B = 265, replicating DeepSeek’s direction, but irrelevant injection did not exceed the raw sliding baseline.At B = 100, GPT-5.5 showed no detectable additional content effect: irrelevant 26.0 vs. real 28.8, p = 0.125.

4.7 External environment boundary: ALFWorld

In ALFWorld, sliding compression produced no retrieval surge, unlike IRBench, indicating that the reacquisition signature depends on the environment’s recoverability structure rather than context shortening alone. Relevant state was generally directly re-observable through available interaction actions.

Method: 18 tasks spanning six ALFWorld task types were run with five paired seeds per task, yielding 180 runs with zero API errors.Retrieval-like, execution-like, and navigation actions were counted separately.
Results: −0.13 paired ∆(retrieval-like) under sliding compression, with a 39/35 seed split symmetric around zero, showed no retrieval surge.∆tools was approximately zero.
Results: 10.0% full versus 8.9% sliding completion showed no difference, with 89/90 paired runs identical.Completion was dominated by the difficulty profile of the sampled tasks.
Interpretation: The reacquisition signature is environment-dependent: IRBench requires re-querying dropped task-graph state, whereas ALFWorld generally permits direct re-observation.The account attributes compression cost to unavailable execution-relevant state and its recoverability.

4.8 The interaction budget: turn horizon and tool budget

The 24-turn horizon, not the auxiliary tool budget, determines termination, and it binds in every High condition. Compression instead reduces progress within that horizon and adds enough reacquisition overhead to exceed the full-context reference budget.

Turn horizon and tool budget: The 24-turn horizon is binding in every High condition, including Full, with 7/10 Full runs and 8–9/10 compressed runs terminating at max turns.The auxiliary α× reference budget (α = 2.0) is diagnostic rather than a termination criterion.
Turn horizon and tool budget: Completion falls from 83% to 66% at 10× compression because less task progress fits within the same horizon.Compression does not introduce turn pressure; it reduces progress within the fixed horizon.
Turn horizon and tool budget: 10/10 Full runs stay within the α · 2.0 tool budget, whereas 0/10 Sliding 5× and 10× runs do.The reacquisition overhead exceeds a reference execution budget that the full-context agent never approaches.
Turn horizon and tool budget: Termination reasons are recorded per run, and every reported condition has 10/10 completed runs for auditing turn exhaustion separately from API errors.Runs are classified as all complete or max turns.

5 Discussion

Context compression can substantially increase state-reacquisition interaction costs without changing task completion, because absent execution-relevant state must be recovered through additional retrieval. The cost depends on what state is retained and how the environment permits its recovery, so completion and interaction overhead should be evaluated jointly.

Interaction cost and completion: Retrieval increases were significant in five of six cells after Holm correction, while completion changes were significant in none.The protocol separates retrieval, reflecting state reacquisition, from execution, reflecting task work.
Recoverability mechanism: 72.9 → 35.8 tool calls (−51%; paired p = .002) after restoring D∗, while restoring R∗ changed retrieval 72.9 → 69.0 (−5%; n.s.).Restoring D∗ also recovered completion from 66% → 80%; R∗ improved completion by +12pp.
Retention interventions: 57% higher retrieval followed replacement of real D content with semantically irrelevant content, whereas random selection matched an offline hindsight oracle.Fine-grained selection among valid D atoms had little measurable marginal effect, but valid task-relevant content mattered.
Practical implications: Evaluators should fix the agent, task, and horizon; jointly record completion and interaction cost; decompose retrieval from execution; and restore dropped state when causal attribution is needed.A strategy that preserves completion but substantially increases reacquisition cost should not be treated as equivalent to baseline solely on completion metrics.
Environment and model boundaries: In ALFWorld, sliding compression produced no retrieval surge, showing that compression cost depends on which execution-relevant state becomes unavailable and how it can be reacquired.The reacquisition pattern was consistent across the three studied models, but its conversion into reduced completion was not.

6 Limitations

The study’s conclusions are bounded by synthetic-adjacent environments, limited models, a single primary compression point, and two operators. Additional limitations concern cost measurement, tool-level attribution, statistical power, and controls whose generality remains untested.

Scope of environments: The core mechanism is measured in two synthetic-adjacent environments, and the reacquisition signature disappears where relevant state can be directly re-observed.The study makes no claim that specific cost magnitudes transfer to open-ended tasks; future work should test partially observable, long-horizon environments.
Scope of models: Three model families are evaluated, so cross-model comparisons are descriptive and absolute tool or token counts are not interpreted across providers.Serving conditions are not fully comparable, though serving details are recorded in the reproducibility appendix.
Experimental design: The fixed 24-turn horizon and single 5× cross-model compression point limit conclusions about when interaction costs convert into failures across horizons and severities.Only DeepSeek receives a full severity sweep, including the 10× point where completion first becomes significant.
Measurement and attribution: Tool-call count measures budgeted interaction currency rather than provider-dependent wall-clock latency or dollar cost, while retrieval/execution attribution remains tool-level.Per-message intentions and turn-level retrieval-burst timing were not analyzed because logs were aggregated to condition-level totals.
Control generality: The controls cover only a designed extractive summary, two operators, one fabrication-based irrelevance manipulation, and partially replicated selection and content interventions.Their specific magnitudes and cross-model generality therefore remain unestablished, and fabricated entities may induce verification as well as semantic-irrelevance effects.
Statistical power: 10 seeds limit statistical power, but the nonsignificant completion changes at 5× coexist with retrieval significance in five of six cells after Holm correction.The study treats the tool-level effect as primary and reports effect sizes and confidence intervals for all comparisons.

A Seed-level association

At 5× compression, seed-level retrieval increases and completion changes were examined within six model–regime cells. High-regime associations were negative, but the exploratory results are limited by small per-cell samples and multiple comparisons.

Method: At 5×, per-seed retrieval-call changes and completion changes were paired within each of six model–regime cells using Spearman correlations.The measures were ΔCR = CR(Sliding) − CR(Full) and completion change ΔQ in percentage points.
Results: Negative associations occurred in all three High-regime cells, with GPT-5.5 High reaching ρ = −0.64, p = .045.The bootstrap confidence interval excluded zero; larger retrieval increases tended to coincide with larger completion decreases.
Results: The two Low-regime cells were undefined because completion was invariant across seeds in both compression conditions.Every seed completed all tasks in both conditions.
Limitations: These associations are exploratory and descriptive rather than causal, based on n = 10 per cell and multiple comparisons.The single significant cell should therefore be interpreted cautiously, and cells were not pooled because baseline difficulty and model response differed.