Compact-Bench: Measuring What Context Compaction Costs Long-Horizon Coding Agents

1 Introduction

• Across 14 backbones, stable Task Preservation damage ranges from 5 −64% of 22 validated full-history successes, with Claude Opus 4.8 preserving the most repairs.The introduction also reports that 120 of 126 first-round damaged instance-backbone pairs met the three-run confirmation criterion.Source: §1 · p. 2
• Compact-Bench combines controlled task continuations with recurring-detail and production-chain measurements to evaluate whether compaction preserves an agent’s ability to resume work.Its three evaluations compare full-history and summary-only continuations from the same repository snapshot and measure recurring details in sessions and across production compactions.Source: §1 · pp. 1–2
• With Kimi-K3 as summarizer, Kimi Code preserves the most task continuations (19/22), while Claude Code retains the most recurring details (15.0%).The task-preservation and detail-retention leaders differ.Source: §1 · p. 2
• Paired traces show that summaries can retain target files and relevant code strings while losing the progress needed to finish a repair.This motivates evaluating both retained information and subsequent agent behavior.Source: §1 · p. 2

2 Related Work

• Compact-Bench pairs fixed-boundary replay with backbone and prompt comparisons, recurring-detail measurements, and repeated production boundaries to link a specific history replacement to task outcomes.The related-work discussion contrasts this design with evaluations of constraints, archived-conversation retrieval, trajectory support, restored-environment execution, and observation masking.Source: §2 · p. 2
• Persistent-memory work maintains or retrieves information outside the active history, while Compact-Bench measures structured information that reappears after compaction in coding-agent workflows.The related methods include trajectory summarization, completed-subtask condensation, joint execution and summarization training, agent-selected compaction timing, and persistent-memory systems.Source: §2 · p. 2
• Prompt-compression methods reduce input length through selection, soft representations, or offline compression, while summary evaluations assess consistency, answerability, and generation quality.Compact-Bench extends the evaluation target to resuming agent work through executable outcomes and deterministic matching of recurring details.Source: §2 · p. 2

3.1 Evaluation Dimensions

• PRE and POST continue from the same snapshot with matched agent, tools, environment, and an eight-step limit; FAIL_TO_PASS tests classify preserved, damaged, rescued, and unsolved outcomes.Stable backbone damage requires POST failure in at least two of three runs; prompt comparisons use single-run results.Source: §3.1 · p. 3, Figure 1 · p. 3
• Recurring-detail retention checks values recurring after an artificial boundary, while production-chain retention follows future details across real compactions and can flag completed writes described as pending.Artificial boundaries enable common-reference comparisons across methods; production boundaries expose successive replacements during ongoing work.Source: §3.1 · p. 3

3.2 Data and Construction

• The benchmark uses 35 localized repairs across seven repositories, 146 sessions for recurring-detail retention, and 106 production sessions containing 224 compaction boundaries.Only 140 production boundaries have at least one future detail available for retention scoring.Source: §3.2 · pp. 3–4
• Five production prompts serve as reference methods for task and recurring-detail evaluations, while production-chain retention scores summaries generated by live systems.The prompts are Claude Code, Codex, Gemini CLI, opencode, and Kimi Code.Source: §3.2 · p. 4

3.3 Scoring

• Task damage is the fraction of PRE-pass cases that fail POST; backbone comparisons use 22 validated PRE-pass cases, with stable damage requiring failure in at least two of three runs.Prompt comparisons report single-run outcomes.Source: §3.3 · p. 4
• Recurring-detail scores match case-sensitive literal values from the pre-boundary history and future payloads in the summary, then average session scores without empty reference sets.Each session's reference set contains details extracted from both its history and future payloads.Source: §3.3 · p. 4
• Production-chain scoring matches summary values against details in up to five future payloads without requiring prior-history recurrence, then averages scores by boundary position.The disorder detector flags completed-file basenames appearing near pending-work phrases in a production summary.Source: §3.3 · p. 5

3.4 Evaluation Validation

• Evaluation checks gold-patch solvability, base-repository test failure, and protocol validity; replay restores the same working-tree snapshot, and tests and string matching provide deterministic endpoints.The agent and judge use the same per-instance interpreter and live working tree.Source: §3.4 · p. 5

4.1 Evaluation Setup

• The setup compares methods from shared histories and repository snapshots, using the same Kimi-K3 continuation agent and an eight-step limit.The backbone comparison fixes Claude Code; the prompt comparison fixes Kimi-K3 as summarizer and varies five production prompts.Source: §4.1 · p. 5

4.2 Task Preservation Results

• Figure 2 shows partially overlapping damage: pytest-10081 is preserved only by Claude Opus 4.8, while nine backbones damage sphinx-9230.Claude Opus 4.8 preserves 17 of the 18 repairs on which the backbones disagree.Source: Figure 2 · p. 6, §4.2 · p. 6
• Stable damage spans 5–64% across 14 backbones; Claude Opus 4.8 preserves 21/22 repairs, while DeepSeek-Flash preserves 8.The comparison fixes the prompt, continuation agent, cached history, and repository snapshot.Source: §4.2 · pp. 5–6, §4.1 · p. 5
• With Kimi-K3 summarizing, Kimi Code yields 19/22 POST passes, while Claude Code retains 15.0% of recurring details on the separate 146-session set.The other prompt pass counts are Claude Code 18/22, Gemini CLI 17/22, Codex 16/22, and opencode 16/22.Source: Table 3 · p. 6, §4.2 · p. 6
• Repeated runs confirm 120 of 126 first-round damaged instance–backbone pairs, with Claude Opus 4.8 and DeepSeek-Flash remaining the lowest- and highest-damage backbones.GPT-6-Astra changes from 11 first-round damaged repairs to 9 confirmed cases.Source: §4.2 · p. 6

4.3 Detail Retention Results

• The mechanical anchor_index retains 79.1% of recurring details, compared with at most 16.0% for LLM summaries; Claude Code summaries are about 10,000 characters on the retention set.The extraction reference reads at most 400,000 serialized characters and keeps at most 40 values per type; prompt structures and output lengths also differ.Source: §4.3 · p. 7
• Production-chain mean retention is 15.7% at the first boundary (n=106), 19.9% at the second (n=27), 4.9% at the third (n=6), and 1.8% at the fourth (n=1).Each mean covers the evaluable subset at that boundary, with its own future-request window.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.4 Diagnosing Task Damage

• Four stable Kimi-K3 failures retain target files, and pytest-7571 retains all five audited gold-patch tokens despite failing to continue successfully.In the sphinx-9230 trace, PRE makes a short edit after building and checking a reproduction, while POST searches and reads without repairing; the observed divergence is from evidence to effective edit.Source: Table 5 · p. 8, §4.4 · p. 8

• Compact-Bench combines paired task continuation, recurring-detail matching, and real production-chain measurements to evaluate what survives compaction.Source: §5 · p. 8
• Across fourteen backbones, Claude Opus 4.8 has the lowest stable task damage, within a 5–64% range across the tested backbones.Source: §5 · p. 8
• Retaining paths and code strings can coexist with losing the progress needed to finish a task, making both retained information and subsequent action relevant to evaluation.Source: §5 · p. 9

• Results are conditional on the tested repository repairs, continuation agent, and budget.Source: Limitations · p. 9
• Detail-retention scores measure literal recurrence, while later production-boundary estimates use small, changing session subsets.Source: Limitations · p. 9
• Private session payloads and session-level label values are unavailable, while Appendix G details released scoring rules, configurations, and outcome tables.Source: Limitations · p. 9

• Recurring-detail and production-chain evaluations use real agentic sessions from online traffic on the authors’ institutional platform under existing data-use agreements.Source: Ethical Considerations · p. 9
• Sessions were analyzed in place, not solicited for the study, and processed to extract structural items such as paths, identifiers, and error types.Source: Ethical Considerations · p. 9
• Private session payloads and session-level label values are not distributed because paths, URLs, and IP addresses can identify users or infrastructure.Source: Ethical Considerations · p. 9
• Task Preservation reuses public SWE-bench instances under their original licenses, and experiments ran through metered API gateways under institutional accounts.Source: Ethical Considerations · p. 9

• Table 6 lists first-round outcomes for all fourteen backbones.Source: A First-Round Task Preservation Outcomes · p. 12
• Pytest-10051 and pytest-7236 are excluded because PRE fails in three independent environments.Source: A First-Round Task Preservation Outcomes · p. 12
• Dashes mark cells outside the cross-backbone PRE-pass comparison, and the caption defines outcome codes and model abbreviations.Source: A First-Round Task Preservation Outcomes · p. 12

B Three-Run Task Preservation Verification

• 120 of 126 first-round damaged instance–backbone pairs receive the stable-damage label, while six show variability.Table 7 identifies which conditions were resampled.Source: B Three-Run Task Preservation… · p. 12

C Detail Retention Breakdowns

• Under Kimi-K3 with Claude Code, mean detail retention is 21.4% for Kimi Code sessions, 9.4% for Claude Code, and 6.6% for opencode.Client groups have different task mixes and detail densities; comparisons within each group hold the sampled subset fixed.Source: C Detail Retention Breakdowns · p. 12
• Production boundaries with explicit summary instructions average 19.0% retention, versus 11.4% without instructions.The groups also vary in underlying sessions and compaction conditions.Source: C Detail Retention Breakdowns · p. 12
• Among 20 sessions with scorable first and last boundaries, the mean last-to-first retention ratio is 4.07.The calculation excludes two sessions with zero first-boundary scores.Source: C Detail Retention Breakdowns · p. 12

D Evaluation Validation Results

• Table 6 encodes preserved, damaged, rescued, unsolved, excluded, and unevaluated outcomes for each backbone–instance cell.The backbone comparison is conditioned on full-history PRE success.Source: Table 6 · p. 13
• Sharing the working directory and restoring its snapshot resolves path mismatches seen in 22 of 34 separate-directory pilot cases.With the benchmark’s shared-directory setting, xarray-6721 and pytest-7521 change from damage to preserved, while sphinx-9230 remains damaged.Source: D Evaluation Validation Results · pp. 12–13
• For selected damaged cases, stable damage requires POST failure in at least two of three seeds; seed-variable cases are excluded from stable-damage counts.Other backbones resample the summary and POST while reusing the first-seed PRE baseline.Source: Table 7 · p. 14
• A human reviewer and Kimi-K3 agreed on 59 of 60 sampled auxiliary labels, or 98.3%.The sample contained 20 decisions and 40 progress items; the headline retention scores instead use structured-value extraction rules.Source: D Evaluation Validation Results · pp. 13–14

E Task Preservation Instances

• Table 12 records the repair instances, with target-test counts having a median of 1 and maximum of 10.Older repositories use Python 3.9 when they cannot import under Python 3.10 or later.Source: E Task Preservation Instances · p. 14
• Across the five prompts, DeepSeek-Flash leads recurring-detail retention on three prompts and GPT-5.6-Terra on two.Table 8 reports mean per-session scores across 144–146 sessions per cell.Source: Table 8 · p. 15
• Table 9 reports Kimi-K3 recurring-detail retention by client and prompt as unweighted mean-per-session scores.Its caption identifies the five prompt labels but provides no score values in the supplied passage.Source: Table 9 · p. 15
• Table 11 contrasts a system interpreter, an instance interpreter with separate directories, and an instance interpreter with a shared directory and snapshot restoration.The benchmark uses the shared-directory, snapshot-restoration setting.Source: Table 11 · p. 15

F Task Preservation Outcomes by Prompt

• Table 13 reports first-round outcomes for 22 validated PRE-pass repairs and two excluded PRE-failing cases under five prompts.Kimi Code’s three failures are requests-5414, sphinx-9230, and sympy-13551.Source: F Task Preservation Outcomes by Prompt · p. 15

G Harness and Reproducibility

• The reproducibility package provides extraction and scoring code, transcript preprocessing, agent inputs and tool schemas, environment recipes, and file schemas.It excludes private session payloads and cached task-preservation histories and snapshots; recomputing session aggregates requires source payloads.Source: G Harness and Reproducibility · p. 15

H Production Compaction Prompts

• The production prompts differ in their handoff instructions, including requests for progress and next steps, first-person notes, or structured technical summaries.The supplied Codex and Kimi Code prompt passages specify handoff content, while Gemini CLI requests a detailed, sectioned conversation summary.Source: H Production Compaction Prompts · pp. 17–18

J Repeated File Reads

• A repeated read counts an explicit read_file call targeting a file already read in Phase A; command-based reads are excluded.Table 15 records at most one such read per condition in each of the four stably damaged cases.Source: J Repeated File Reads · p. 18, Table 15 · p. 19

K Data Schemas

• The supplement defines separate records for task-preservation instances, recurring-detail samples, and production-chain retention results.The recurring-detail sample schema stores extracted labels as kind–value pairs, while production-chain records include per-boundary retention fields.Source: K Data Schemas · p. 18

L Supplementary Configuration and Reference Rules

• Session completeness filters out pairs crossing a compaction marker or a large token drop, then checks matching message prefixes and cached-token continuity when available.Completeness is rounded to three decimals before filtering at 0.8.Source: L Supplementary Configuration and… · p. 18

I Worked Example: Action Traces

• In the paired sphinx-9230 trace, PRE repairs the relevant call, while POST continues inspecting tests and reproduction code until its continuation budget is exhausted.The trace reports 8 PRE actions and 13 POST actions within the eight-step budget.Source: I Worked Example: Action Traces · p. 18, Table 14 · p. 19

L Supplementary Configuration and Reference Rules

• Production-chain write-conflict detection uses patterns in the preceding payload and a nearby pending phrase in the summary, so its rate measures conflicts captured by those patterns.Retention includes boundaries with at least one evaluable detail; the last-to-first ratio excludes null final values and zero denominators.Source: L Supplementary Configuration and… · p. 19