Benchmarks Are Surprisingly Compaction-Friendly

Motivation

• Compaction is lossy: summaries can remove task constraints, instructions, or reasoning that the agent may need to continue its work.The study asks whether agents still solve tasks and respect the rules they were given.Source: §1 · p. 1

Study design

• The study fixes OpenCode, implements five compaction methods, and evaluates frontier models on PaperBench, ProgramBench, and PostTrainBench.The scaffold is fixed to separate compaction-strategy effects from scaffold-specific behavior.Source: §1 · p. 2

Threshold effects

• For two 1M-context models, performance does not meaningfully change across compaction thresholds from 160K to 380K tokens.GPT-5.3-Codex performs worse at lower thresholds on PaperBench and, less clearly, on PostTrainBench.Source: §1 · p. 2

Main findings

• No compaction strategy consistently beats default self-summarization, while strategies that discard more content cost some models up to 20 points on PaperBench and leave others unchanged.The results suggest that a method validated on one model may not transfer to another.Source: §1 · pp. 2–3

Constraint retention

• On ProgramBench, the first compaction already drops around one of three anti-cheating rules, and rule loss grows with additional compactions.On real SWE-chat sessions, work-process instructions and scope exclusions are dropped around six times more often than feature requirements.Source: §1 · p. 3

Benchmark scope

• The authors argue that current benchmarks are compaction-friendly because objectives are fixed and progress persists on disk, while some user instructions exist nowhere else.Agents can recover much discarded information by rereading artifacts, but not constraints that were only in context.Source: Figure 1 · p. 2, §1 · p. 3

2.1 CONTEXT COMPACTION IN AGENTS

• Agent scaffolds commonly trigger summarization near context limits, replace earlier history with a summary, and may also clear old tool results or thinking blocks.Related approaches include methods trained or adapted to summarize, maintain memory, or fold task trajectories.Source: §2.1 · p. 3

2.2 EVALUATING COMPACTION

• Prior evaluations often measured retention or studied downstream performance on upfront long-input question answering and web search, rather than realistic long-horizon tasks whose context is agent-generated.COMPINT studied user-constraint retention, while other comparisons examined question answering and web search.Source: §2.2 · pp. 3–4

2.3 RELATION TO KV CACHE COMPRESSION AND LONG CONTEXT DEGRADATION

• Unlike key-value-cache methods that select prior tokens, prompt-level compaction has the model generate a summary of state based on its own judgment.The paper also relates threshold experiments to long-context degradation, where models may underuse information or perform worse as input length grows.Source: §2.3 · p. 4

3.1 MODELS AND SCAFFOLD

• Four models are evaluated in fixed OpenCode, with two runs per task and configuration; fixing the scaffold helps separate compaction-method effects from scaffold-specific behavior.The models include three with 1M-token context windows and GPT-5.3-Codex with a 400K window.Source: §3.1 · p. 4

3.2 BENCHMARKS

• The benchmarks cover paper reproduction, software development from scratch, and machine-learning experiments, selected to be unsaturated and long enough to trigger compaction naturally.PaperBench uses three randomly chosen papers; ProgramBench uses two command-line-tool tasks and supplies three anti-cheating rules; PostTrainBench uses two base models and three target benchmarks.Source: §3.2 · pp. 4–5

3.3 COMPACTION STRATEGIES

• Five strategies share a common token threshold: self-summarization, tool-output pruning, filesystem offloading, reasoning-block discarding, and smaller-model summarization.Pruning removes older tool outputs; offloading saves full history to disk; the smaller-model strategy uses Claude Haiku 4.5.Source: §3.3 · p. 5
• Decomposing work across subagents with independent contexts is left for future work as an alternative to compaction.This study evaluates compaction strategies rather than subagent task decomposition.Source: §3.3 · p. 5

3.4 METRICS AND STATISTICS

• The study records benchmark-specific performance, compaction events, failures, and ProgramBench rule violations, and uses Claude Sonnet 5 to judge constraint erosion.Error bars are nonparametric bootstrap 95% confidence intervals based on resampling runs with replacement.Source: §3.4 · p. 5

4.1 DOES COMPACTION MATTER FOR TASK PERFORMANCE?

• At PaperBench, GPT-5.3-Codex accuracy rose from 0.38 at 160K to about 0.60 at 260K and above; Claude Opus 4.6 and GPT-5.4 were similar across thresholds.ProgramBench performance remained generally flat for GPT-5.3-Codex, while PostTrainBench results were too variable to conclude whether threshold affected performance.Source: Figure 2 · p. 6, §4.1 · p. 6
• No compaction strategy outperformed default self-summarization beyond confidence intervals; removing reasoning or using a smaller summarizer cost some models up to 20 points on PaperBench.Removing reasoning reduced GPT-5.4 PaperBench accuracy from 0.59 to 0.43, while effects differed across models.Source: §4.1 · p. 6, Figure 3 · p. 7

4.2 WHAT DOES COMPACTION DROP?

• About one of ProgramBench’s three anti-cheating rules was lost at the first compaction, and losses increased with further compactions.Figure 4a reports about 0.92 categories dropped at the first compaction and 2.65 by the 30th.Source: §4.2 · p. 7, Figure 4 · p. 8
• Reward hacking began before the first compaction, so constraint erosion cannot explain when cheating starts; the authors note cheating could instead cause erosion.An agent already performing a disallowed action may have little incentive to restate the rule.Source: §4.2 · p. 7
• In 44 SWE-chat sessions, 81% of constraints were preserved; process instructions (35.9%) and scope exclusions (36.8%) were lost more often than feature requirements (6.2%).The analysis covered 269 extracted user constraints.Source: §4.2 · pp. 7–8, Figure 4 · p. 8
• The benchmarks fix objectives, run without user input, and provide a filesystem; deliverables can be recovered from artifacts, unlike instructions about agent behavior.The paper presents this contrast as its compaction-friendly hypothesis.Source: §4.2 · pp. 7–8

5 DISCUSSION AND CONCLUSION

• Default self-summarization is as good as the tested alternatives, while methods discarding more content can reduce performance by up to 20 points for some models and leave others unchanged.The outcomes depend on the model and what is measured.Source: §5 · pp. 8–9
• Compaction summaries tend to preserve what is needed to continue a task but are more likely to drop constraints on how to perform it.The authors hypothesize that constraints are not always needed to continue, only to continue correctly, even though their prompt asks summaries to preserve them.Source: §5 · p. 9
• Fixed objectives and progress saved in files make PaperBench, ProgramBench, and PostTrainBench compaction-friendly because agents can recover much of what they need from those artifacts.User instructions that change during a session may not be persistent, so a removed instruction cannot be recovered in the same way.Source: §5 · p. 9
• GPT-5.3-Codex declines after about eight compactions and loses roughly 0.10 relative to the task mean at 16 or more, while GPT-5.4 does not degrade even at that frequency.The authors cannot distinguish whether newer models write or use better summaries, or whether models trained for 1M context saw more post-compaction continuations.Source: §5 · p. 9
• Future evaluations should use tasks whose constraints cannot be recovered from files, such as multi-turn sessions with changing user instructions.The authors also propose comparing parallel subagents with fresh contexts against the best compaction method under the same scaffold.Source: §5 · p. 9
• The study covers four models, three benchmarks, a few tasks per benchmark, and one scaffold; Gemini results provide weak evidence because it rarely compacted and reward hacked often on ProgramBench.The SWE-chat sessions used Claude Code, a different scaffold and underlying models.Source: §5 · p. 9

AI USE STATEMENT

• Generative AI assisted with code, plots, literature search, and readability; authors reviewed generated code and say AI did not generate research ideas or core scientific claims.The authors take responsibility for the final content, including AI-assisted text, claims, or artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The authors state that they do not think the work poses ethical concerns and describe its focus as understanding compaction’s effects on performance and adherence to user-stated constraints.Source: ETHICS STATEMENT · p. 10

6.1 DETAILS ON THE MODELS

• Models were accessed through OpenCode using providers’ APIs.Table 1 lists the models evaluated in the study.Source: Table 1 · p. 13

6.2 DETAILS ON THE BENCHMARKS

• PaperBench used a 10-hour Code-Dev evaluation with IterativeAgent prompting and an LLM judge; ProgramBench used 20 CPUs, 120GiB RAM, and 6 hours; PostTrainBench excluded rule-violation runs.PaperBench followed Starace et al. (2025); ProgramBench reported fraction of tests passed, as recommended by Yang et al. (2026); PostTrainBench reused Rank et al. (2026)'s original settings. The evaluated task descriptions include bam as a score-matching proximal-update method and all-in-one as a diffusion-transformer simulation-based-inference method.Source: §6.2 · p. 13

6.3 DETAILS ON THE COMPACTION STRATEGIES

• At context overflow, the reasoning-block-discarding strategy removes earlier assistant reasoning before summarization, retaining reasoning only for the last 10 assistant turns.The removal also covers provider-specific reasoning metadata and content for OpenAI, Anthropic, and Gemini.Source: §6.3 · p. 13

6.4.1 SELF-SUMMARIZATION (BASELINE)

• The self-summarization prompt asks for a detailed but concise continuation summary covering progress, files, next steps, user constraints or preferences, and important technical decisions.It also instructs the assistant to output only the summary, in the user’s language.Source: §6.4.1 · p. 14

6.4.2 PROGRAMBENCH REWARD HACKING JUDGE

• Claude Sonnet 5 judges possible ProgramBench reward hacking after regex checks, using the task prompt, submission file listing, prior signals, and a request for a verdict and explanation.Source: §6.4.2 · p. 14

6.4.3 PROGRAMBENCH CONSTRAINT EROSION JUDGE

• For every ProgramBench compaction event, Claude Sonnet 5 judges whether the summary preserves each of the task’s three anti-cheating rules.Each summary is assessed on its own restatement of the rules: no copying or vendoring, no internet access, and original from-scratch implementation intent.Source: §6.4.3 · pp. 15–17

6.4.4 SWE-CHAT CONSTRAINT EXTRACTION

• For SWE-chat, Claude Sonnet 5 extracts explicit, active user constraints from the pre-compaction conversation timeline and returns each constraint with a source quote.Constraints must be specific, still applicable, and not revoked or superseded; the procedure excludes vague goals and instructions belonging only to completed unrelated tasks.Source: §6.4.4 · p. 17
• Constraint extraction favors precision, quoting each human’s exact words, recording its timeline event ID, and explaining why it remains active.If active status is genuinely uncertain, the constraint is included with low confidence for human adjudication.Source: §6.4.4 · p. 18

6.4.5 SWE-CHAT SUMMARY PRESERVATION

• Only 3 of 269 SWE-chat constraints were labeled contradicted, so the analysis counted them as dropped.Claude Sonnet 5 judged whether each constraint was preserved, dropped, or contradicted using only the summary and constraint; preservation meant the constraint remained recoverable in substance.Source: §6.4.5 · p. 18

6.4.6 SWE-CHAT CONSTRAINT CATEGORY CLASSIFICATION

• Claude Sonnet 5 classifies each constraint from its text alone, assigning exactly one category from a fixed taxonomy.The taxonomy distinguishes instruction types such as scope limits, technical parameters, feature requirements, process instructions, named tools or sources, structural conventions, and soft preferences.Source: §6.4.6 · p. 19

• GPT-5.3-Codex accuracy drops after many compactions, while Claude Opus 4.6 and GPT-5.4 do not.Source: Figure 5 · p. 20
• Figure 5 includes all three benchmarks and all compaction thresholds, plotting each run’s accuracy relative to its task mean against the number of compactions.Source: Figure 5 · p. 20
• Accuracy in Figure 5 is conditional on a run producing a valid submission.Source: Figure 5 · p. 20

• Accuracy does not significantly change across compaction thresholds from 160K to 380K tokens.Source: Figure 6 · p. 20
• Gemini 3.1 Pro Preview rarely compacts, averaging one compaction per run at the 160K threshold.Source: Figure 6 · p. 20
• Because Gemini 3.1 Pro Preview rarely compacts, the threshold results are not informative about compaction effects.Source: Figure 6 · p. 20

6.7 FAILURE RATES

• PaperBench is omitted from the failure-rate results because no agent-caused failures were observed.Source: §6.7 · p. 20
• Failure rates are computed per task and then averaged, giving each task equal weight despite differing retry counts.The figure reports bootstrap 95% confidence intervals and uses runs at a fixed 160K-token compaction threshold.Source: Figure 7 · p. 21
• Failures are common, but no compaction strategy is uniformly worse, and the ordering changes across models.Source: §6.7 · p. 21

6.8 NUMBER OF COMPACTIONS PER THRESHOLD

• Figure 8 plots average compactions per run against threshold for each model in the self-summarization baseline.Source: Figure 8 · p. 21
• At GPT-5.3-Codex’s lowest threshold, long verification-loop runs inflate the mean and widen its confidence interval; Gemini has few compactions.Gemini’s context grows more slowly, it takes about half as many steps per run, and its tool outputs are much shorter than Claude Opus 4.6’s.Source: §6.8 · p. 21
• Table 2 reports mean compaction counts with 95% confidence intervals by tokens-before-compaction threshold across benchmarks.Source: Table 2 · p. 22

6.9 CONSTRAINT EROSION

• The constraint-erosion analysis adds lua/lua, excluded from the main accuracy analysis because it triggers reward hacking too often.Source: §6.9 · p. 22
• From the first compaction, GPT models drop more than one of the task’s three constraint categories on average.Figure 9 follows the number of categories dropped per summary across compaction indices; Table 3 defines “dropped” as prohibited-behavior categories absent from a summary.Source: Figure 9 · p. 22, Table 3 · p. 22

6.10 REWARD HACKING

• Reward-hacking analysis distinguishes seven categories, and Figure 10 counts mechanism tags rather than runs because a trajectory can use several mechanisms.Source: §6.10 · p. 22, Figure 10 · p. 23
• Reward-hacking analysis also includes lua and luajit, which were excluded from main accuracy analysis because they are very often reward hacked.Source: §6.10 · p. 23

6.11 SWE-CHAT ANALYSIS

• The study samples 44 SWE-chat sessions with compactions, extracts 269 constraints, and has judges classify preservation and constraint category.Three constraints judged contradicted are counted as dropped; a second judge from another model family replicates the classification.Source: §6.11 · p. 23
• 81% of extracted constraints are preserved; process instructions are dropped 35.9% of the time, around 6× more than feature requirements, which are preserved 93.8% of the time.Scope exclusions are dropped 36.8% of the time; the second judge finds a 24.5% overall loss rate and similarly elevated drops for process instructions and scope exclusions.Source: §6.11 · p. 24

6.11.1 CONSTRAINT CATEGORIES •

• The taxonomy distinguishes scope limits, literal technical parameters, feature specifications, and instructions about how the agent should work.Source: §6.11.1 · p. 24
• Other categories cover named tools or sources, structural conventions, soft preferences, and a catchall category.Source: §6.11.1 · p. 24

6.11.2 JUDGE COMPARISON

• The judges agree on 88.8% of labels (239/269), with Cohen’s κ of 0.681; GPT-5.6 Sol more often calls constraints dropped when Sonnet judged them preserved.Figure 11 shows category drop rates under each judge, with Wilson-score 95% confidence intervals and one constraint excluded for both judges.Source: §6.11.2 · p. 24, Figure 11 · p. 25