PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Compacting Language Model Agents

Tianjian Li, Jingyu Zhang, William Jurayj, Xi Wang, Chuanyang Jin, Mehrdad Farajtabar, Eric Nalisnick, Daniel Khashabi

arXiv:2606.23525v2cs.CL

TL;DR

Long agent trajectories accumulate stale context, while fixed-interval compaction ignores whether reasoning is mid-derivation or converging. SelfCompact lets the model invoke summarization under a lightweight timing rubric, matching or exceeding fixed-interval summarization across six benchmarks and seven models at 30–70% lower cost.

  • Problem

    Fixed-interval compaction cannot distinguish mid-derivation or mid-search states from resolved or converging trajectories, risking poorly timed summarization.

  • Method

    SelfCompact pairs a self-invoked summarization tool with a lightweight rubric specifying when to compact and when to suppress compaction, without fine-tuning.

  • Results

    SelfCompact matches or exceeds fixed-interval summarization across six benchmarks and seven models, improving over no compaction by up to 18.1 points on math and 5–9 points on agentic search at 30–70% lower cost.

  • Takeaways & Limitations

    A lightweight rubric can supply effective adaptive compaction timing to open-weight models at inference time without training.

  • Takeaways & Limitations

    The evaluation covers only open-weight models, so frontier systems may detect context rot without a rubric.

Abstract

from arXiv · show

Long agent traces composed of chains of thought and tool calls accumulate stale content that anchor subsequent generations, and eventually outgrow the context window. Existing scaffolds mitigate it with fixed-interval compaction triggered at a token threshold. Such triggers pay no heed to trajectory structure, risking discard of partial results mid-derivation or mid-search. We propose SelfCompact, a scaffold that allows the model itself to decide when and how to compact. Specifically, it pairs two inference-time elements: (i) a compaction tool the model invokes to summarize the accumulated context, and (ii) a lightweight rubric specifying when to fire (a sub-task has resolved, or the trajectory is converging) and when to suppress (mid-derivation, or when stuck). Both are needed. The tool alone is unevenly used across open-weight models, often invoked at unhelpful moments or not at all; the rubric alone cannot act. Together, they elicit effective adaptive compaction without any fine-tuning or external supervision. We present empirical results on six benchmarks (competitive math and agentic search) and seven models. Our results show that SelfCompact matches or exceeds fixed-interval summarization at a fraction of the token cost, improving over a no-summarization baseline by up to 18.1 points on math and 5-9 points on agentic search at 30-70% lower per-question cost. Our results expose a meta-cognitive gap: although unprompted models cannot reliably tell when their own context is rotting, a lightweight rubric closes this gap, reframing when to compact as a capability that scaffolds can supply without training.

1 Introduction

Longer reasoning and agent trajectories accumulate stale content that causes context rot, while fixed token-threshold compaction ignores reasoning state. SelfCompact addresses this by pairing a model-invoked compaction tool with a rubric for adaptive timing, achieving effective training-free compaction across seven models.

  • Motivation: Longer LM trajectories accumulate flawed reasoning, obsolete search results, and failed programs that anchor subsequent generations, producing context rot.Reasoning models may deliberate for tens of thousands of tokens, while agentic systems add search and code-execution outputs.
  • Problem: Existing systems trigger compaction using rigid token thresholds, which cannot account for whether the model is mid-derivation, converging, or stuck.Deployed and academic systems rely on fixed token intervals; one example triggers when token usage exceeds 30% of maximum context.
  • Method: SelfCompact pairs an inline compaction tool with a lightweight rubric directing the model when to fire after resolution or convergence and when to suppress mid-derivation or when stuck.The model emits <summarize>, after which the scaffold runs summarization and resumes generation.
  • Results: Across seven open-weight models, the rubric enables effective adaptive compaction without fine-tuning and matches or exceeds fixed-interval summarization at lower token cost.The evaluation covers four Qwen competition-math models and three deployed search agents.
  • Ablation: Ablations show that the rubric is crucial: the tool alone produces uneven invocation behavior, whereas guidance closes the gap.Some models invoke the tool at unhelpful moments, while others do not invoke it at all.
  • Contribution: The work introduces and evaluates rubric-based adaptive compaction timing in a training-free setting, which the authors identify as novel.The contribution frames when to compact as an inference-time meta-cognitive capability rather than something baked into model weights.

2 Preliminaries

Long-horizon agents accumulate prior thoughts, tool calls, and observations in a single rolling context, reaching millions of tokens across many turns. Existing compaction strategies use token-budget or fixed-interval heuristics rather than trajectory state.

  • Context management in long-horizon agents: Long-horizon agents accumulate prior thoughts, tool calls, and observations in a single rolling context.A SWE-rebench coding-agent session can average 8M tokens and 154 turns per problem.
  • Current strategies: Reactive compaction triggers only when the rolling context approaches the model’s token budget.This treats compaction as overflow prevention.
  • Current strategies: Periodic compaction fires at a fixed interval—every k turns or every k tokens—regardless of trajectory content.Both current heuristics ignore the state of the trajectory.

3 Our Approach: SELFCOMPACT

SelfCompact equips a language model with a self-applied summarization tool that condenses the original prompt and partial continuation before generation resumes. Its inference-time procedure uses rubric-gated context compaction while preserving the trajectory when compaction is reverted.

  • Context compaction: SelfCompact’s summarizer maps the original prompt and a possibly partial continuation to a condensed context, after which generation resumes from the prompt and summary.The summarizer is the same model as the generator, rather than an external model.
  • Algorithm: The procedure takes a prompt, model, probe interval, step budget, rubric prompt, and summarizer prompt as inputs.These inputs define the inference-time control and summarization components of the algorithm.
  • Control flow: When the generated response is a final answer, SelfCompact returns it; otherwise, reverting compaction leaves the trajectory unchanged.The algorithm explicitly removes the rubric and response messages when reverting.

16 end

SelfCompact combines a model-invoked summarization tool with a lightweight rubric that adaptively decides when to compact. It uses cached-prefix inference and operates without fine-tuning, avoiding unnecessary recomputation while preserving trajectory structure.

  • Method: SelfCompact pairs a summarization tool with a rubric that identifies resolved sub-tasks and suppresses compaction during stuck or mid-derivation trajectories.The tool alone fires unevenly across models, while the rubric supplies concrete, citeable conditions the model can verify locally.
  • Summarizer design: The summarizer appends an instruction to the existing trajectory, preserves its KV cache, replaces the history with the generated summary, and then resumes decoding.Appending rather than substituting means prefill applies only to the added instruction, not the full trajectory.
  • Triggering: At periodic intervals, the model samples a COMPRESS or CONTINUE verdict, with intervals measured in reasoning-trace tokens for math or tool calls for search.Summarization occurs only when the verdict is COMPRESS; otherwise generation resumes from the unchanged trajectory.
  • Workflow: Experiments compare no compaction, fixed-interval compaction, and SelfCompact, which summarizes only after a COMPRESS verdict and continues against the resulting summary.On agentic search, the workflow alternates search, judging, summarization when triggered, and subsequent search.
  • Cost analysis: SelfCompact adds at most two LLM calls per probe interval and avoids the O(L2) re-prefill cost of naive re-encoding by reusing the cached prefix.The rubric probe is essentially free because it generates only a short verdict on top of the cached trajectory; the summarizer runs only after COMPRESS.

4 Experiments

Across competition math and agentic search, SELFCOMPACT consistently outperforms no compaction and fixed-interval summarization under matched budgets, while substantially reducing per-question cost. Its benefits arise from adaptive timing: rubric-guided compaction avoids damaging compressions, fires earlier when useful, and helps most on difficult problems.

  • Competition math: SELFCOMPACT achieved the best performance in 11 of 12 competition-math settings under matched token budgets.The evaluation covered four Qwen model configurations and three competition-math benchmarks, with 16 responses generated per question.
  • Competition math: On Qwen3.5-9B, SELFCOMPACT improved over baseline by 16.4, 10.0, and 18.1 points on IMO-Answerbench, HMMT Nov, and HMMT Feb, respectively.These were the largest reported gains for the thinking-enabled models.
  • Adaptive timing: 40.4% of fixed-interval summarization transitions caused degradations, while an oracle skipping summaries when the current answer was correct achieved 52.9%.The oracle improved by +11.5 over fixed interval and +14.0 over baseline, indicating headroom for adaptive policies.
  • Agentic search: On BrowseComp-Plus, SELFCOMPACT improved over no compaction by +8.5, +9.2, and +5.3 absolute points for GLM-4.7-Flash, MiniMax-M2.5, and Mimo-V2-Flash, respectively.SELFCOMPACT was the strongest method for all three deployed agents, with accuracy ordered baseline < fixed-interval ≤ SELFCOMPACT.
  • Agentic search: BrowseComp-Plus cost dropped by 67% (0.12 →0.04), 63% (0.19 →0.07), and 33% (0.24 →0.16) for GLM-4.7-Flash, MiniMax-M2.5, and Mimo-V2-Flash, respectively.KV-cache reuse keeps rubric probes affordable because the trajectory cache is preserved.
  • Adaptive timing: SELFCOMPACT’s rubric-fired summaries occurred before the fixed 30% context threshold across all three agentic-search models and delivered its largest gains on the two hardest question bins.The rubric’s left-skewed trigger distribution indicates earlier adaptive compression, while easy-bin differences remained within sampling noise.

5 Related Work

Prior context-compaction methods predominantly use fixed token or trajectory intervals, while SelfCompact dynamically determines when compaction should occur without model training. Related work also includes KV-cache eviction and compression methods that reduce inference memory and compute costs.

  • Context compaction in frontier models: API-based models commonly trigger context compaction automatically when context length reaches a target threshold, making the operation effectively fixed-interval.This practice has prompted community complaints.
  • Learning to compact during post-training: Post-training methods periodically summarize accumulated thoughts or actions and resume from the compressed state, addressing stale and rapidly growing agent trajectories.These methods share a generate–summarize–resume paradigm.
  • Learning to compact during post-training: Most existing methods trigger compaction at fixed token-count thresholds, risking mid-derivation context loss; SelfCompact instead fires after sub-task resolution or convergence and suppresses compaction mid-derivation or when stuck.The rubric explicitly targets trajectory structure rather than token count alone.
  • Training-free compaction: The closest training-free approach uses a frontier model as an external summarizer after each full search trajectory, whereas SelfCompact dynamically determines when compaction should fire.SelfCompact therefore differs from fixed post-trajectory intervals.
  • KV Cache Eviction: KV-cache eviction reduces inference memory and compute by retaining or compressing entries using recency, attention scores, or richer importance measures that are often learned.This line of work is complementary to context compaction.

6 Limitations

The study evaluates only open-weight models and focuses on training-free interventions, leaving frontier-model metacognition and reinforcement-learning extensions for future work.

  • Model capability gap: The evaluation covers only open-weight models, so frontier systems may detect context rot without a rubric; SelfCompact is intended to complement any model.The scaffold specifically targets deployed open-weight agents where the capability gap exists.
  • No reinforcement learning: The work excludes reinforcement learning to isolate the rubric’s contribution, although prior work suggests RL can teach when and what to compact.The authors view the rubric as a behavioral target that RL could distill into the policy.

7 Conclusion

SELFCOMPACT is a rubric-gated scaffold that compacts agent trajectories at closed reasoning units instead of fixed intervals. Across six benchmarks and seven models, it matches or exceeds fixed-interval summarization at 30–70% lower cost, supplying adaptive compaction without training.

  • 7 Conclusion: SELFCOMPACT compacts agent trajectories at closed reasoning units rather than on fixed intervals.It uses a rubric-gated summarization scaffold.
  • 7 Conclusion: 30–70% lower cost: SELFCOMPACT matches or exceeds fixed-interval summarization across six benchmarks and seven models.The comparison covers six benchmarks and seven models.
  • 7 Conclusion: A lightweight rubric supplies the capability to decide when to compact without training.This reframes when to compact as a capability provided by the scaffold.

A Math experimental setup

The math experiments evaluate four Qwen models across three competition-math benchmarks under fixed sampling and generation budgets. They compare no summarization, fixed-interval summarization, autonomous SelfCompact, and rubric-guided firing with answer- or next-step-preserving branches.

  • Models: Four open-weight Qwen models are evaluated: two instruction-tuned models and two thinking-disabled variants.Qwen3-4B-Instruct-2507, Qwen3-30B-A3B-Instruct-2507, Qwen3.5-4B, and Qwen3.5-9B are served via vLLM.
  • Benchmarks: The evaluation uses IMO-Answerbench, HMMT November 2025, and HMMT February 2026, with final answers scored by math_verify.
  • Sampling parameters: All models use temperature 1.0, top-p 0.7, n = 16 samples per question, and report mean ± std over samples.Each generation round is capped at 16,384 tokens, trajectories allow 12 rounds, and summaries are hard-truncated at 512 tokens.
  • Methods: The study compares BASELINE, FIXED-INTERVAL SUMMARY, SELFCOMPACT, and SELFCOMPACT W/ RUBRICS under distinct summarization triggers.BASELINE uses one 16,384-token round without summarization; fixed-interval summarization fires every round, while SelfCompact fires when the model emits the summarizer stop token.
  • Methods: The rubric fires compression iff Q1 = Y or (Q2 = Y and Q3 = Y), using a lock-in branch for final answers and a reset branch for named next steps.Q1 checks for a specific final answer, Q2 checks whether the last two rounds added no non-trivial fact, and Q3 checks whether an exact next step can be written.

B Agentic-search experimental setup

The agentic-search experiments evaluate three deployed agents on three benchmarks using a shared decoding configuration and an unmodified ReAct-based scaffold. The scaffold combines model-triggered compression with a rubric requiring closed-unit, summarizable, progressing, and non-stuck trajectories before compression.

  • Models and serving: Three deployed agents—GLM-4.7-Flash, MiniMax-M2.5, and Mimo-V2-Flash—run through OpenRouter via the AggAgent runtime with the unmodified react_agent_selfcheck scaffold.The rubric and summarizer prompts are reproduced verbatim in Box B.
  • Benchmarks: Three benchmarks—BrowseComp, BrowseComp-Plus, and DeepSearchQA—are each sub-sampled to 150 questions.This follows the sampling procedure used by Sun et al. (2025) and Lee et al. (2026).
  • Sampling parameters: All three models use temperature 1.0, top-p 0.95, 10,000 maximum output tokens per call, disabled parallel tool calls, and a 100-call per-trajectory cap.Per-model context windows, fixed-interval triggers, rubric backstops, and decoding flags are provided in Table 7.
  • Scaffold details: Each trajectory follows a standard ReAct loop in which agents search, inspect or retrieve documents, reason over results, and either continue tool use or answer.BrowseComp and DeepSearchQA use Serper search and crawl4ai-based page extraction, while BrowseComp-Plus uses get_document_bcp.
  • Search rubric: Compression fires only when the trajectory is a closed unit, reducible to 3–5 cite-able facts, has progressed, and is not stuck.The probe requires evidence for each decision and uses the rule COMPRESS iff C1 = Y ∧ C2 = Y ∧ C3 = Y ∧ N1 = N.

C Cost analysis of summarization · D Trajectory token consumption

The paper models summarization cost through cumulative prompt and completion tokens, with compaction reducing later prompt lengths substantially. It also decomposes trajectory token consumption into output, prefill, and cached-input charges across assistant, rubric, and summarizer calls.

  • C Cost analysis of summarization: Under single-rate billing, per-question cost is pcache · Nprompt + pout · Nout across assistant, rubric-probe, and summarizer calls.Nprompt and Nout are accumulated over every LLM call in the trajectory.
  • C Cost analysis of summarization: The calculation averages token usage and model-price costs over 150 sampled questions per benchmark.Usage records are loaded, token totals accumulated, prices applied from Table 8, and means reported in USD per question.
  • C Cost analysis of summarization: Compaction saves cost by shortening every subsequent prompt, collapsing 50–100k-token trajectories to approximately 1–3k-token summaries.This produces 20–80× shrinkage of post-compaction prompts under the single-rate cache model.
  • C Cost analysis of summarization: For math, Qwen runs use local vLLM, so the reported [Xk] budget represents generated and refinement tokens rather than dollar cost.Fixed-interval summarization is matched to SelfCompact within ±3k tokens, yielding identical compute within each model block.
  • D Trajectory token consumption: A SelfCompact trajectory proceeds from the question through assistant turns to a rubric-triggered summary, followed by further assistant turns.The lifecycle is represented as Q → A1 → A2 → · · · → Ak → S → A′1 → A′2 → · · ·.
  • D Trajectory token consumption: Each produced token incurs one output charge, one next-call prefill charge, and one cached-input charge for every later reuse before prefix reset.This accounts for the distinct billing stages over a token’s lifetime.
  • D Trajectory token consumption: Table 9 reports cumulative prompt and completion tokens, splitting prompts into first-time prefill and cache reads; its two-rate costs exceed the single-rate approximation slightly.The single-rate main-text cost is recovered from Nprompt and Nout, while Table 9 applies separate prefill and cached-input prices.
  • D Trajectory token consumption: For math benchmarks, Table 10 separates cumulative refinement-summary prompts from per-round generation output, with Total equal to Prompt + Output.Its Output column matches the [Xk] budget tag in Table 1.

E Qualitative trajectories

Across three BrowseComp-Plus questions on MiniMax-M2.5, SELFCOMPACT answered correctly while fixed-interval compaction and no compaction answered incorrectly. The trajectories illustrate that compaction timing—waiting for resolved sub-tasks rather than summarizing on a blind clock—determines success.

  • Overall comparison: Across all three questions, SELFCOMPACT answered correctly, whereas fixed-interval compaction and the no-compaction baseline answered incorrectly.The comparison used fixed-interval compaction at 30% of the context window and SELFCOMPACT, which compacts when the rubric judges a sub-task closed.
  • Compaction timing: Fixed-interval compaction summarizes whatever is in context whenever the prompt reaches ≈60k tokens, including dead-end searches.This corresponds to 30% of MiniMax-M2.5’s context window, while SELFCOMPACT can let the prompt grow past 30% during exploration.
  • Case A: Whitesnake: In the Whitesnake case, fixed-interval compaction remained anchored to a wrong Paul McCartney/Wings lead, while SELFCOMPACT broke out and reached the correct answer.The gold answer was Whitesnake, formed by David Coverdale, an ex-Deep Purple musician.
  • Case B: Majida El Roumi: In the Majida El Roumi case, fixed-interval compaction froze a wrong lead and answered “None,” while SELFCOMPACT identified Majida El Roumi.The question’s gold answer was Majida El Roumi, whose “Habibi” is built on Albinoni’s Adagio.
Loading 2606.23525v2…