PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous Context Management

William Hoy, Jingxuan Fan, Nurcin Celik, Xu Pan

arXiv:2609.35540v1cs.AI

TL;DR

Long-horizon agents must preserve useful information, but full interaction histories grow and can impair performance. This paper updates memory at every turn and trains CCM with privileged full-history distillation. CCM reduces context usage, with task-performance effects varying across models and benchmarks.

  • Problem

    Full interaction histories grow during long-horizon tasks, are bounded by context windows, and can degrade model performance as inputs lengthen or include irrelevant information.

    Source: §1 · p. 1

  • Method

    CCM updates memory at each action, and training combines GRPO with a frozen copy of the student's initial model scoring sampled actions under reconstructed full history.

    Source: §1 · pp. 1–2, §4.1 · p. 5

  • Results

    CCM reduces context usage, while task-success effects vary: three of four TerminalBench-2 models decline, and training improves CCM over GRPO on WebShop and modestly on Endless Terminals.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CCM supports agent operation with substantially reduced retained context, and privileged full-history distillation can improve its task performance in the evaluated settings.

    Source: §1 · p. 2, §5 · p. 9

    The reported shopping and terminal benchmark results do not establish that CCM agents are safe or reliable for unrestricted deployment.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Long-horizon large language model (LLM) agents commonly retain their complete interaction history until compaction is triggered at a predefined threshold. We study Continuous Context Management (CCM), which performs compaction at every turn to prevent interaction history from accumulating in the active prompt. At each turn, a CCM agent emits an updated memory together with an environment action; its next prompt contains the original task, retained memory, and newest observation rather than the complete transcript. We first evaluate CCM without fine-tuning on TerminalBench-2 using Claude Sonnet 4.6, Claude Opus 4.6, GLM-5, and Kimi K3. CCM substantially reduces cumulative input usage and active-prompt size, although it lowers task success for most models while preserving performance for Kimi K3. We use GRPO with privileged full-history distillation to improve CCM in open-weight models. A frozen copy of the student's initial model scores each sampled student action under the complete history reconstructed from that student's rollout, providing dense action-token supervision without a separate teacher rollout or reference solution. On WebShop, this objective substantially improves CCM over GRPO at both evaluated model scales and surpasses full-history GRPO for Qwen3-4B-Instruct, though not for Qwen3-8B. On Endless Terminals, the augmented method provides a modest improvement over GRPO, with both CCM policies outperforming the untrained full-history baseline. These results demonstrate that CCM is a viable inference paradigm for agents operating with substantially reduced retained context and that its performance can be improved through reinforcement learning with privileged full-history distillation.

1 INTRODUCTION

Long-horizon agents need to preserve useful information, but full interaction histories grow and can impair performance; CCM updates memory at every action step. The paper evaluates CCM and introduces privileged full-history distillation to improve its task performance.

Contribution

  • CCM updates retained memory alongside every action, replacing the complete interaction transcript in the next prompt with updated memory and the newest observation.The method addresses the growth of full histories and the performance degradation associated with longer or irrelevant inputs.Source: §1 · p. 1

Motivation

  • Compact active context can reduce KV-cache memory, which may allow larger models to run within the same GPU memory budget.The paper identifies local deployment on consumer-grade GPUs as one relevant setting.Source: §1 · p. 2

Results

  • CCM substantially reduces cumulative input usage on TerminalBench-2, but success falls for Sonnet, Opus, and GLM-5 while increasing slightly for Kimi K3.Monetary savings depend strongly on provider prompt-caching policies and token prices.Source: §1 · p. 2

Contribution

  • A frozen copy of the student's initial model scores sampled action tokens under full history, adding supervision without a separate teacher rollout or successful reference trajectory.The paper combines this objective with reinforcement learning to train memory management and action generation.Source: §1 · p. 2

Results

  • On WebShop, full-history distillation substantially improves CCM over GRPO at both evaluated model scales; on Endless Terminals, it modestly improves over GRPO, while both trained CCM policies outperform the untrained full-history baseline.The training recipe is evaluated while using only a fraction of the prompt context required by full-history baselines.Source: §1 · p. 2

2 RELATED WORKS

The related work situates CCM among long-horizon context-management methods and on-policy distillation approaches. It distinguishes this work through out-of-the-box CCM evaluation and a combination of trajectory-level reinforcement learning with token-level distillation.

Context management for long-horizon agents

  • Prior context-management methods include external memory, periodic summarization, structured compression, evolving context representations, and trained summarization or compression decisions.MEM1 is closest in formulation: it emits an internal state each turn and prunes earlier turns.Source: §2 · p. 2
  • Unlike MEM1, this study evaluates frontier models using CCM without additional training and combines episode-level GRPO with token-level distillation rather than relying on outcome rewards alone.MEM1 trains per-turn consolidation from outcome rewards with PPO.Source: §2 · p. 3

Reinforcement learning and on-policy distillation

  • The method adapts SDAR's confidence-gap gating to multi-turn CCM, comparing sampled action-token probabilities under compact context and a frozen initial model conditioned on reconstructed full history.The auxiliary signal requires no separately trained teacher, additional teacher rollout, or successful reference trajectory.Source: §2 · p. 3

3 CONTINUOUS CONTEXT MANAGEMENT

CCM forms each new prompt from the task, retained memory, and newest observation rather than the full transcript. On TerminalBench-2, it reduces context usage across four models, while task-success and monetary-cost effects vary by model and provider caching.

3.1 METHOD DESCRIPTION

  • At each turn, CCM receives the preceding observation and retained memory, emits updated memory followed by an action, and builds the next prompt from the task, new memory, and newest observation.Earlier responses and observations are not directly included in later prompts.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 TERMINALBENCH-2 RESULTS

  • The evaluation covers all 89 Terminal-Bench 2.0 tasks using Claude Sonnet 4.6, Claude Opus 4.6, GLM-5, and Kimi K3, comparing pass@1 against a full-history baseline.Table 1 reports task success, token usage, and estimated API cost per task; GLM-5 estimates do not include prompt caching because it is unavailable for that model.Source: §3.2 · pp. 3–4, Table 1 · p. 4
  • Mean cumulative input tokens fall by 83.1% for Claude Sonnet 4.6, 81.5% for Claude Opus 4.6, 84.3% for GLM-5, and 75.8% for Kimi K3; pass@1 declines for the first three and rises slightly for Kimi K3.Median episode-level peak prompts decrease from 9,472–17,560 tokens with full history to 3,243–4,280 with CCM; CCM memories peak at 561–1,403 tokens.Source: Table 1 · p. 4, §3.2 · p. 4
  • Estimated costs vary with prompt caching: CCM costs $0.472 versus $0.468 per task for Sonnet, while full-history prompting is cheaper for Opus and Kimi and CCM is cheaper for GLM-5.The Opus estimates are $0.831 for full history and $1.087 for CCM; Kimi estimates are $0.302 and $0.622, and GLM-5 estimates are $0.515 and $0.124.Source: §3.2 · p. 4

4 IMPROVING CONTINUOUS CONTEXT MANAGEMENT VIA REINFORCEMENT LEARNING

This section trains CCM with GRPO plus privileged full-history action-token distillation, then evaluates task performance and general-capability retention. Distillation improves CCM strongly on WebShop but modestly on Endless Terminals, with retention effects varying by model scale, environment, and task.

4.1 GRPO WITH SELF-DISTILLATION

  • A frozen copy of the student’s initial model scores sampled actions under reconstructed full history; confidence-gap gates weight action-only distillation alongside GRPO on memory and action tokens.The teacher scores the same student-sampled action tokens, and the distillation loss is averaged over valid action tokens.Source: Figure 2 · p. 5, §4.1 · pp. 5–6

4.2 WEBSHOP RESULTS

  • For Qwen3-4B-Instruct, distillation raises WebShop exact success by 62.50 percentage points, from 24.22% to 86.72%, exceeding full-history GRPO’s 75.78%.Figure 3 tracks checkpoint performance for both model scales.Source: Figure 3 · p. 6, §4.2 · p. 6
  • For Qwen3-8B, distillation raises WebShop exact success by 46.09 percentage points, from 11.72% to 57.81%, while full-history GRPO remains strongest at 80.47%.The distilled policy’s partial score is 0.728 versus 0.744 for CCM + GRPO.Source: Figure 3 · p. 6, §4.2 · p. 7
  • Distillation increases memory-update rates and reduces cumulative input versus CCM + GRPO: 65.0% for Qwen3-4B-Instruct and 43.8% for Qwen3-8B.The update rates rise to every turn for Qwen3-4B-Instruct and 96.9% of turns for Qwen3-8B; input reductions partly reflect earlier task completion, not context compression alone.Source: §4.2 · p. 7

4.3 ENDLESS TERMINALS RESULTS

  • On Endless Terminals, distilled CCM peaks at 32.00% success (step 50) versus 29.00% for CCM + GRPO (step 30); full-history GRPO reaches 35.33%.Both trained CCM policies outperform the untrained CCM and full-history policies. The untrained baselines are close (21.00% vs 20.33%), possibly because persistent sandbox files and shell state allow recovery of omitted information.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

4.4 GENERAL-CAPABILITY RETENTION

  • Retention effects are nonuniform: privileged distillation provides its clearest benefit for 4B WebShop, with effects varying across scales, environments, and evaluation tasks.For Qwen3-4B-Instruct, the distillation policy changes MMLU-Pro, HellaSwag, and IFEval by −1.03, −5.62, and −0.12 percentage points from the pretrained model.Source: §4.4 · p. 9

5 DISCUSSION AND CONCLUSION

CCM enables agents to operate with compact, continually updated memory, substantially reducing retained context. Its benefits vary across models and environments, and shorter prompts do not necessarily lower API costs.

5 DISCUSSION AND CONCLUSION

  • CCM substantially reduces retained context, and privileged full-history distillation improves task accuracy while mitigating forgetting, with the clearest joint benefit on WebShop with Qwen3-4B-Instruct.Smaller active contexts could accommodate larger models or more simultaneous tasks within the same GPU-memory budget.Source: §5 · p. 9

AI USE STATEMENT

  • Generative AI tools assisted with manuscript wording, LaTeX formatting, and bibliographic metadata checks against primary sources.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • Benchmark results from shopping and terminal environments do not establish that CCM agents are safe or reliable for unrestricted deployment.The paper notes that retained memories can omit information or contain sensitive observations, motivating safeguards and human oversight for consequential applications.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility materials specify protocols, prompts, optimization, splits, rewards, decoding, rollout limits, model variants, and training framework, though hosted API results may vary with provider updates.Source: REPRODUCIBILITY STATEMENT · p. 10

APPENDIX CONTENTS

  • Table 5 covers pass@1 and mean cumulative token usage over 89 Terminal-Bench 2.0 tasks, with full-history prompting as the baseline.Source: Table 5 · p. 14

A TERMINAL-BENCH 2.0 INFERENCE CONFIGURATION

The Terminal-Bench 2.0 configuration evaluates CCM against full-history prompting on a fixed 89-task set. The accompanying GRPO objective uses group-normalized environment advantages and a clipped policy objective with a reference-policy regularizer.

A TERMINAL-BENCH 2.0 INFERENCE CONFIGURATION

  • Evaluation measures pass@1 on 89 fixed Terminal-Bench 2.0 tasks, comparing standard Terminus-2 with complete history against CCM prompts containing the task, prior memory, and latest observation.Each task allows at most 50 turns, with up to 8,192 generated tokens per turn and temperature 0.6.Source: A TERMINAL-BENCH 2.0 INFERENCE… · p. 14
  • Cumulative token counts are provider-reported means per task summed across episode calls; three models use prompt caching, while GLM-5 is evaluated without it.Source: A TERMINAL-BENCH 2.0 INFERENCE… · p. 14
  • CCM prompts omit the full previous trajectory and instead provide the task description, prior context_window, and latest terminal output for the agent to preserve durable state.Source: A TERMINAL-BENCH 2.0 INFERENCE… · p. 14

B GRPO OBJECTIVE

  • GRPO samples groups of student trajectories, computes group-normalized environment advantages, and applies a clipped policy objective with a reference-policy regularizer using a sampled KL estimator.The objective aggregates valid student-generated tokens with a masked token mean.Source: B GRPO OBJECTIVE · p. 16

B.1 OPTIMIZATION HYPERPARAMETERS

The optimization configuration keeps GRPO hyperparameters identical between the two CCM training methods, while adding specified distillation hyperparameters to the augmented method.

B.1 OPTIMIZATION HYPERPARAMETERS

  • CCM + GRPO and CCM + GRPO + Distillation use identical GRPO hyperparameters.Source: §B.1 · p. 16
  • The distillation-augmented method additionally uses λdistill = 0.01 and β = 5.Source: §B.1 · p. 16
  • Table 6 reports optimization hyperparameters for WebShop and Endless Terminals.Source: Table 6 · p. 16

C DETAILS ON THE DISTILLATION OBJECTIVE

The distillation objective uses a frozen initial model to score student-generated action tokens under full-history context, complementing GRPO’s task-level supervision. Its relationship to on-policy distillation is local and does not establish exact reverse-KL minimization for the full training procedure.

Motivation

  • The full-history teacher is not guaranteed to be stronger or correct, and large teacher–student probability gaps need not indicate proportionally reliable supervision.The paper motivates the signal as capturing information potentially missing from compact memory, not as a guarantee of better judgments.Source: C DETAILS ON THE DISTILLATION OBJECTIVE · p. 17

Distillation objective

  • Full-history distillation scores the same sampled action token under compact student context and privileged full-history context, using a detached teacher–student probability gap to gate its likelihood gradient.The teacher is a frozen copy of the student’s initial model; GRPO remains the task-level objective for memory and action generation.Source: C DETAILS ON THE DISTILLATION OBJECTIVE · pp. 16–17

Expected update

  • The displayed expectation is the expectation of the implemented sampled-token gradient, not a fully differentiated current-policy expectation that includes sampling-distribution terms.This distinction constrains how the local update should be interpreted.Source: C DETAILS ON THE DISTILLATION OBJECTIVE · p. 18

Connection to OPD

  • At fixed contexts, the expected SDAR update approaches a scaled OPD gradient for small log-probability gaps, but saturates for large gaps and may then differ in direction.The centered SDAR weight preserves token-level preference sign and ordering while bounding its magnitude.Source: C DETAILS ON THE DISTILLATION OBJECTIVE · p. 18

Scope and limitations

  • The local connection does not differentiate through memory generation, preceding action tokens, or visited histories, and does not establish exact reverse-KL minimization for the complete procedure.Rollout reuse after parameter updates, token filtering, and random eligible-token normalization can also change the expected batch gradient.Source: C DETAILS ON THE DISTILLATION OBJECTIVE · p. 19
  • Subtracting one-half preserves the expected gradient under stated assumptions but generally changes finite-batch gradients; the bounded gate does not guarantee KL reduction or task improvement.The implementation retains the positive sigmoid gate; centering is used only for analysis.Source: C DETAILS ON THE DISTILLATION OBJECTIVE · p. 19

D ADDITIONAL ABLATION RESULTS

In WebShop, privileged action-token distillation alone yields nearly zero exact success, so the reported results do not support replacing GRPO with distillation.

D.1 DISTILLATION WITHOUT GRPO

  • Distillation-only training gives memory tokens no direct training signal, whereas GRPO supervises both memory and action tokens.The ablation retains full-history scoring on student-generated action tokens while setting the GRPO loss weight to zero.Source: §D.1 · p. 19
  • At checkpoint 100, Qwen3-4B-Instruct solves no WebShop tasks and Qwen3-8B solves only one of 128 with distillation alone.Both policies are evaluated under CCM on the fixed 128-task set.Source: §D.1 · p. 19
  • Privileged action-token distillation therefore does not replace verifier-driven GRPO in these WebShop experiments.The reported main experiments use distillation as an auxiliary objective alongside GRPO.Source: §D.1 · p. 19

E.1 BENCHMARKS AND DATA SPLITS

This material specifies the WebShop and Endless Terminals setups, model and evaluation configurations, per-turn prompts, and the procedure used to condense qualitative trajectories.

E.1 BENCHMARKS AND DATA SPLITS

  • WebShop training samples task IDs 500 and above, while evaluation uses IDs 0–127; Endless Terminals uses fixed train, validation, and held-out test splits.WebShop trains for 100 updates; Endless Terminals trains for 60 and evaluates on 300 held-out tasks.Source: §E.1 · pp. 19–20

E.4 EVALUATION

  • Checkpoint evaluation uses pass@1 on fixed task sets, with temperature 0.4 for WebShop and 0.6 for Endless Terminals.Both use seed 42; episodes are limited to 15 turns for WebShop and 16 for Endless Terminals.Source: §E.1 · pp. 19–20, §E.4 · p. 20

E.1 BENCHMARKS AND DATA SPLITS

  • WebShop’s shaped reward combines its partial score with an exact-completion bonus, awarding exactly successful trajectories 10.Unsuccessful trajectories receive rewards from 0 to 2 according to partial progress.Source: §E.1 · p. 19

E.2 ROLLOUT CONFIGURATION

  • The rollout configuration has the self-managed student emit one memory block followed by one environment-action block at every turn.The referenced configuration lists WebShop generation limits of 1,024 tokens for Qwen3-4B-Instruct and 2,048 for Qwen3-8B.Source: §E.2 · p. 20, Table 8 · p. 20

E.3 MODELS AND SYSTEMS

  • WebShop experiments use Qwen3-4B-Instruct-2507 and Qwen3-8B, while Endless Terminals experiments use Qwen3-8B.Training uses VERL with FSDP policy updates and colocated vLLM rollouts on one node.Source: §E.3 · p. 20

E.5 CCM PROMPT TEMPLATES

  • Per-turn prompts pair the task with retained memory and the newest observation; CCM responses update memory and provide an environment action.Endless Terminals supplies the original task separately, while WebShop includes admissible actions in its per-turn prompt.Source: §E.5 · pp. 20–21

F QUALITATIVE TRAJECTORY ANALYSIS

  • The qualitative analysis presents condensed matched trajectories, retaining turns that change memory or environment state, repeated failure modes, final effective actions, and verifier outcomes.Equivalent consecutive turns are summarized explicitly.Source: F QUALITATIVE TRAJECTORY ANALYSIS · p. 22

F.1 WEBSHOP

The WebShop examples contrast exact purchases by distillation-augmented policies with failures by untrained and GRPO-only policies. The two Endless Terminals examples split their matched successes between the trained CCM variants.

F.1 WEBSHOP

  • Neither untrained full-history nor untrained CCM policies purchase a product satisfying the requested WebShop options in the matched example.The full-history policy repeats broad searches; the CCM policy preserves constraints but pages through irrelevant results.Source: §F.1 · p. 22
  • Qwen3-4B-Instruct with CCM plus GRPO selects color but repeatedly misses size 1x, then purchases with partial score 0.857.The memory remains effectively unchanged after the initial search.Source: §F.1 · p. 22
  • The distillation-augmented Qwen3-4B-Instruct WebShop example completes the required search–product–color–size–purchase sequence and receives reward 1.0.Its memory tracks the constraints and interaction stage, including the selected size before purchase.Source: §F.1 · p. 23
  • The Qwen3-8B distillation-augmented example also purchases the target product with both required options selected and receives exact reward 1.0.Its memory retains the constraints and selected actions in a structured representation.Source: §F.1 · p. 23

F.2 ENDLESS TERMINALS

  • The two matched Endless Terminals examples split outcomes: one is solved only by distillation-augmented CCM, the other by CCM plus GRPO alone.The examples also contrast early termination with continued action after a verifier-valid state.Source: §F.2 · p. 23, Table 10 · p. 24

F.2.1 PERMISSION-AUDIT SETUP: DISTILLATION-AUGMENTED CCM SUCCESS

In the permission-audit task, the distillation-augmented policy completes the setup and passes verification, while the untrained and GRPO policies fail in different ways. In the Git export task, CCM + GRPO passes verification despite redundant commands, whereas the distillation-augmented policy fails; neither trained objective dominates both examples.

F.2.1 PERMISSION-AUDIT SETUP: DISTILLATION-AUGMENTED CCM SUCCESS

  • The untrained full-history policy adds a trailing SELinux marker, while untrained CCM falsely records unexecuted operations as complete and terminates.CCM + GRPO later overwrites the verification log with only its third line, leaving one line instead of three.Source: §F.2.1 · p. 24
  • The distillation-augmented policy completes the permission setup in one command, requests completion on turn 2, and passes all three verifier tests.Its trajectory is described as reaching a verifier-valid state without redundant commands.Source: §F.2.1 · p. 24

F.2.2 GIT TRACKED-FILE EXPORT: CCM + GRPO SUCCESS

  • CCM + GRPO eventually passes all five Git-export verifier tests, but repeatedly rewrites valid content instead of terminating.It establishes a correct state early, then continues issuing redundant commands until the turn limit.Source: §F.2.2 · p. 25
  • The distillation-augmented policy fails Git export after reinitializing Git unnecessarily and repeatedly using an invalid option, leaving incorrect contents and no trailing newline.The verifier reports three failed tests.Source: §F.2.2 · p. 25
  • Across these examples, neither trained CCM objective dominates every task, and termination behavior varies substantially by condition.Distillation terminates promptly after permission-task success; CCM + GRPO continues after reaching a correct Git-task state.Source: §F.2.2 · p. 25

Related papers

Loading 2609.35540v1…