Context Language Models

1 Introduction

• Context management is not traditionally a native LM capability, and prior approaches rely on hand-engineered or agent-optimized harnesses or fixed-action tools.The paper contrasts these methods with unrestricted model access to context.Source: §1 · p. 1
• CLMs treat context as a model-controlled artifact, allowing context updates beyond the standard LM practice of appending generated tokens.The implementation mirrors context into a file that the model can append to or freely edit, with changes synchronized to the live context.Source: §1 · p. 1
• Context as a file extends to multi-agent workloads, where CLMs can manage multiple agents’ context files.The paper also introduces ContextBench to diagnose context management separately from reasoning and knowledge.Source: §1 · pp. 1–2
• CLMs outperform baselines across long-horizon tasks, including 11.4% higher accuracy with 21.5% fewer prefix-reuse FLOPs on BrowseComp-Plus.They also score 5% higher with 59% fewer prefix-reuse FLOPs on 12-hour EdgeBench, and achieve 65% greater end-to-end speedup at the same compute on a 24-hour six-repository agent-swarm task.Source: §1 · p. 2
• With Suffix Cache Reuse, CLMs reduce server-side compute by 35% at matched performance compared with standard SGLang serving.The method reuses cached states beyond matching prefixes to reduce re-prefilling while empirically preserving task performance.Source: §1 · p. 2
• CLMs can follow natural-language instructions and evolve reusable skill documents, improving held-out ContextBench accuracy by up to 35.9 points at lower compute.The paper also reports online reinforcement learning with a success-gated efficiency advantage for stepwise GRPO.Source: §1 · pp. 2–3

2 Related Work

• Existing methods increasingly let models choose context operations, but constrain them to a human-defined action space.Examples include deciding when to compact, triggering compaction or retrieval, and selecting context fragments.Source: §2 · p. 3
• Recursive Language Models access long inputs through a REPL variable, but retrieved information is still appended to a live context that grows over time.CLMs instead make the live context editable.Source: §2 · p. 3
• The paper points to extended related-work discussion on meta-harness optimization, reinforcement learning, cache reuse, and external memory.That discussion is provided in Appendix A.Source: §2 · p. 3

3 Pilot Study with ContextBench: A Diagnostic for Context Management

• ContextBench tests selective retention, in-place updates, and exact recall through Needle Retention, Sudoku Sketchpad, KV Store, and Log Triage.The benchmark is designed to isolate context management from other reasoning or knowledge capabilities.Source: §3 · p. 3
• At a 32K context limit and up to 24× input pressure, existing strategies fail to perform perfectly across these simple tasks.Summary-based compaction can lose or hallucinate information; methods without flexible in-place editing must regenerate Sudoku state after each fine-grained edit; coding tools can offload but not evict information on demand.Source: §3 · p. 3
• Figure 2 illustrates the four ContextBench tasks and compares CLM with baselines using GPT-5.4 at a 32K context limit.The reported failures motivate fully adaptive, model-controlled context management.Source: §3 · p. 3, Figure 2 · p. 4

4.1 Formal Definition and Implementation with Context as a File

• CLMs directly create the next context, with θ allowing arbitrary model-controlled context-management functions rather than only predefined harness tools.The implementation mirrors live context in an editable file synchronized with the model; multiple context files support multi-agent workflows.Source: §4.1 · p. 4
• Figure 3 illustrates context behaviors including in-place agent-status updates, role creation, irrelevant-result removal, reusable compaction functions, and summarization of 21K tokens.The examples come from zero-shot CLM evaluation experiments.Source: §4.1 · p. 5, Figure 3 · p. 5

4.3 (More) Efficient CLM Serving with Suffix Cache Reuse

• Prefix-reuse FLOPs measure theoretical inference cost under prefix-cache serving, while Suffix Cache Reuse reuses cached states for surviving tokens to reduce re-prefilling.Standard serving re-prefills tokens after the first prefix mismatch; SCR can reuse preserved suffix states, including reasoning-token states stripped in some chat-serving setups.Source: §4.1 · p. 5, §4.3 · p. 6, Figure 4 · p. 6

4.2 In-Context Learning and Reinforcement Learning for CLMs

• CLMs can use in-context instructions or skill documents for context management, and a prompt-evolution loop selects skills using training rollouts and development-set evaluation.The final selected skill is evaluated once on the held-out test split.Source: §4.2 · pp. 5–6
• Stepwise GRPO assigns trajectory-level outcome advantages to each segment, while a success-gated efficiency advantage rewards lower prefix-reuse FLOPs among successful trajectories.When a group has fewer than two successful trajectories, its efficiency advantage is set to zero.Source: §4.2 · p. 6

4.3 (More) Efficient CLM Serving with Suffix Cache Reuse

• Table 1 reports that CLMs outperform specialized OpenEvolve evolutionary workflows on mathematical optimization problems.The comparison uses Claude 4.6 Sonnet, a 32K context limit, and a cap of 100 scored attempts or five hours.Source: Table 1 · p. 7
• Figure 5 reports that CLMs perform better than action-based and harness-defined baselines at lower cost on coding and deep research tasks.All methods use Qwen3.6-27B, a 32K context limit, and a 100-turn cap; blue dashed lines mark the Pareto frontier.Source: Figure 5 · p. 7

5.1.1 Coding and Deep Research Tasks

• 59.4% accuracy on BrowseComp-Plus exceeded the strongest baseline by 11.4% relative, with 21.5% fewer prefix-reuse FLOPs; CLMs matched the strongest baseline on TB2.1 at 70% of its FLOPs and scored 73.7% versus 67.0% on TBLite.The comparisons use Qwen3.6-27B with a 32K context limit.Source: §5.1.1 · p. 7

5.1.2 Open Discovery Problems

• CLM achieved the highest best-of-run score on all four mathematical optimization problems, outperforming a specialized evolutionary workflow with less fixed orchestration.The evaluation used Claude 4.6 Sonnet and the same evaluator.Source: §5.1.2 · p. 8
• 44.6 with 179 prefix-reuse PFLOPs per trial on EdgeBench-10, versus summarization's 42.3 with 437 PFLOPs; subagents added little benefit on this single-repository benchmark.With Claude 4.6 Sonnet, CLM and its subagent variant reached 51.0 and 50.4, versus 42.3 for summarization.Source: §5.1.2 · p. 8, Figure 6 · p. 8
• 65% greater downstream speedup over initial releases at the same spend was achieved by the CLM agent swarm on four unseen packages.Six agents jointly optimized interdependent repositories; the figure reports geometric-mean speedup over 17 evaluation tasks.Source: §5.1.2 · p. 8, Figure 6 · p. 8

5.2 Learning Better Context-Management Strategies in Context or in Weights

• A single natural-language instruction steered compaction timing, semantic boundaries, or context backup without changing the harness or model parameters.The demonstrations covered three distinct context-management behaviors.Source: §5.2 · p. 8, Figure 7 · p. 9
• On ContextBench's KV Store task, assisted and self-evolution improved over initialization and expanded the performance–cost Pareto frontier, with evolved skills strictly dominating the starting point.Assisted evolution used Qwen3.6-27B with Claude Fable 5.1 as proposer; self-evolution used Opus 5 for both roles.Source: §5.2 · p. 8, Figure 8 · p. 9
• After reinforcement learning, Qwen3.5-9B gained 13.7 points to 42.5% on held-out BrowseComp-Plus, matching the trained summary harness at 1.34 versus 2.19 PFLOPs per question.Adding the efficiency reward further reduced inference cost without a clear accuracy loss for either CLM or summary harness.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 (More) Efficient Serving with Suffix Cache Reuse

• Suffix Cache Reuse matched standard SGLang serving with 65.0% of its empirical prefix-reuse FLOPs on BrowseComp-Plus and also reduced re-prefilling after reasoning-token stripping.Figure 9 compares task accuracy and FLOPs per question, and decomposes server-side compute across turns and turns following context edits.Source: §5.3 · p. 9, Figure 9 · p. 9

6 Discussion and Future Work

• Editable context can become a channel for prompt injections or self-generated instructions to persist across turns, motivating work on attack surfaces and defenses that preserve context integrity.Prior work observed unauthorized instructions inserted into compaction summaries that subsequently affected task behavior.Source: §6 · p. 10
• Future directions include scaling CLM reinforcement learning and developing a pipeline to distill context-management strategies from harnesses into CLMs.The paper frames harness operations as context transformations that could be translated into CLM actions and internalized in model weights.Source: §6 · p. 10

A Extended Related Work

• Harness-scheduled systems prescribe when and how context updates occur, including threshold-triggered compaction or updating at every turn.Reinforcement learning can improve model behavior within these prescribed procedures without changing their schedule or procedure.Source: A Extended Related Work · p. 13
• Tool-based approaches let models select among predefined operations such as compaction, offloading, retrieval, or folding, while CLMs provide direct read-write access to live context through general-purpose tools.The compared tools differ in supported operations, and their underlying strategies remain predefined by their interfaces.Source: A Extended Related Work · p. 13
• Meta-optimization improves reusable context-management procedures through feedback, a goal shared by CLM skill evolution, which guides edits to live context.Related approaches optimize harnesses, memory procedures, context-engineering skills, agent programs, or prompts.Source: A Extended Related Work · p. 13
• RLMs expose long inputs for programmatic reading but not direct editing of the evolving interaction history; CLMs make live context read-write, so the approaches are complementary.RLM-style access can keep large inputs outside context until needed, while CLMs manage information brought into context and generated history.Source: A Extended Related Work · p. 13
• Suffix Cache Reuse retains cached states for unchanged text after a context edit, avoiding suffix recomputation; it is integrated into SGLang and extended to hybrid attention architectures.Related methods address cache reuse in prompt modules, retrieved documents, edited code, or evicted reasoning blocks.Source: A Extended Related Work · p. 14
• CLM stepwise GRPO assigns trajectory-level outcome advantage to each segment and rewards lower FLOPs only among successful trajectories, accounting for recomputation from context edits.The cited prior objectives instead emphasize response or trajectory length, while the CLM efficiency signal measures trajectory-level FLOPs under prefix caching.Source: A Extended Related Work · p. 14
• Context management controls what the model sees at each invocation, whereas external memory stores information beyond the current context; CLMs can complement memory by deciding how retrieved information enters or leaves context.The distinction depends on the information's role, not its storage format.Source: A Extended Related Work · p. 14

B Suffix Cache Reuse

• For full-attention layers, SCR diffs consecutive prompts, relocates up to K surviving spans, reuses their cached keys and values, and re-rotates keys to their new positions.Relocated entries use session-private cache slots; if those cannot be allocated, the server falls back to standard re-prefilling.Source: B Suffix Cache Reuse · p. 15
• For linear-attention layers, SCR snapshots the recurrent state before an edit and continues from it, while the edit is reflected in full-attention layers.The inserted text is not recomputed in the linear-attention layers, though its representation is recomputed in full-attention layers and can affect later linear-attention inputs.Source: B Suffix Cache Reuse · p. 15
• Capping relocations at K reuses the K longest surviving spans, limiting how many approximate pre-edit states are introduced in one edit.The approximations can compound when many spans are relocated before the model adapts on later turns.Source: B Suffix Cache Reuse · p. 15
• With K=6, the small-scale BrowseComp-Plus sensitivity study found performance robust across K values, with cache-reuse gains largely saturated and no observed performance degradation.The study used 64 questions with Qwen3.6-27B; K=6 was chosen as a conservative balance between reuse and approximation per edit.Source: B Suffix Cache Reuse · pp. 15–16, Figure 11 · p. 16
• SCR reuses 7.8% of all prompt tokens beyond prefix-cache hits on BrowseComp-Plus: 5.3 percentage points from reasoning stripping and 2.5 from other context edits.The reasoning-stripping benefit also applies to standard reasoning-model serving without model-driven context editing.Source: B Suffix Cache Reuse · p. 16
• SCR removes most re-prefilling of unchanged suffix tokens, but substantial unchanged-prefix prefill remains because of SGLang’s current hybrid-model caching behavior.Linear-attention recurrent states are stored only at cached request boundaries, so an interior divergence may force cache matching to fall back to a shorter prefix.Source: B Suffix Cache Reuse · pp. 16–17, Figure 13 · p. 17

C Prefix-Reuse FLOPs Computation

• Prefix-reuse FLOPs count trajectory-wide decoding, prefilling, and re-prefilling; at turn t, the server prefills U_t=P_t−R_t tokens after reusing R_t tokens of the P_t-token prompt.G_t is the number of generated tokens; an edit invalidates cached computation from the edited message onward under standard prefix caching.Source: C Prefix-Reuse FLOPs Computation · p. 17
• The per-turn cost accounts for token-processing and context-dependent attention computation, and trajectory cost sums the per-turn F_t values across turns.The attention terms distinguish prefill from decoding; the accounting omits embedding and output layers, recurrent-state updates, and several lower-order operations.Source: C Prefix-Reuse FLOPs Computation · p. 17
• 10.81 × 10^14 FLOPs is the cost of an illustrative turn with no reusable prefix, 7.7 times the append-only turn’s cost.For the same illustrative lengths, an append-only turn costs 1.41 × 10^14 FLOPs, while an edit leaving 10,000 reusable tokens costs 5.74 × 10^14 FLOPs.Source: C Prefix-Reuse FLOPs Computation · p. 18, Figure 14 · p. 18
• For Qwen3.6-27B, the illustrative calculation uses a 20,000-token prompt, 500 generated tokens, and reusable-prefix lengths of 18,000, 10,000, or zero tokens.The figure divides computed FLOPs by operation and shows additional computation that would be required without prefix caching.Source: Figure 14 · p. 18, C Prefix-Reuse FLOPs Computation · p. 18

D ContextBench

• ContextBench tasks grade what the agent kept, updated, or offloaded from inputs that generally exceed the context window, with answers held only in files not credited.Operations arrive as user messages before the agent acts, and the agent signals readiness for the next operation after handling each one.Source: D ContextBench · p. 18, Table 3 · p. 19
• Context pressure is total environment input over an episode divided by the 32,768-token context limit, so 1× marks where keeping everything fills the context.Task levels raise pressure by varying one generator setting at a time; levels below 1× test cases where keeping everything still fits.Source: D ContextBench · p. 18, Figure 15 · p. 19
• The pilot gives each method the same task instruction and a method-specific skill describing its tools, actions, verification, and common failure modes.The instructions aim to isolate context-management capability from task understanding or tool-use differences, with skills written at comparable detail.Source: D ContextBench · pp. 18–19

Other benchmark configurations

• TerminalBench 2.1 and TBLite use verifier-based scoring under token and turn limits, while math-optimization experiments use four problems and one run per method.TerminalBench covers 89 tasks; math optimization uses Claude 4.6 Sonnet and stops after 100 evaluated attempts or five hours.Source: E Experimental Configurations · p. 22

BrowseComp-Plus

• BrowseComp-Plus evaluates all 830 questions with a 23,560-token budget and 100 turns; an answer must be judged correct and complete, and unanswered questions count as wrong.The search tool returns ten 512-token snippets, and document reading is capped at 8,192 tokens.Source: E Experimental Configurations · p. 22

EdgeBench-10

• EdgeBench-10 uses 10 runnable tasks with three seeds, 12-hour network-disabled runs, 32,000-token budgets, and scores based on the better of the best submission or final repository grade.The Qwen3.6-27B subagent setup allows up to six concurrent subagents; reported cost includes their computation.Source: E Experimental Configurations · p. 22, Table 5 · p. 22

In-context steering

• In-context steering tests threshold, question-boundary, and backup instructions against no-instruction controls on BrowseComp-Plus with Claude 4.6 Sonnet.The three studies use different budgets and report compaction timing, boundary-aligned compaction, or the fraction of edits preceded by a full context copy.Source: E Experimental Configurations · p. 23

In-context evolution

• In-context evolution selects skills using a development split of 102 instances per task, after staged training-instance checks; the frozen archive is evaluated once on a 102-instance test split with three seeds.Four proposers work in parallel, and a skill enters the Pareto frontier if no earlier skill is both at least as accurate and at least as cheap.Source: E Experimental Configurations · p. 23

Reinforcement learning

• Reinforcement learning trains Qwen3.5-9B on 3,040 OpenResearcher prompts for 70 GRPO steps, then evaluates the selected checkpoint on all 830 BrowseComp-Plus questions.CLM uses a 28K context budget and up to 80 turns; its efficiency advantage is applied to context-management tokens.Source: E Experimental Configurations · pp. 23–24

TerminalBench 2.1, TBLite and BrowseComp-Plus with models of different sizes

• 59.4% accuracy on BrowseComp-Plus puts Qwen3.6-27B CLM highest, and CLM lies on the Pareto frontier across all three benchmarks.The comparison covers BrowseComp-Plus, TerminalBench 2.1, and TBLite.Source: Figure 17 · p. 24, F Supplementary Results · p. 24

Math optimization problems

• Figure 18 plots best-so-far scores against scored attempts for four open optimization problems; runs use Claude 4.6 Sonnet with a 32K context limit and stop after 100 attempts or five hours.Dots represent scored candidates and insets show final-score ranges.Source: F Supplementary Results · p. 24, Figure 18 · p. 25

EdgeBench-10 with a 128K context budget

• At 128K, CLM with subagents scores 50.2 using 219 prefix-reuse PFLOPs per trial, versus 47.3 using 142 for CLM and 47.8 using 222 for summarization.At 32K, the two CLM variants finish within 0.4 points of each other, at 44.2 and 44.6; context-managing methods improve over twelve hours while the base harness stops improving within two.Source: F Supplementary Results · pp. 24–25, Figure 19 · p. 25

In-context evolution

• Assisted evolution raises development accuracy on Needle Retention (97.6% to 100.0%), Sudoku Sketchpad (45.3% to 65.8%), KV Store (22.3% to 83.8%), and Log Triage (0.0% to 100.0%).The selected KV Store skill raises held-out test accuracy from 38.3% to 74.2%.Source: F Supplementary Results · p. 25, Figure 20 · p. 26
• In self-evolution, Opus 5 starts at 94%–100% accuracy, and evolved skills either lower cost at the same or higher accuracy or raise accuracy further.Figure 20 shows assisted evolution with Qwen3.6-27B and Claude Fable 5.1 proposing skills, and self-evolution with Opus 5 as both agent and proposer; skills use training rollouts, not the held-out evaluation set.Source: F Supplementary Results · p. 25, Figure 20 · p. 26

Reinforcement learning

• Figure 21 tracks BrowseComp-Plus accuracy and prefix-reuse PFLOPs during training for Qwen3.5-9B, comparing CLM and summary-harness runs with and without a FLOPs reward.The evaluation uses a 32K context limit and runs through training step 70.Source: F Supplementary Results · p. 25, Figure 21 · p. 26

• Without context-length awareness, agents may depend on frequent external interventions that can leave stale or redundant information in later turns after editing.Source: G Analysis on the Context Length… · p. 27
• Existing models show limited context-length awareness at long lengths; token-count hints improve estimation, with hints closer to the estimation point more effective.The diagnostic compares estimates with provider-reported prompt lengths for 50 inputs, with no hint or a token-count anchor at 25%, 50%, or 75% of the input.Source: G Analysis on the Context Length… · p. 27, Figure 22 · p. 27
• Models often predict recurring bucketed values at longer contexts; Claude-4.6-Sonnet tends to underestimate, while GPT-5.4 aligns most strongly with actual token counts among the three models.The paper conjectures that bucketed estimates may reflect token-counting patterns from pretraining or post-training.Source: G Analysis on the Context Length… · p. 27, Figure 22 · p. 27