Source-linked AI summary
Context as an Environment: Programmatic Context Management for Long-Horizon Agents
Yin Lin, Elaine Ang, Erkang Zhu, Bolin Ding, Jingren Zhou
TL;DR
Long-horizon agents must manage growing histories within bounded model contexts, while compression and external memory expose only lossy representations. Scroll instead makes context selection executable over a persistent Session Environment and outperforms the best reported system on BEAM10M by 5.1 points.
Problem
Long-running agent trajectories exceed bounded context windows, while compression and external memory can discard details needed later.
Method
Scroll preserves history in an append-only Event Log and uses a persistent sandboxed Python namespace, with model-written programs selecting and printing bounded context views.
Results
Scroll beats the best-performing reported system by 5.1 points on BEAM10M and is competitive with the strongest reported systems on LongMemEvalS.
Takeaways & Limitations
Context management becomes an explicit model policy over persistent state, with exec retrieving and computing context and print controlling what enters the next working view.
Takeaways & Limitations
The reported comparisons do not reproduce the baselines, because independent reproductions have repeatedly disagreed over evaluation setup.
Abstract
from arXiv · showhide
LLM agents increasingly take on long-running tasks whose history grows far beyond a single model context window. Existing approaches compress earlier interactions or extract selected information into fixed memory representations, committing to what to preserve before future needs are known. We present Scroll, a context manager that treats each agent session as an executable Session Environment. The environment is backed by an append-only Event Log and a sandboxed, persistent Python kernel. The kernel maintains a typed namespace across model calls, allowing tool outputs, retrieved history, and derived state to be bound to variables rather than serialized into the prompt at each call. Model-written code searches, materializes, and transforms session state through exec; only explicitly printed projections enter the model's working view for the next call. Context management thus becomes a programming task that inherits the improving coding abilities of LLMs, while the Event Log preserves lossless historical ground truth. As the working view approaches its budget, stale spans are evicted but remain recoverable: an eviction index keeps compact landmarks tied to exact Event Log addresses, so that the agent navigates directly to evicted regions instead of searching the full log. With Qwen3.8-Max as the backbone, Scroll achieves 94.8% on LongMemEval_S; 73.1% on BEAM_10M, surpassing the best published memory system by 5.1 points; and 86.7% on LOCA_256K, exceeding the best published long-horizon agent by 37.4 points.
1 Introduction
Long-horizon agents must manage growing histories under bounded and practically limited context windows. Scroll replaces lossy, advance-fixed compression with an executable environment that preserves history and lets the model retrieve, compute, and expose context at query time.
- Long-running agent trajectories accumulate interactions, tool results, failures, and revisions while each model call operates over a bounded context window.
- Existing compression and external-memory systems can permanently hide details because they decide what to preserve before future information needs are known.
- Scroll stores history in an append-only Event Log and maintains a typed namespace in a persistent Python kernel outside the model context.
- The model uses exec to search, retrieve, invoke tools, and compute over session state, while only explicit print output enters the next model context.
- Eviction bounds the working view without deleting records, and an address-anchored eviction index preserves direct recoverability of evicted events.
2 Scroll Context Manager
Scroll models context management as a programmatic choice of bounded working views over a persistent Session Environment. Its Event Log, durable payload storage, persistent namespace, exec/print interface, and recoverable eviction mechanism preserve history while supporting selective computation.
- Scroll treats an agent session as a persistent environment whose prompt contains only a bounded working view.
- 2.1 Problem Formulation: The formal state includes an event sequence, referenced payloads, and auxiliary derived state, while each model call consumes a view constrained by the context budget.
- 2.1 Problem Formulation: Unlike compression or extraction, Scroll defers view selection to query time through a model-written program operating on lossless session state.
- 2.3 Programmatic Context Construction: The model-facing interface factorizes context construction into location, materialization, computation, and exposure.
- 2.2 Persistent Session Environment: The Event Log preserves typed interactions with stable sequence addresses, while durable storage keeps large raw payloads recoverable through pointers and lazy handles.
- 2.2 Persistent Session Environment: A persistent sandboxed Python kernel retains resident values and lazy handles across calls, with provenance metadata linking derived objects to source events.
- 2.3 Programmatic Context Construction: Exec computes over resident and retrieved state, and only explicitly printed output crosses into the model-visible working view.
- 2.4 Recoverable Context Eviction: When the view exceeds budget, eviction protects active and recent content, removes older spans, and records recoverable landmarks in a tiered index.
3 Experimental Setup
Scroll is evaluated on long-horizon history retrieval and reasoning benchmarks and on an environment-based reasoning-and-acting benchmark. Experiments use standardized context-management rules, persistent internal state across memory sessions, and multiple backbone models.
- The evaluation covers interaction-history retrieval and reasoning alongside agentic reasoning and acting in environments whose state grows over time.
- Long-term memory retrieval and reasoning: LongMemEval tests evidence retrieval across conversational sessions, including scattered evidence, temporal dependencies, knowledge updates, and increasing distractor history.
- Long-context reasoning and acting: LOCA evaluates tool-using agents across environment descriptions scaling from 8K to 256K tokens, with experiments focused on the two largest regimes.
- The main experiments use Qwen3.8-Max, with Scroll implemented on QwenPaw and orchestrated in benchmark-provided environments.
- Scroll exposes context construction through exec tools that run Python in a persistent kernel, retain intermediate computation, and print only budgeted output.
- History recall searches and expands evicted records into Python objects that the model can filter, combine, or aggregate before printing.
- The benchmarks use one system prompt and shared context-management rules, with only task-specific rubrics or official task instructions added.
- Additional evaluations change only the foundation model across Qwen3.7-Max, Deepseek-v4-pro, GLM-5.2, Kimi-K2.7, and Qwen3.6-35B-A3B.
4 Results
Scroll outperforms or matches strong alternatives across long-term memory and long-context tasks, while ablations show that lossless records, a persistent kernel, and eviction indexing each contribute to performance. Its token usage remains a small fraction of the available corpus.
- Scroll beats the best-performing system by 5.1 points on BEAM10M.
- Long-context reasoning and acting: On LOCA, Scroll and CodeAct achieve the best performance and the smallest decrease as environment descriptions grow.The comparison uses Qwen3.8-Max with shared tools and differs only in context management.
- Backbone comparison: 86.7 versus 22.7 at LOCA_256K shows a 64-point spread between the strongest and weakest backbones using Scroll.All backbones benefit, but weaker models make more execution errors or terminate prematurely on longer, more complex trajectories.
- Ablations: 19.9 overall is the lossy variant’s score after discarding original records, with near-zero performance on tasks requiring exact historical values.
- Ablations: 7.3 points separates full Scroll from Scroll w/o REPL, while removing the eviction index costs 1.8 points overall.The REPL gap concentrates on composing evidence from many records; the index gap concentrates on scattered evidence found through keyword search.
- Cost and efficiency: 105K median input tokens on BEAM10M represent about 1% of the corpus, while output is an order of magnitude smaller across benchmarks.Figure 4 summarizes input tokens, output tokens, and agent turns; reported counts are not latency or dollar cost.
5 Related Work
Related work spans compression and external memory, executable code interfaces, and lossless history storage. Scroll combines these strands through a queryable Event Log, executable resident state, and indexed navigation over retained events.
- Context compression and external memory: Compression methods shorten active trajectories, while external-memory systems extract and retrieve facts, episodes, or notes.
- Code as the agent–environment interface: CodeAct and related systems use executable code or sandbox variables so selected tool-result projections, rather than all raw outputs, enter context.
- Lossless session history and navigation: Scroll combines a queryable append-only Event Log with external payload references and an executable resident namespace.
- Lossless session history and navigation: Its eviction index links tiered, sequence-addressed headlines to retained events and payloads, enabling programmatic recovery of located regions.
6 Conclusion
Scroll makes context management an explicit model policy over a persistent Session Environment. The model retrieves and computes with exec, selects context through print, and relies on the harness for deterministic storage, execution, and recovery.
- Scroll treats context management as an explicit model policy over a persistent Session Environment.
- The model uses exec to retrieve and compute over externalized state, while print selects what enters the next context.
- The harness provides deterministic storage, execution, and recovery for those model decisions.The paper proposes distilling retrieval and context-injection decisions from frontier-model traces into smaller models.
A.1 Per-Question-Type Results on LongMemEval
Scroll’s LongMemEval accuracy declines from S to M primarily on question types requiring evidence aggregation across more sessions. The agent-written code can locate a supporting session but may miss part of the needed evidence when distractors increase.
- 88.0% to 81.2%: multi-session accuracy drops from LongMemEvalS to LongMemEvalM.
- 100.0% to 83.3%: single-session preference accuracy declines from S to M.
- More irrelevant sessions cause the agent-written code to locate one supporting session while often missing part of the evidence.
A.2 Per-Category Results on BEAM
On BEAM10M, Scroll is strongest when answers require locating exact raw records and preserving their order or provenance. It underperforms ingestion-heavy baselines on tasks whose graded answer is already a condensed view over many records, while multi-session reasoning remains weak.
- Strengths: Scroll leads most clearly on knowledge update, contradiction resolution, and information extraction, which require locating and reconciling exact raw records.Scores are 92.5 versus 45.0–75.0, 88.1 versus 32.5–58.8, and 75.0 versus 51.2–66.3, respectively.
- Strengths: 92.5 versus 45.0–75.0: Scroll’s knowledge-update score exceeds the reported baseline range.
- Strengths: 88.1 versus 32.5–58.8: Scroll’s contradiction-resolution score exceeds the reported baseline range.
- Strengths: 75.0 versus 51.2–66.3: Scroll’s information-extraction score exceeds the reported baseline range.
- Limitations: Scroll underperforms the strongest baselines on summarization, preference following, and temporal reasoning.Scores are 70.5 versus 91.9 for Exabase M-1, 89.1 versus 97.5 for Hindsight, and 47.5 versus 58.8 for Exabase M-1.
- Limitations: Multi-session reasoning is the weakest category for every system, with scores ranging from 9.6 to 26.1.Scroll’s misses stem from over-precise filters that undercount the evidence set.
B Additional LOCA Results
The LOCA comparison is a system-level comparison against prior published results using different backbone models. The appendix illustrates both Scroll’s strongest and weaker BEAM10M categories through logged trajectories and its executable retrieval workflow.
- Table 7 compares Scroll with published LOCA systems using different backbone models, so the comparison is system-level rather than controlled.
- The benchmark instructions are used unmodified for LOCA, with only environment metadata added.
- The BEAM10M appendix presents two successes and two failures from categories where Scroll is strongest or trails the best published systems.The examples include knowledge update, contradiction resolution, preference following, and summarization.
D.1 Success (knowledge update): resolving a superseded value by Event Log order
A successful knowledge-update trajectory searches for the relevant session, materializes its records in Event Log order, and filters cached rows to identify the latest user-provided values. It resolves the update from 14 tasks and 85% to 17 tasks and 88%.
- One coarse search identifies Session 42 among decoy sprints, with both competing values already surfaced in the hits.
- 14 tasks and 85% precede 17 tasks and 88% in Session 42, establishing that the later values supersede the earlier ones.
- The submitted answer reports 17 Jira tasks and an 88% sprint-completion target for 2024-11-05.
- The trajectory materializes Session 42 in address order and then filters the cached rows in the kernel without rereading the Event Log.
D.2 Success (contradiction resolution): retrieving both sides of a conflict and ordering them
The successful contradiction-resolution trajectory retrieves evidence for both sides of Pamela’s conflicting history, verifies provenance, and orders the events to produce a nuanced resolution.
- Success: 1.0 judge score was achieved on all four contradiction-resolution rubric criteria.The probe required identifying the conflict, citing the correction and earlier claim, and resolving which statement was correct.
- Retrieving both sides: The trajectory searched separately for affirmative and negated claims, then materialized and checked their provenance.It used a claim-side query, a negation-term query, and subsequent retrieval steps to distinguish user-authored evidence from assistant-generated text.
- Ordering the conflict: The record showed earlier plans for Pamela to coordinate vendors and brief entertainers, followed by the user’s Session 37 correction that this was a misunderstanding.The earlier claim concerned setup-time coordination; the correction denied that Pamela briefed entertainers or saved coordination time during setup.
- Resolution: A later Session 73 confirmation narrowed Pamela’s contribution to calling 3 vendors for ratings and saving about 15 minutes of follow-up time.The user confirmed this post-event contribution, distinguishing it from the corrected setup-time claim.
- Resolution: The final resolution retained the correction while preserving the narrower later confirmation: no setup-time coordination, but post-event vendor feedback calls.The trajectory therefore separated two different time-saving claims rather than treating the later evidence as a reversal of the correction.
D.3 Failure (preference following): competent retrieval on the wrong axis
The preference-following failure was not caused by an inability to execute retrieval patterns, but by framing the task around mapping tools instead of the user’s toll-avoidance preference.
- Failure: 0.0 judge score was assigned on all three preference-following rubric criteria.The answer retrieved mapping-tool evidence but omitted the graded toll-avoidance preference.
- Missed evidence: The relevant toll preference was present in the Event Log, including a route question about avoiding tolls and a hybrid route adopted because it saved $15 in tolls.The failure occurred before retrieval ran: query formulation selected the wrong semantic axis.
- Interpretation: The failure shows that successful search execution cannot compensate when the probe’s question framing omits the criterion being graded.The trajectory performed multi-query sweeps, provenance checks, recency checks, and expansions, but all remained tool-framed.
- Wrong axis: All six initial queries named mapping tools, while none included toll, fee, avoid, or cost.The probe was framed as identifying a preferred mapping tool, so retrieval never targeted the route-constraint axis.
- Retrieval outcome: The trajectory repeatedly retrieved evidence that the user used Google Maps and Roadtrippers together.Those results were relevant to tool preference but not to the rubric’s stated preference for avoiding tolls.
D.4 Failure (summarization): positional sampling misses mid-session evidence
The summarization failure arose after complete session-level coverage: positional and narrowly thematic sampling skipped mid-session evidence containing the graded habits and driving-limit facts.
- Failure: 0.42 judge score satisfied two of six rubric criteria fully and one partially.The missed criteria concerned facts that were present in the materialized sessions but outside the sampled positions.
- Coverage: 4,448 events across Sessions 61–80 were merged with gap-free session-level coverage.The trajectory paged through the date range after SQL reached its 1000-row cap.
- Positional sampling: Head-and-tail sampling skipped mid-session facts in long mixed-topic sessions, including the 3-hour driving cap and sleep and hydration goals.The graded facts were located in Session 77 and Session 66 but never entered the sampled working material.
- Thematic sampling: The trajectory’s thematic search covered growth vocabulary but never queried driving hours, fatigue, sleep, or hydration.Consequently, it surfaced growth-related material while missing the habit dimensions required by the probe.
- Answer consequences: The resulting answer reported travel phases, growth quantifications, and trip lessons while omitting the graded driving, sleep, and hydration evidence.The answer included 40% more patience, 35% more flexibility, an 82% self-rating, and four trip lessons.