Source-linked AI summary
ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents
Peng Xu, Zuyu Zhang, Yuze Sun, Feng Tian, Long Wang, Chen Zhang
TL;DR
Long-horizon LLM agents must assemble prompts under hard context-window and byte-sensitive caching constraints, while production handling is often scattered across local components. ContextPipe treats this as database-style query execution through a structured catalog, five-phase pipeline, deterministic optimizer, and EXPLAIN ANALYZE trace. In a preliminary SWE-bench Pro Qutebrowser evaluation, it reduced context volume, LLM calls, and response time versus Flat, at the cost of lower cache-hit ratio.
Problem
Long-horizon agents must repeatedly assemble bounded prompts from diverse context sources while managing latency, cost, and byte-stable prefix caching, but production logic is scattered.
Method
ContextPipe uses a five-phase pipeline, structured lifecycle catalog, deterministic cache-aware optimizer, and EXPLAIN ANALYZE trace for context construction.
Results
30% fewer total tokens, 23.1% fewer LLM calls, and 8.7% lower response time were reported versus Flat, with lower cache-hit ratio.
Takeaways & Limitations
ContextPipe makes prompt assembly auditable, replayable, and failure-isolated while trading lower context volume and calls against cache-hit ratio and cost dependence on cached-token pricing.
Takeaways & Limitations
The Structured-vs-Flat comparison is the first planned mechanism ablation, and future work targets coarse compaction and fixed cache-marker placement.
Abstract
from arXiv · showhide
Long-horizon large language model (LLM) agents require context assembly: the runtime must decide what to include in each prompt, in what order, and when to compact history under a hard context-window budget and a byte-sensitive prompt cache. In production agentic systems, this logic is scattered across prompt builders, ad hoc compaction routines, cache-break workarounds, and per-provider shims. We argue that context assembly is structurally isomorphic to query execution in a relational database: both execute under a hard budget, exploit a tiered cache, and leverage statistics. We adopt this discipline in ContextPipe: a five-phase pipeline (Plan Bind Optimize Execute Feedback) backed by a structured data-source catalog, a deterministic cache-aware optimizer, and an EXPLAIN ANALYZE trace. We show that context in ContextPipe is auditable, replayable, and failure-isolated. A preliminary evaluation using the SWE-bench Pro Qutebrowser subset shows that, compared with the append-only context construction policy, ContextPipe reduces total token volume by 31%, LLM calls by 23%, and response time by 9%, at the cost of a lower KV cache-hit ratio.
1 Introduction
Long-horizon LLM agents must repeatedly assemble prompts from diverse sources under context-window, latency, cost, and byte-stable caching constraints. ContextPipe applies database-style planning and optimization to make these decisions structured and auditable.
- Agent turns assemble system prompts, tool schemas, history, memory, skills, and runtime identity into one bounded API request.
- Raw concatenation fails when histories or tool results dominate, requiring repeated decisions about retention, summarization, and deferral.
- Production systems distribute context handling across builders, compaction routines, cache markers, and attachment injection, making decisions collectively opaque.
- Database optimizers provide the motivating analogue: declarative intent, catalogs, hard resource limits, tiered caches, and EXPLAIN ANALYZE reasoning.
- ContextPipe introduces five explicit phases, a deterministic cache-aware optimizer, a structured source catalog, and mechanisms for adapting across turns.
2 The ContextPipe Runtime
ContextPipe routes each LLM turn through a single staged pipeline that separates planning, I/O, optimization, execution, and feedback. Typed artifacts, deterministic transformations, provider policies, and recorded statistics support testing, replay, and auditability.
- Each turn follows Plan → Bind → Optimize → Execute → Feedback, with defined input, output, and purity constraints.
- Plan: Plan computes pressure and a section manifest from a catalog snapshot, including sections, cache scopes, token budgets, and cache strategy.
- Plan: Predictive pressure adds response, thinking, and schema reserves, using typed estimates to select compaction conservatively for anticipated demand.
- Bind: Bind concurrently fetches planned content and emits typed artifacts that preserve provider invariants such as tool-call pairing and thinking blocks.
- Optimize: Optimize aligns, compacts, and spills bound artifacts under gated limits, recording blocked transformations and producing a serializable request and trace.
- Execute and Feedback: Execute can return an EXPLAIN-only trace without contacting the model, while Feedback records token, cache, truncation, and cache-break statistics for later planning.
3 Context Optimization
ContextPipe formulates context optimization as a constrained placement problem but deliberately replaces intractable utility-maximizing search with a deterministic, auditable policy. Context pressure and predictive reserves determine how aggressively content is compacted.
- The optimizer orders sections, chooses which survive compaction, and places cache markers under token, precedence, protection, and tool-pairing constraints.
- Never-priority sections, including Identity and Constraints, are protected from compression.
- The general optimization problem is a precedence-constrained knapsack with additional order-sensitive cache effects, so ContextPipe does not search it.
- Context pressure is the quantity driving compaction, with predictive pressure incorporating reserves for expected response, thinking, and schema growth.
- Predictive pressure lets Plan compact before a likely prompt-too-long error rather than after it occurs.
3.3 Tier-Gated Compaction
Tier-gated compaction selects increasingly aggressive transforms from pressure while preserving deterministic cache alignment. Explicit gates, breakers, protected content, and trace records make applied and skipped decisions observable.
- Predictive pressure selects a discrete compaction tier using fixed thresholds, including AggressivePrune at 0.90 ≤P.
- The optimizer takes the more aggressive raw-or-predictive tier, and RecoveryState can escalate it further without relaxation.
- Normal performs no compaction; TrimSchemas prunes unlikely schemas; CompactHistory clears oldest tool results; AggressivePrune also drops oldest rounds.
- Compaction is bounded by gates, a clear-token circuit breaker, spill behavior, a four-round minimum before dropping, and protection for system messages.
- Reorderable sections are stably sorted by ascending volatility while Identity and Constraints remain anchored.
- Cache markers follow scope boundaries so stable prefixes remain byte-identical, while provider policies and marker caps constrain placement.
3.5 Spill
Spill preserves oversized context through stable references while protecting core sections and isolating backend failures. The optimizer’s recorded decisions support auditability, replay determinism, and failure-safe execution.
- 3.5 Spill: 10,000 tokens is the spill threshold for replacing oversized sections with lightweight SpillReferences when a backend is configured.Identity, Constraints, and Working Memory are never spilled.
- 3.5 Spill: Spill rehydration is fail-open: backend failure yields a placeholder and a skippedrehydration trace entry rather than aborting the turn.
- 3.5 Spill: Identity and Constraints retain leading positions, and reordering never crosses cache-scope class boundaries.
- 3.5 Spill: Every serialized tool result remains paired with its tool call, while cleared oversized results retain recoverable references.
- 3.5 Spill: Every applied or skipped optimizer transformation is logged, making the final ordering, mask, and markers reconstructable from the trace and inputs.
- 3.5 Spill: Replay determinism follows from total ordering, pure tier selection, deterministic marker placement, immutable statistics, and deterministic serialization.
- 3.5 Spill: Failed execution leaves persistent state unchanged except for an explicit failure record, preventing invalid usage samples and unsafe next-turn context.
- 3.5 Spill: These properties are implementation specifications whose value includes auditability, replay, retry safety, and stronger determinism than utility-search optimization.
3.8 The Statistics Subsystem
PipelineStats provides isolated, snapshot-based statistics for planning reserves, cache behavior, and section budgets. Failed turns do not contaminate these statistics, while stage-specific state supports serializable execution.
- 3.8 The Statistics Subsystem: PipelineStats is read-only during Plan and write-only during Feedback, with no read–write interleaving within a turn.
- 3.8 The Statistics Subsystem: p75 reserves are used in steady state, p95 reserves during recovery, and empty buckets return a fixed 500-token floor.
- 3.8 The Statistics Subsystem: Reserve buckets are keyed by model and query source, using capped PercentileDigests that evict medians after 512 entries to preserve tail quantiles.
- 3.8 The Statistics Subsystem: Cache-hit ratio and per-section token usage use separate exponential moving averages to guide future cache assessment and budget allocation.
- 3.8 The Statistics Subsystem: Failed executions add only a failure record, while concurrent Bind reads an immutable snapshot and Feedback mutates statistics after execution.
4 ContextSources: The Pipeline Catalog
ContextSources turns an accumulated, opaque runtime state into a queryable catalog that separates source lifecycle, location, and bind cost. Dedicated latches and emergent-context safeguards make cache behavior and deferred discoveries explicit.
- 4 ContextSources: The Pipeline Catalog: A flat runtime struct of roughly 200 mixed-lifecycle fields prevents Plan from enumerating sources or deciding what to fetch.
- 4 ContextSources: The Pipeline Catalog: ContextSources classifies each source by lifecycle, location, and bind cost, allowing Plan to select stable tiers before Bind performs I/O.
- 4 ContextSources: The Pipeline Catalog: SessionLatches evaluate lazily once and then freeze for the session, covering beta headers, cache-scope eligibility, and provider feature flags.Freezing prevents mid-session changes that would invalidate the KV-cache prefix.
- 4 ContextSources: The Pipeline Catalog: EmergentContext queues tool- and auxiliary-generated discoveries after Plan for consumption at the next turn’s Bind stage.
- 4 ContextSources: The Pipeline Catalog: Emergent items use a per-turn TTL, content hashes, and per-list caps to prevent stale attachments from being silently double-injected during replay or resume.
- 4 ContextSources: The Pipeline Catalog: Runtime catalog schemas let operators inspect what the pipeline would do without waiting for a user turn, redeploying, or attaching a debugger.
- 4 ContextSources: The Pipeline Catalog: Disjoint stage-specific state makes each pipeline stage unit-testable from synthetic reserves and section lists without a live provider.
5 Explain Analyze
ContextPipe’s EXPLAIN ANALYZE trace joins pre-execution planning with post-execution cache feedback for each turn. It serves as both an operational audit log and a basis for detecting regressions and recovery conditions.
- 5 Explain Analyze: Figure 2 joins the pre-execution EXPLAIN trace with post-execution ANALYZE feedback and identifies Global, Session, and unscoped regions.
- 5 Explain Analyze: EXPLAIN records pressure, tier selection, token plans, compaction decisions, and cache-marker positions before execution.
- 5 Explain Analyze: ANALYZE records actual input and output tokens, cache reads and creation tokens, and deltas from planned estimates after execution.
- 5 Explain Analyze: The trace logs optimizer decisions, skipped transformations, cache markers, emergent injections, and recovery escalations on the turn they occur.
- 5 Explain Analyze: Eight alert rules detect cache breaks, cold starts, regressions, predictive misses, compaction cascades, recovery loops, pressure spikes, and emergent-list overflow.
6 Implementation
ContextPipe grounds context assembly in typed pipeline data and stage contracts, while ForkPrefix addresses cache sharing between parent and child agents. The implementation also provides shadow rollout and evaluates structured context optimization against Flat on SWE-bench Pro Qutebrowser.
- Components: The runtime separates pipeline data types from orchestration for live state, I/O, tool dispatch, and feedback into subsequent invocations.Pipeline data includes pressure, reserves, catalog snapshots, cache policy, trace records, and typed artifacts.
- Typed artifacts: Typed artifacts preserve section metadata, message roles and tool-call identifiers, schema hashes, and spill references before one-time serialization at Execute.This design keeps provider-wire serialization at a single execution boundary.
- ForkPrefix: ForkPrefix captures a frozen, byte-identical parent prefix with canonical bytes, per-tool schema hashes, cache-key inputs, and an authoritative SHA-256 drift record.A first-turn probe compares child cache reads with the parent estimate and emits an audit event per spawn.
- ForkPrefix: The designed snapshot is immutable, validates cache identity before sharing, and supports skip-cache-write mode for auxiliary forked children.These properties hold once an executor consumes the snapshot; the default server executor does not yet wire in full child-side consumption.
- Evaluation: Table 4 compares Flat and Structured on SWE-bench Pro Qutebrowser using DeepSeek-V4-Pro across 3 instances and 3 repeats per condition.The caption reports interleaved matched pairs and per-cell means, with changes defined as (Flat − Structured)/Flat.
- Shadow-pipeline rollout: Mandatory shadow mode compares hashes, roles, cache markers, and token estimates through shadow-only, verification, flip, and retirement rollout stages.The new path initially runs beside the active path without dispatching requests.
7 Evaluation
ContextPipe is preliminarily evaluated against a fully gated-off Flat policy on three Qutebrowser instances, showing lower context volume, calls, completion tokens, and response time, but worse cache-hit ratios and conditional cost advantages.
- Workload: 3 of 79 Qutebrowser instances were evaluated preliminarily, with each instance run three times.The workload was selected to stress the context window, prompt cache, and compaction ladder simultaneously.
- Results: 30% lower total tokens, 39% fewer repeated cache reads, 23.5% fewer completion tokens, 23.1% fewer LLM calls, and 8.7% lower response time versus Flat.Structured is the full ContextPipe pipeline, whereas Flat disables all optimizer gates while preserving binding order.
- Cache trade-off: 86.3% median per-cell cache-hit versus 95.6% for Flat, while fresh uncached input tokens doubled.The lower cache-hit ratio results from compaction changing the sent prefix.
- Cost: r*=0.145 is the billed-cost break-even cached-token price ratio; at r≈0.1, Flat costs approximately 11% less despite sending 43% more total context.Structured has lower billed cost when cached tokens cost more than 14.5% of fresh tokens, including −13% at r=0.25.
- Diagnostics: The Structured trace attributes cache dips to specific optimizer decisions and shows rapid recovery after schema trimming and history compaction.TrimSchemas produces an 8% one-call hit followed by 93–100% hits, while CompactHistory produces 11% followed by 99% on the next call.
- Limitations: Mechanism ablations A1–A4 remain open, and the experiments cover only 3 of 79 instances.The authors identify limited coverage and incomplete mechanism ablations as threats to validity.
8 Related Work
ContextPipe addresses the composition of context under pricing and window constraints, complementing work on retrieval, memory, caching, agent orchestration, and query optimization. Its database-style abstraction combines these concerns through an auditable, heuristic, feedback-driven execution pipeline.
- Scope: Prior work addresses retrieval, long-context modeling, caching, serving, agent loops, or orchestration, but not their direct composition during each context-assembly turn.ContextPipe composes selected content onto a priced substrate under a hard window budget.
- Agent memory and compaction: Unlike MemGPT’s two-tier memory, ContextPipe catalogs eight lifecycle tiers and incorporates predictive pressure, cache alignment, and provider policy into planning.MemGPT pages between main context and an external store at token thresholds.
- Agent memory and compaction: ContextPipe and PEEK both structure state beyond flat transcript concatenation, but PEEK uses an operating-systems context map while ContextPipe uses database-style execution.The differing abstractions organize long-horizon agent state in distinct ways.
- Prompt caching and serving: ContextPipe chooses byte layout, breakpoint placement, and placeholder shaping, parameterized by ProviderCachePolicy rather than provider-specific glue.Serving systems provide the mechanisms that price a byte layout; ContextPipe determines the layout.
- Query optimization: Unlike Selinger-style cost optimization, ContextPipe uses heuristic, feedback-driven decisions because information value per token cannot be precisely quantified.The pipeline remains auditable, reversible, and gated while borrowing staged planning, statistics, buffer pools, and EXPLAIN.
9 Conclusion and Future Work
ContextPipe frames long-horizon context construction as query execution, using explicit planning, optimization, execution, feedback, and trace recording to make assembly auditable, replayable, and failure-isolated. Future work targets the measured token/cache tradeoff, broader evaluation, and remaining mechanism ablations.
- ContextPipe constructs context as a query execution problem with a five-phase pipeline, lifecycle-indexed catalog, deterministic cache-aware optimizer, explicit compaction gates, and EXPLAIN ANALYZE tracking.
- The design makes prompt assembly auditable, replayable, and failure-isolated.
- Future work addresses the measured total-token versus fresh-token/cache-hit-ratio tradeoff through evaluation on more benchmark cases.
- Finer-grained compaction would replace whole-chunk threshold escalations with smaller earlier increments, producing smaller cache breaks.
- Adaptive cache-marker placement would use per-boundary hit/miss history to reduce the prefix fraction resent after compaction invalidates upstream markers.
- Larger task panels from additional benchmarks and repositories would test whether the measured effects hold beyond the SWE-bench Pro Qutebrowser codebase.