PaperlayerView exact revision on arXiv

Source-linked AI summary

Parsing the Stream: A Live Trace Model for Long-Horizon Agents and Their Observers

Egor Pakhomov, Erik Nijkamp

arXiv:2609.01466v1cs.AI

TL;DR

Long-horizon traces outgrow both human observers and agents with bounded context, while separate systems serve these consumers from the same stream. The paper introduces a live trace model that folds events into typed state and compiles per-consumer views, finding that compact deterministic state supports monitoring and accumulation tasks where growing contexts fail, subject to task-schema coverage and benchmark limitations.

  • Problem

    Long-horizon agent traces underserved both human observers monitoring runs and agents that must fold history into bounded context, while separate systems serve these consumers.

  • Method

    The paper uses an append-only typed event ledger, a deterministic single-pass fold into RunState, versioned derived nodes, and per-consumer observer and worker views compiled from the same state.

  • Results

    Across the evaluated task families, success tracked maintaining the required statistic in bounded state: deterministic folding supported monitoring and accumulation tasks where raw or growing contexts failed, while cheaper alternatives could match accuracy in some settings.

  • Takeaways & Limitations

    The fold's transferable value is conditional on schema coverage; beyond comparable accuracy, it provides deterministic auditability and serves the observer from the same state.

  • Takeaways & Limitations

    The benchmark co-evolved with the system and rewards the statistics tracked by the fold, so the reported advantage is operation-conditional and requires evaluation on externally authored tasks.

Abstract

from arXiv · show

A long-horizon agent's trace outgrows both of its consumers: the human observer monitoring the run, and the agent itself, whose bounded context the trace must be folded back into. We present a live trace model, an append-only event ledger folded incrementally into typed run state and compiled into per-consumer views, and evaluate it for both consumers against deterministic ground truth. For the observer side, evaluated with an LLM reader as proxy, the compiled view answers monitoring questions using approximately 14x and 15x fewer input tokens (by reader) and at 5-7x lower cost than a budget-capped single-call reading of the raw trace, with higher accuracy (0.85-0.87 versus 0.48). Because the questions were co-designed with the view schema, we treat the token and cost reduction, conditional on schema coverage, as the transferable result. For the agent, on 120-link sequential-dependency tasks, mechanisms that maintain the task's running statistic in per-step state succeed where full-context prompting fails (30/30 versus 8/30 under a clean protocol, n=30, labeled descriptive owing to benchmark-system co-development); a prompt-level scratchpad matches the fold's accuracy at lower cost, and a two-arm decomposition attributes the fold's accuracy to its deterministic aggregate and its cost advantage to its compactness. The fold's remaining value over cheaper alternatives is deterministic auditability and serving the observer from the same state. We derive eleven candidate requirements for trace folding from observed failures and delimit them with an order-sensitive task family on which the fold ceases to help. Code, benchmarks, a regenerable synthetic corpus, and all workbench traces are released.

1 Introduction

The paper addresses how a single long-horizon trace can serve both human observers and bounded-context agents by folding events into shared state and compiling consumer-specific views.

  • Motivation: A raw trace underserved both consumers: observers faced 0.479 accuracy over 779K tokens, while full-context workers reached 7/30 success at 120 links.The observer result used a frontier-tier reader on raw tails; the worker’s cumulative billed input reached 2.37M tokens.
  • Measured claim: A deterministic bounded fold shows no observed accuracy disadvantage against retaining full history alongside the harness-computed aggregate, at roughly one-quarter the cost.The comparison is limited to accumulation tasks and tested sample sizes; the same state also provides an auditable observer view.
  • Approach: The model parses an append-only stream into a typed ledger, folds it into RunState, materializes versioned nodes, and compiles per-consumer views.The state includes tools, files, turns, costs, facts, and deterministic aggregates; a curator refreshes the worker view from its recorded trace.
  • Requirements: The architecture deduplicates repeated usage, prevents terminal validity states from reopening, and supports hindsight re-parsing.These design rules are each traced to measured failures.
  • Evaluation scope: The paper derives eleven fold requirements from live failures and tests their boundary on an order-sensitive task family where the aggregates cease to help.It also introduces COMPREHEND and CONTINUE as auditable, regenerable evaluation instruments.

2 Related Work

Related work spans context compression, agent memory, observability, event-sourced architectures, and typed agent models, while this paper combines these foundations around a live run substrate.

  • Crowded areas: Context compression, retrieval, and observability are established areas, but prior systems respectively address histories, cross-session memory, or transport-level spans rather than a semantic live-run model.The paper positions its contribution at their intersection.
  • Adjacent foundations: The layer stack applies event sourcing and CQRS to agent runs, using incremental materialized views as the streaming analogue.These foundations motivate compiling multiple views from one event-derived state.
  • Closest neighbors: Closest neighbors differ in consumer or scope: VISTA models agent working memory, ESAA targets project governance, and PROJECTMEM projects cross-session summaries.LangGraph establishes a single typed event-stream pattern, while the present model targets live observability and worker context together.
  • Trace substrate: The adapter emits typed events per content block, preserves causal and cache-aware fields, and fingerprints content-bearing payloads with SHA-256.Payloads are stored by reference while bounded excerpts remain tamper-evident.
  • Ingestion: Under append-at-tail discipline, byte offsets provide sufficient resume tokens, while deduplication prevents up to 3.49× token-accounting inflation.The parser handled 104 MB across ten large transcripts in 0.4 seconds, excluding optional extraction and view materialization.
  • State and views: RunState preserves repeated observations through occurrence identity and aggregate-preserving eviction, while derived nodes use versioned validity and hindsight re-parsing.Observer and worker views compile from the same state, with the worker view refreshed from its own trace.
  • Failure-driven design: Development-era renderer defects violated the no-silent-truncation requirement, motivating corrected header wording and coverage stamps.The ambiguity concerned a partially evicted store and an extraction count showing only in-view facts.
  • Scope of determinism: Determinism applies to the non-extractor path; optional LLM extraction is memoized and provenance-carrying but not reproducible across cache loss or model retirement.This is a declared exception to layer determinism.

4 Instruments

The evaluation uses deterministic ground truth and separates observer comprehension, implementation fidelity, pathology detection, and agent continuation under controlled conditions.

  • Evaluation protocol: All scores use deterministic ground truth rather than LLM judges, with answers graded mechanically and agent outcomes verified against defined task state.Ground truth is constructed, independently computed, or derived from the ledger.
  • DETECT: P = 1.0/R = 1.0 with zero false positives was achieved by rule-based detectors on 48 labeled pathology instances across 30 synthetic traces.This establishes implementation consistency, not real-world validity.
  • FIDELITY: 8/8 bookkeeping fields matched a mutation-tested independent oracle, establishing fidelity of the incremental fold.The extended fact and aggregate recount supports the machinery used for agent-side results.
  • CONTINUE: CONTINUE evaluates a shared worker model across scatter, fix, chain, prose-chain, and altchain families with matched seeds and pre-committed call-index error schedules.Thinking mode was disabled, and the schedule is tied to call index rather than semantic operation.

5 Results

The results show that schema-aware trace views improve long-horizon observation and support sequential-dependency tasks, while exposing trade-offs, protocol boundaries, and residual worker-facing ambiguity.

  • 5.1 The observer leg: 0.86 versus 0.51 for Sonnet 5 and 0.81 versus 0.46 for Haiku 4.5: compiled views outperform raw traces on observer accuracy.Transcript-level bootstrap intervals are disjoint for both readers.
  • 5.1 The observer leg: The compiled view costs 5–7× less than same-reader raw reading, with the smaller reader on the view scoring 0.850 at $0.08 versus 0.479 at $2.37 for the frontier reader on raw tails.The paper identifies the same-reader ratio as the appropriate summary because the approximately 30× cross-tier ratio conflates model pricing with representation.
  • 5.1 The observer leg: The view’s advantage concentrates on whole-run aggregation questions that capped raw reading cannot retain, while recency-based questions remain comparatively condition-insensitive.Macro-averaged accuracy is 0.850 versus 0.385 for Sonnet 5 and 0.815 versus 0.382 for Haiku 4.5 on the remaining five question types.
  • 5.2 The agent leg: the crossover and its controls: The fold is correct in 30/30 runs, but five worker misses arose when workers added a delta already included in the rendered aggregate; coverage stamps recovered 5/5 failing seeds.The resulting design implication is to make aggregate coverage explicit to the worker.
  • 5.3 Eleven requirements, and their boundary: The reported comparisons remain descriptive because the development process was adaptive, and the frozen headline and post-hoc fix cells were not pooled.The requirements support no stronger claim than that each violation measurably broke continuation or corrupted state in this setting.

6 Discussion

The results suggest that maintaining task-relevant state deterministically can shift comprehension work from downstream consumers, but the parser introduces its own measured failure axes and the claims remain bounded by evaluation scope.

  • Success tracked the requirements rather than the specific mechanism: bounded state maintained the needed statistic, whereas uncapped summarization degraded to 3/10 at 120 links.Self-managed notes, masked hybrids, calculator tools, and the fold succeeded when they preserved the required statistic in bounded form.
  • Parsing the stream once moves comprehension work out of downstream consumers, with incremental deterministic cost replacing repeated billing for growing contexts.The paper observes this pattern in both the observer and agent evaluations, where compact representations sustain tasks that growing contexts do not.
  • Extractor systems have independent failure axes, including schema drift, refusals, and batch nondeterminism, which must be measured rather than assumed away.Availability depends on content and model, and a safety-filtered batch pipeline inherits an availability distribution that it has not measured.
  • The transferable claims are limited to the two-consumer framing, deterministic-ground-truth instruments, failure-derived requirements, and the boundary delimiting applicability.The system was evaluated at modest sample sizes on a single vendor’s stack.

7 Limitations •

The evaluation is constrained by sample size, benchmark–system co-evolution, design choices, single-vendor coverage, unexamined trust boundaries, and untested session and retrieval settings.

  • Only the 120-link full-versus-curated comparisons and accompanying n = 30 scratchpad cells are adequately powered; other results are directional.The paper notes that n = 3 overstated the 60-link collapse, reporting 0/3 versus 6/10.
  • The chain family rewards the statistics tracked by the fold, and the requirements were developed against these tasks and seeds.Held-out seeds are consistent so far, but the altchain boundary shows the advantage is operation-conditional; externally authored tasks remain the decisive test.
  • The curated arm bundles selection with computation, although controls and a two-arm decomposition separate boundedness from aggregate computation on tested operations.The decomposition reports view+full-history at 10/10 and view-minus-aggregates at 4/10; operations not precomputed by the fold remain untested.
  • COMPREHEND uses schema-co-designed questions, capped-tail baselines, development exposure, an LLM observer proxy, and an adapter-derived 99.95% tool-call correlation.Synthetic replication improves auditability but does not remove these design limits.
  • The entire stack uses one vendor, including workers, readers, extractors, pricing, caching, and refusal behavior.
  • Trust-boundary issues are unaddressed: trace-derived tool outputs are fed back to the worker without analyzing prompt injection, provenance policy, or secret redaction.Verbatim validation establishes provenance, not safety.
  • Multi-session and multi-agent ledgers remain untested, and retrieval is excluded from chain tasks by construction.A task family where retrieval is available to all arms yet insufficient alone is still needed.

8 Artifacts

The release includes the implementation, benchmark harnesses, synthetic-corpus generator, workbench traces, scoreboards, spend ledger, and regression tests, while real transcripts remain withheld.

  • The release contains tracelab, four benchmark harnesses, the workbench with task families and pre-committed schedules, and every synthetic workbench run trace.
  • The package also includes a seeded COMPREHEND corpus generator, a scoreboard of variants and failures, the spend ledger, and 99 regression tests.
  • The synthetic corpus regenerates byte-identically from code, while rescoring requires the named model endpoints; the twelve real transcripts are withheld as personal working sessions.The instrument can be rerun on a reader’s own transcripts.

A The development ladder (how the requirements were found)

The eleven requirements emerged from failures during sequential development, with fixes tested by regression checks; the ladder exposed state, aggregation, extraction, and validation defects.

  • The development ladder: The requirements were extracted from sequential-development failures, and each fix was pinned by a regression test before development continued.
  • The development ladder: The initial structure-only view scored 0.20 on scatter because referenced file contents were absent; content pinning recovered 1.000 on paired seeds.
  • The development ladder: The newest-wins fact store scored 0.000 on chains because thirty same-key deltas collapsed into one, exposing the need to preserve repeated updates.
  • The development ladder: Deterministic aggregates reached 5/5 at approximately 60% of the full arm’s cost after end-stage arithmetic failures over 30 values.
  • The development ladder: At 120 links, cap eviction dropped early values and re-reading after folding double-counted by exactly +65, motivating aggregate-preserving eviction and reread safeguards.
  • The development ladder: Feeding established key vocabulary raised the small extractor from 0.667 to 1.000 at unchanged cost, while bisection and fallback recovered refusal losses from 0.000 to 0.667.
  • The development ladder: Verbatim validation eliminated a residual +41 over-count, and coverage stamping recovered 5/5 failing seeds while preserving 5/5 passing controls.

B Protocol details

The protocol fixes decoding, error injection, scoring, corpus handling, and renderer provenance to make worker and reader comparisons reproducible and auditable.

  • Protocol controls: One sample, default decoding, and disabled thinking are used for every worker-loop and reader call.The returned API model identifier is recorded on every call, and fold and flat-log conditions exclude subagent streams.
  • Scoring: Accuracy is the unweighted mean of per-question scores in [0,1], with 70 of 72 possible real-corpus questions available.Two transcripts lack a question type’s ground truth.
  • Corpus disclosure: The real corpus contains twelve authors’ sessions from one framework, selected for size and treated as exploratory evidence unavailable for external audit.The synthetic replication provides the complementary evaluation corpus.
  • Reproducibility: Renderer variants are recoverable from run keys because v35 and earlier used the pre-stamp renderer, while v36 and later used the shipped post-stamp renderer.The variants differed in header wording and coverage-stamp presence.
  • Oracle checks: An independent raw-JSONL recount reproduces every per-key fact and aggregate sum on five chain-120 traces with 0 mismatches.Bookkeeping fields were separately oracle-checked on 8/8 cases with mutation testing.

C Full cell inventory (CONTINUE)

The CONTINUE inventory reports attempted-run costs, successful-run cost heuristics, parser comparisons, and a boundary family for long-horizon tasks.

  • Cost inventory: $32.1, $2.32, $1.20, $0.67, $14.88, and $2.98 are the heuristic costs per successful 120-link run for full context, curated, cached scratchpad, minimal tally, calculator, and hybrid.The metric is mean cost divided by success rate and penalizes inexpensive failure, but is not an operational forecast.
  • Cost inventory: The heuristic cost metric is computed from development-era 120-link cells, while other CONTINUE costs are reported per attempted run.Fixed-seed failures are not independent retries.
  • Parser comparisons: 1.000 accuracy is achieved by both the tuned-pattern parser and small-model parser on prose-chain 60-link tasks, while the fixed-pattern parser scores 0.000.The corresponding costs are $0.63, $0.80, and $0.62 respectively.
  • Parser comparisons: 3/3 is the small-model parser’s score at 120 links, with extraction cost $0.046 and zero refusals.The 60-link comparison uses n = 3.

D Cache accounting detail

Cache accounting shows that conversational caching substantially reduces full-context cost, while curated-view savings are limited by refresh writes.

  • Breakpoint behavior: 0 cache reads and 512,521 cache-write tokens make the naive single-block breakpoint cost $2.06 versus $2.03 uncached.The cache-write tokens incur a 1.25× premium.
  • Condition comparison: 5.4× is the chain-60 full-context cost reduction from $2.03 flat uncached to $0.374 conversational cached.Each run uses 501K reads and 41K writes.
  • Condition comparison: $0.74 to $0.66 is the smaller chain-60 cost reduction for curated views, with 46K reads and 158K writes per run.Refresh flushes dominate the curated condition.

E COMPREHEND detail

COMPREHEND evaluates monitoring answers from capped raw tails, flat logs, and compiled views using deterministic question generation and exact grading.

  • Real-corpus view results: 1.0 is achieved on latest_ask, top_tool, and files, while dangling reaches 0.917 in the real-corpus view condition with Sonnet 5.n_turns and last_error account for the remaining error.
  • Real-corpus view results: 0.126 is the raw-tail score for the files question with Sonnet 5, showing that aggregation questions are unavailable from a capped large-transcript tail.The question types use substring, exact-match, set-F1, or set-F1-or-none grading as specified.
  • Inventory scope: Table 5 inventories full CONTINUE cells, with costs per attempted run unless otherwise noted and altchain rows forming the §5.3 boundary family.The table is the full cell inventory for the CONTINUE evaluation.
  • Replication: 0.889 versus 0.347 is the v0 pilot view-versus-raw accuracy on three same-pool transcripts with Sonnet 5.The synthetic corpus reports view/flat/raw accuracies of 1.000/0.722/0.673 for Sonnet 5 and 0.944/0.638/0.600 for Haiku 4.5.
Loading 2609.01466v1…