PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory as Infrastructure: Reliability Engineering for Persistent Agent Memory in Months-Long LLM-Assisted Development

Mike Helwig

arXiv:2609.05510v1cs.SE

TL;DR

Months-long coding-agent work across repeated context compactions raises memory challenges, including repeated work and risks from noisy or silently failing memory. This paper reports on SIx Harness, a memory and continuity system with reliability monitoring, and presents a months-scale instrumented production record. The authors also identify limits of their single-deployment evidence and provide a pre-registered ablation protocol for further testing.

  • Problem

    Months-long agent work faces repeated rediscovery even when relevant design documents are present in context.

    Source: §1 · pp. 1–2

  • Method

    The paper reports on a deployed memory harness with operator-triaged anti-recurrence records, a session-start health gate, and per-hook heartbeat monitoring.

    Source: §2 · p. 4, §3.2 · p. 5, §3.4 · p. 6

  • Results

    78,933 hook invocations yielded 85 recorded failures, none silent, with none in the final 20 days.

    Source: §1 · p. 2, §4.1 · p. 8

  • Takeaways & Limitations

    The authors conclude that reliability engineering can be applied to agent memory, and make unresolved causal questions testable through a pre-registered ablation protocol.

    Source: §4.3 · p. 11, §9 · p. 14

    The evidence is from one project, operator, agent, stack, and domain, with no control arm and self-reporting by the system's operator-builders.

    Source: §2 · p. 4, §6 · pp. 11–12

Abstract

from arXiv · show

LLM coding agents are crossing from task-scale to project-scale: single assisted efforts that run for months, across repeated context compactions, on codebases far larger than any context window. We report operational experience from one such effort (a research project under a single continuous Claude Code session line since January 2026, driving a 633,000-line codebase, with its memory subsystem continuously instrumented since July 2026) and describe SIx Harness, the open-source memory and continuity infrastructure that emerged from keeping it coherent. The harness combines per-project long-term memory (hybrid lexical-vector retrieval over SQLite, fully local), precision-gated context injection, anti-recurrence stores for decisions and dead-ends, conventions engineered to survive compaction, and, the part we argue is missing from operational practice, reliability engineering for the memory subsystem itself: a session-start health gate with discriminated failure modes, heartbeat telemetry designed so that no enumerated failure mode can pass unrecorded, and alert-fatigue budgeting borrowed from SRE practice. We present what we believe is the first months-scale instrumented operational record of a persistent agent-memory subsystem in production development use: 78,933 hook invocations; 85 recorded failures, none silent: 84 in the subsystem's first three weeks, one since, none in the final 20 days; an injection layer whose ten-day precision instrument shows zero false fires against an intact denominator; and three production incidents traced from instrument reading to structural fix. From the record we distill seven design principles, state our limitations plainly (N=1, no control arm, self-reported), and publish a tagged pre-registered ablation protocol that any team can run with the released MIT-licensed kit.

1 Introduction

Months-long coding-agent work faces memory failures that range from losing context and repeating work to noisy retrieval and silent system failure. The paper presents SIx Harness and an instrumented operational record, arguing that persistent agent memory needs reliability engineering.

1 Introduction

  • Persistent agent work encounters four memory failures: knowledge loss, repeated work, context degradation, and silent memory death.The paper argues that silent memory death can make the other failures invisible unless the memory system is monitored.Source: §1 · pp. 1–2
  • SIx Harness combines local hybrid retrieval, precision-gated injection, anti-recurrence stores, compaction-surviving conventions, and reliability engineering for memory.Its reliability layer includes a health gate, heartbeat telemetry, freshness objectives, and attention-aware alerting.Source: §1 · p. 2
  • The paper reports a months-scale instrumented production record: 78,933 hook invocations, 85 recorded failures, an injection-precision record with an intact denominator, and three incident traces.The authors describe it as the first such record to their knowledge.Source: §1 · p. 2
  • Seven design principles distill failures into opposed operational rules for preventing both rediscovery and action on decayed records.The paper presents this as a workflow-level instantiation with in-vivo evidence.Source: §1 · p. 2
  • A tagged, pre-registered ablation protocol invites others to test causal questions that this N=1 record cannot answer.The protocol is designed for use with the released kit.Source: §1 · p. 2

2 The case: one session line, one codebase, eight months

The case study follows one logical Claude Code session line across eight months of continuous development on a large research codebase. Memory conventions and tooling accumulated over time, with reliability instrumentation and injection telemetry added later; the agent also operates much of the system it uses.

2 The case: one session line, one codebase, eight months

  • The case is a 633,446-line codebase developed almost entirely through Claude Code since January 2026, as one logical session line re-anchored after compaction.The instrumented final two months recorded 90 session-boundary crossings; earlier boundaries were uncounted.Source: §2 · p. 3
  • Conventions began in January 2026 with a one-line memory index and resume anchor, invariant files, append-only ledgers, and a project constitution.These practices began without tooling.Source: §2 · p. 3
  • The memory core entered production on 2026-05-24 with hybrid BM25 and vector retrieval over a single SQLite file.It ingests documents, ledgers, transcripts, and git history.Source: §2 · p. 3
  • The reliability layer entered production on 2026-07-02, beginning continuous instrumentation of hook invocations and memory health.The record reached 78,933 invocations through 2026-08-30.Source: §2 · p. 3
  • A denominator-complete injection instrument recorded every prompt, injection, latency, and gate verdict for ten calendar days beginning 2026-08-19.The window was defined by the instrument's deployment date, not selected after the fact.Source: §2 · pp. 3–4
  • The session lifecycle runs maintenance, health checking, and rendering at SessionStart, gates retrieval per prompt, and captures and triages records at SessionEnd.The figure describes the SessionEnd loop as feeding the next session.Source: Figure 2 · p. 4
  • At the time of writing, the memory state included 19,065 indexed chunks, 389 curated memory files, and triaged decision and dead-end records.Of 154 captured decisions and 186 captured dead-ends, 59 and 47 respectively survived operator triage.Source: §2 · p. 4
  • The agent substantially builds and daily operates the memory infrastructure under human direction and review, a role the authors also identify as a source of reporting bias.The agent runs triage queues, reads the session-start gate, refreshes stale corpora, and traces hook failures.Source: §2 · p. 4

3 The system

SIx Harness combines retrieval, curated continuity records, selective context injection, reliability checks, and workflow artifacts. Its design prioritizes detecting memory faults and managing what enters or interrupts the reader’s context.

3 The system

  • The session flow maintains and checks memory at startup, conditionally retrieves on prompts, and ingests transcripts for decision and dead-end triage at session end.The system also injects selected invariants, corrections, and decisions at startup.Source: §3 · p. 4

3.1 The memory core: boring storage, honest degradation

  • The memory core uses local SQLite hybrid search, degrades to BM25-only if embeddings fail, and rejects vectors with mismatched dimensions.The authors reject graph stores for this setting, arguing that their observed failures concerned pipeline reliability rather than data-model expressiveness.Source: §3.1 · p. 5

3.2 Anti-recurrence stores: curated, not scraped

  • 154 captured decisions and 186 dead-ends yielded 59 and 47 confirmed records after operator triage, while corrections are resurfaced at session start.Rediscovery stopped after an inventory requirement and a DO-NOT-REBUILD injection were introduced together, so their separate effects cannot be determined.Source: §3.2 · p. 5

3.3 Precision-gated injection: the economics of the reader’s attention

  • The injection cascade first gates retrieval on build, investigate, or decision intent, then requires rare-token relevance, applies per-session damping, and exempts machine-generated turns.The authors chose auditable corpus-statistics gates to limit irrelevant interruptions and preserve reader trust.Source: §3.3 · pp. 5–6

3.4 The reliability layer: SRE for a memory subsystem

  • 85 hook failures were recorded, none discovered by suspicion, within a reliability layer built around a session-start health gate and per-hook heartbeats.The gate checks freshness, integrity, coverage, and end-to-end embedding behavior; alerts are designed to identify actionable failures.Source: §3.4 · pp. 6–7

3.5 Conventions as shipped artifacts

  • Shipped workflow artifacts preserve continuity across compaction, while a tiered ground-truth hierarchy requires using tier-2 facts and verifying tier-3 claims before acting.Artifacts include a resume-anchored memory index, invariant files, an append-only arc ledger, and a constitution of standing rules.Source: §3.5 · p. 7

4 The operational record

The operational record tracks 78,933 hook invocations and 85 recorded failures, alongside a denominator-complete window for measuring injection precision and three incidents traced to structural fixes.

4.1 The long haul: 78,933 invocations and a maturation curve

  • 85 failures were recorded across 78,933 invocations: 84 in the first three weeks, one thereafter, none in the final 20 days, and none discovered by suspicion.The instrumented period ran from July 2 through August 30, 2026.Source: §4.1 · pp. 8–9
  • 20 Windows encoding failures and 34 FTS5 or database-lock failures ended after UTF-8 enforcement and bounding an unbounded query pattern, respectively.The encoding failures occurred on day one; the FTS5 and lock failures ran from July 2–6 and did not recur after the query was bounded.Source: §4.1 · p. 8
  • Thirteen dead-pipe echo aborts stopped after hook-echo write failures were absorbed, while four duplicate-chunk collisions stopped without an attributable fix.The collisions are recorded as luck, not maturation.Source: §4.1 · p. 8
  • 14 low-rate FTS5 parsing failures from control characters ended after input sanitization at ingestion boundaries, with the last recorded failure on August 10.The failures were reported from July 8 through August 10.Source: §4.1 · p. 8

4.2 The instrumented window: ten days at full resolution

  • 3 injections among 14 intent-bearing prompts were all judged relevant, with zero false fires on 54 non-intent prompts.The old pre-gate behavior fired on roughly four in five intent-bearing prompts; the ten-day window was light-duty, with 2,121 invocations, an idle gap, and a network outage.Source: §4.2 · p. 9, Figure 4 · p. 9

4.3 Three incidents and a delivery case

  • 11.3-second hook wall time fell to 1.6 seconds after fixes for IPv6-first localhost resolution and mismatched probe/request context options.The hook’s 3-second budget was exceeded nearly fourfold; the 580 offline tests could not catch these live-latency defects.Source: §4.3 · p. 10
  • A 768-dimensional fallback could not serve a 1024-dimensional index, adding an approximately 18-second model load and timeout chain before configuration-time dimension validation was added.A circuit breaker and widened budget were also introduced after injections computed for one prompt were lost at 8.1 seconds against the 3-second hook budget.Source: §4.3 · p. 10
  • Two topical injections fired on machine-generated turns with no reader; the remaining relevant delivery reached a human-authored intent-bearing turn, but prevented rework remains unmeasured.The authors identify that counterfactual as a target of the pre-registered A1 ablation.Source: §4.3 · pp. 10–11

5 Design principles

The design principles treat persistent agent memory as operational infrastructure, balancing reliable continuity with relevance, verification, and alert discipline.

5 Design principles

  • Run memory as infrastructure, with health checks, discriminated failure modes, heartbeats, freshness objectives, and one legible session-start verdict.Source: §5 · p. 11
  • Write a heartbeat row for every hook invocation, including routine ones, so failure rates and precision have an instrumented denominator.Source: §5 · p. 11
  • Favor precision over recall in injected context because irrelevant firings spend reader trust, and default to silence.Source: §5 · p. 11
  • Counter rediscovery and record decay with opposed rules—reuse without re-deriving, but verify before acting—arbitrated by a tiered ground-truth hierarchy.Source: §5 · p. 11
  • Make every RED actionable; treat cold as distinct from dead and optional as distinct from broken, and reject misconfiguration at configuration time.Source: §5 · p. 11
  • Make continuity an artifact: put compaction-critical knowledge in files the workflow revisits, since injection alone does not guarantee uptake.Source: §5 · p. 11
  • Favor simple storage contracts over novel representations, including refusing mismatched vectors, degrading search rather than blocking, and logging each degradation once.Source: §5 · p. 11

6 Limitations and threats to validity

The evidence comes from one deployment without a control arm, and the authors identify limits in measurement, representativeness, and generality. These constraints preclude causal effect claims and leave some practical costs and counterfactual outcomes unmeasured.

6 Limitations and threats to validity

  • N = 1: one project, human operator, agent, stack, and domain; the project was never run without the harness over a comparable period.The authors state that effect sizes are not causal claims.Source: §6 · p. 11
  • Model generations, tooling improvements, and operator learning span the eight-month record and can improve outcomes independently of memory infrastructure.The labels “topically justified,” “intent-bearing,” and “false fire” are author-adjudicated without an independent rater.Source: §6 · pp. 11–12
  • The pre-gate baseline’s sample denominator was not preserved, and the precision window is bounded by instrument deployment rather than selected representativeness.That window was light-duty—about a sixth of lifetime average load—and included an idle gap and an outage.Source: §6 · p. 12
  • Builder involvement creates observer effects, while self-report comes from the operator-builders, including an agent the system serves.Instrumentation was designed to resist motivated reading, but the authors emphasize that design intent is not independence.Source: §6 · p. 12
  • The harness’s own overhead—triage minutes, injected tokens, and maintenance attention—was not instrumented, so net-cost accounting is needed before claiming net benefit.The authors say the quantities of greatest interest, such as rediscoveries prevented and knowledge not lost, are counterfactuals observation cannot count.Source: §6 · p. 12
  • A second independently published system tempers the single-deployment concern for the design principles, but the memory-subsystem operational record remains N=1.The single-operator, local-only scope sidesteps team-scale memory-poisoning threats; shared memory would require admission control and provenance.Source: §6 · p. 12

7 A pre-registered ablation protocol

The authors preregister two ablations to test injection and gate functions, with explicit decision rules and a commitment to report outcomes regardless of direction. The protocol preserves telemetry and denominators where possible and treats underpowered quiet windows as inconclusive.

Protocol and replication

  • The full protocol was tagged and preregistered before any ablation run, and the authors commit to reporting results regardless of direction and invite hostile replication.The kit installs on any Claude Code project in one command, and its metrics query stores the kit creates.Source: §7 · pp. 12–13

A1: injection off

  • A1 disables injection for 14 days while keeping intent detection, relevance gating, and telemetry active, so suppressed firings remain recorded in the denominator.It compares re-proposal incidents per intent-bearing prompt with the preceding 14 days; support requires at least two scored incidents in the ablated window and fewer in the intact window at comparable volume.Source: §7 · p. 12

A2: gate off

  • A2 disables the gate for 14 days while retaining heartbeats and degradation logs, and measures time from first degradation evidence to remediation.Support requires at least two degradations and median detection latency at least 4× the intact median; fewer than two occurrences is inconclusive.Source: §7 · p. 12
  • A quiet A2 fortnight proves nothing: the observed base rate is roughly one degradation per multi-day idle gap.The protocol treats fewer than two degradation occurrences as inconclusive.Source: §7 · pp. 12–13

8 Related work

Prior work richly studies memory architectures, benchmarks, and agent capabilities, but the authors find little operational reporting on memory-system reliability in production. They position their months-scale record alongside controlled memory-failure studies, longitudinal development reports, and software-engineering work on knowledge loss.

Memory architectures

  • Memory-architecture research spans retrieval systems, surveys, and the ACI thesis, but the authors identify memory-system operation in production as unreported.The Always-On Agents survey of 435 works finds more emphasis on accumulating and retrieving state than on governing, recovering, or relinquishing it.Source: §8 · p. 13

Memory failure studies

  • Recent memory-failure studies measure aging, stress failures, staleness, and invalid-memory reuse under controlled faults; the authors frame these as in-vitro companions to natural incidents.They report finding no prior operated, health-gated, alert-budgeted reliability layer for a memory subsystem specifically.Source: §8 · p. 13

Verification and reliability practice

  • The authors connect their approach to prior verification work, where a harness governs model claims about data; here, the same stance governs a development agent’s memory subsystem.The earlier line concerns metadata reconstruction and design principles.Source: §8 · p. 13

Production experience with AI-assisted development

  • Longitudinal production-development reports are memory-blind, while the nearest neighbor covers 391 sessions on one project over about a month as qualitative action research.That study reports “Index Sickness” and a minimal baseline-log fix.Source: §8 · pp. 13–14
  • The two operational records differ on scaffolding: Written by AI describes standing symbolic constraints, while this paper’s injection is episodic, relevance-gated, silent by default, and health-monitored.Both reports warn that un-curated accumulation degrades the channel it fills; whether gated injection yields net benefit remains open and motivates A1.Source: §8 · p. 14

Software-engineering antecedents

  • Pre-AI software-engineering research addressed knowledge loss under developer turnover, stale documentation, and knowledge held in developers’ heads; the authors frame compaction as analogous turnover.They argue this makes turnover economics a more relevant ancestral frame than context-window engineering.Source: §8 · p. 14

9 Conclusion

At tenure scale, the authors frame agent memory as load-bearing infrastructure that can fail quietly. Their single-deployment record suggests reliability engineering can be applied to agent memory, while preregistered protocols make unresolved questions testable.

9 Conclusion

  • The record comprises 78,933 invocations, 85 recorded failures with none silent, none in the final 20 days, an accountable injection channel, and three incidents traced to structural fixes.The authors conclude that reliability engineering transplants to agent memory and that unresolved questions can be made precise, preregistered, and inexpensive for others to test.Source: §9 · p. 14

Artifact availability

SIx Harness v0.2.2 is released under the MIT license with an offline test suite, installer, and tagged pre-registration. Its documented queries, schemas, and instrument code are available, while project-identifying instrument stores are not released.

Artifact availability

  • SIx Harness v0.2.2 is MIT-licensed and its repository includes a 580-test offline suite.An archival DOI deposit accompanies submission.Source: Artifact availability · p. 14
  • The release includes a one-command installer and the tagged prereg-ablation-v1 pre-registration.Source: Artifact availability · p. 14
  • Documented queries, schemas, and instrument code are released, but project-identifying instrument stores are withheld.The paper says its reported numbers are re-derivable from documented queries against those stores.Source: Artifact availability · p. 14
Loading 2609.05510v1…