PaperlayerView exact revision on arXiv

Source-linked AI summary

ECHO: A Cognitively Inspired, Auditable Memory Plane for Long-Horizon Agents

Yu Qian, Hong Miao, Boyang Guo, Tingyi Jiang, Shan Zhao, Tianxing Le, Lintian Li, Meng Liu

arXiv:2608.21755v1cs.AI

TL;DR

Long-horizon agents need memory that can identify relevant experience, resolve revisions, and expose checkable provenance. ECHO addresses this with an auditable architecture separating immutable experience, bitemporal authority, candidate discovery, provenance closure, and answer realization, then evaluates retrieval and context construction across benchmark and matched-QA settings. The results include strong frozen retrieval measurements, a failed fresh-history BEAM gate, and a matched sample favoring Mem0 OSS, while audited query expansion limits the interpretation of visible-dataset retrieval scores.

  • Problem

    Long-horizon agents need memory that identifies relevant experience, resolves revisions, and exposes checkable provenance.

  • Method

    ECHO separates immutable experience, typed bitemporal state, hybrid candidate discovery, provenance closure, and controlled surface realization.

  • Results

    ECHO reports high frozen retrieval and evidence-coverage measurements, but the fresh-history BEAM gate fails and Mem0 OSS outperforms ECHO on the matched 91-question QA sample.

  • Takeaways & Limitations

    The contribution is architectural and methodological: make currentness, support, conflict, and failure observable while aligning claims with their evidence boundaries.

  • Takeaways & Limitations

    Visible-dataset retrieval scores are descriptive because source-specific query-expansion phrases were used, and the matched comparison has only eight history clusters.

Abstract

from arXiv · show

Long-horizon agents need memory that identifies relevant experience, resolves revisions, and exposes checkable provenance. We present ECHO (Embodied Context and History Orchestration), an auditable memory architecture and service prototype inspired by episodic encoding, consolidation, contextual reinstatement, reconsolidation, and executive control. This is functional inspiration, not neural equivalence; the empirical analysis focuses on retrieval and context construction. Development runs reach 96.29% Hit@10 and 73.64% turn Recall@5 on 1,536 LoCoMo category 1-4 questions, and 97.60% Hit@10, 88.84% turn Recall@5, and 88.71% session Recall@5 on all 500 LongMemEval-S questions. A five-history BEAM gate fails, and in a separate matched 91-question QA sample Mem0 OSS scores 64.84% versus ECHO's 41.76% (exact McNemar p = 0.00107), with a history-cluster interval crossing zero. A post-hoc audit found source-specific phrases in the query-expansion rules. Although no gold answer field entered the runtime, expansion-enabled retrieval scores are therefore descriptive development measurements, not independent confirmation.

I. INTRODUCTION

ECHO frames long-horizon memory as auditable infrastructure that preserves experience, resolves revisions by time and authority, retrieves supporting context, and separates internal derivation from visible answers. The introduction positions its evidence as stage-specific and explicitly reports both positive retrieval findings and failures that bound the claims.

  • Motivation: A useful memory system preserves former values, represents revisions without rewriting history, selects the value valid for the requested time, and returns supporting evidence.The motivating example contrasts maintaining an auditable past with merely storing past records.
  • Motivation: Memory failures can persist despite favorable scores when retrieval omits evidence, stale values outrank current revisions, or correct evidence yields an incorrect answer.The paper separates durable capture, structured-state evolution, context-sensitive discovery, and controlled realization into distinct responsibilities.
  • Architecture: ECHO combines immutable events, typed revisions, hybrid candidate routes, bitemporal authority, provenance closure, and an executive boundary for visible answers.The cognitive mapping motivates this decomposition but does not claim neural, biological, or clinical equivalence.
  • Evaluation: LongMemEval-S can show high question hits and evidence coverage, while a fresh-history BEAM gate fails and the matched QA sample favors Mem0 OSS.These outcomes distinguish architectural guarantees from what the current ranker and answerer demonstrated.
  • Architecture: The system specifies a typed bitemporal authority component where valid time, transaction time, revision state, provenance, and conflicts are explicit.Semantic similarity cannot determine factual currentness.
  • Evaluation: The evaluation reports retrieval, provenance-closure, bounded-context behavior, frozen benchmark artifacts, a BEAM pilot and gate, and a protocol-matched 91-question QA comparison.The study retains audits, failures, latency, context size, and hashes while disclosing query-expansion contamination risk.
  • Evaluation: The evidence is stage-separated: retrieval, sampled QA, and product scores use different estimands, so results remain scoped to their evaluation protocols.The paper presents an architecture-and-audit study rather than silently converting one empirical measure into another.

A. Events, States, and Two Time Axes

ECHO separates immutable event history from typed state revisions and resolves states using valid time, known time, revision status, provenance, and ledger generation. Retrieval proposes candidates and evidence closures, but only the ledger determines authority before budgeted context packing.

  • Interaction histories are append-only event streams whose immutable source identity, actor, text, order, and timestamps support typed projections and state revisions.
  • State records separate valid time from transaction time while retaining revision relations, immutable provenance, ledger generations, and status.Valid time represents when a proposition is true; transaction time represents when the system records it.
  • Ledger resolution admits a state only when its key matches the requested valid and known-time view and its generation is visible.The resolution set is defined by key, both time axes, and the visible generation bound.
  • If no supported active revision exists, ECHO marks evidence missing; incompatible unresolved revisions produce conflict disclosure or abstention rather than similarity-based selection.Admissibility requires compatible time, revision status, provenance, and visible ledger generation; semantic similarity can discover but not authorize a state.
  • Candidate discovery unions lexical, semantic, typed-state, and neighborhood routes, then packs complete atomic evidence closures under a context budget.The packer may defer a closure that does not fit but cannot retain a conclusion while dropping required support.
  • Operation types such as FACT, COUNT, LIST, TEMPORALLOOKUP, and TEMPORALARITHMETIC determine required evidence completeness and the visible answer surface.Internal completeness is distinct from visible completeness.

C. Objective and Claim Taxonomy

ECHO evaluates retrieval, context construction, answer correctness, safety, and cost as distinct objectives rather than collapsing them into one retrieval score. Its cognitive framing supplies testable engineering commitments, while claims remain bounded by evaluation protocol and evidence scope.

  • The reader maps frozen context and internal derivation to a surface-constrained answer, reporting correctness alongside support recall, abstention, token use, and latency.
  • The evaluation objective combines end-to-end answer correctness with provenance-closure and generation-fencing constraints, exposing safety and cost failures hidden by Hit@k.
  • Only frozen candidate and fresh confirmatory evaluations support final superiority language, and only when their preregistered gates pass.
  • ECHO translates episodic encoding, consolidation, reconsolidation, contextual reinstatement, and executive control into explicit system commitments without claiming biological equivalence.
  • The paper frames cognitive inspiration as a decomposition whose value lies in independently auditable invariants and ablations.

B. Target Runtime and Evaluation Scope

ECHO’s target runtime separates durable ingestion, asynchronous projection and indexing, synchronous ledger authority, provenance-aware packing, and controlled answer realization. The study reports each result at its corresponding service or domain-test boundary.

  • Durable ingestion is decoupled from optional projection and index maintenance, while the query path returns from candidate discovery to the bitemporal ledger before evidence packing.
  • Projection converts raw events into typed propositions while preserving source pointers, and deduplication uses event and typed-entity identity rather than repeated wording.
  • The ledger appends revisions instead of overwriting state, allowing supersession, revocation, unresolved conflicts, and late-arriving transaction-time evidence.
  • Hybrid candidate routes combine lexical, semantic, typed-state, and neighborhood retrieval; fusion ranks inspection candidates but cannot determine currentness.
  • The packer groups count, temporal, and current-state evidence sets, deduplicates and prioritizes them, and restores source order and timestamps within token budgets.Budgets are 768, 2,048, or 7,000 estimated tokens.
  • ECHO records hashes, source IDs, hit counts, truncation, closure IDs, and currentness ownership so evidence interventions can be tested independently of generated answers.
  • Internal derivation may enumerate or normalize evidence, while the public result is constrained by the operation’s visible contract.

G. Failure Semantics

ECHO’s evaluation protocol preserves failed rows, separates service and domain-level boundaries, and documents comparison-arm and hardware constraints. The resulting claims are tied to fixed scopes, protocols, and reproducibility artifacts.

  • Reader transport, parsing, and judge failures score zero, while deterministic abstentions and fixed judgment reuse prevent post-hoc retry or row filtering gains.
  • The empirical study reports service-level retrieval and domain-level ledger and closure tests at their corresponding evaluation boundaries.
  • The Mem0 comparison is pinned to archived software, model, dependency, and protocol metadata, but its stock adapter drops benchmark timestamps and lacks complete source-turn lineage.
  • Mem0 therefore cannot expose ECHO-style Hit@10, turn/session recall, or provenance MRR for that arm.
  • Local measurements use a specified Windows, WSL2, Docker, CPU, RAM, GPU, and memory-limit configuration distinct from managed Mem0 and author-reported product results.
  • Frozen retrieval scopes report registered and evaluated counts to prevent denominator drift across LoCoMo, LongMemEval-S, and BEAM measurements.
  • The study asks separately about annotated evidence recovery, benchmark-type limitations, Mem0 comparison, latency and context costs, and load-bearing components.

B. Datasets

The evaluation uses frozen retrieval artifacts, a scoped BEAM pilot and gate, and a matched 91-question ECHO–Mem0 OSS comparison. Protocols preserve distinct estimands, paired uncertainty, and dataset-specific limitations.

  • Evaluation scope: 1,536 LoCoMo category-1–4 rows, 500 LongMemEval-S questions, and a local 100K BEAM conversation define the primary evaluation scopes.LoCoMo category 5 is reported separately, and the BEAM development result is not an official BEAM-1M or BEAM-10M evaluation.
  • Evaluation scope: Top-10 retrieval uses HNSW, eight workers, query expansion with at most three subqueries, and dataset-specific candidate and context ceilings.LoCoMo and BEAM use 256 candidates and 29.4 kB context; LongMemEval-S uses 512 candidates and 60 kB.
  • Audit boundaries: The frozen query-expansion rules contain source-specific phrases, so expansion-enabled retrieval and ablation values are development-stage diagnostics pending a source-neutral rewrite and clean rerun.The public adapters exclude answer, reference, rubric, and support fields, but expansion can still inject an answer-bearing token absent from the question.
  • Matched comparison: The matched comparison contains 91 questions from eight histories, with a shared local qwen3:4b answerer and binary judge but system-specific saved contexts.The sample includes 65 LoCoMo, six LongMemEval, and 20 local BEAM-100K questions.
  • Matched comparison: Mem0 is 23.08 percentage points higher in binary QA accuracy on the fixed sample, while the history-cluster sensitivity interval crosses zero.The protocol-level result does not support a population advantage or retrieval-quality ordering.

E. Leakage and Invariance Audits

The audits separate retrieval evidence from answer accuracy and test whether development findings generalize to fresh histories. They expose coverage gaps, a failed five-history BEAM gate, and scope limits on comparative claims.

  • Metric invariance: Hit@10 records whether at least one annotated turn appears, whereas turn recall measures how much required evidence appears; the metrics are not interchangeable.This distinction makes question-level hits insufficient for judging complete evidence delivery.
  • Retrieval coverage: 86.96% Hit@10 and 66.83% turn recall make LoCoMo category 3 the weakest slice, with misses dominated by multi-turn and indirect support.Category 1 also has a coverage gap at 71.38% turn recall.
  • BEAM diagnostics: 100% Hit@10 coexists with 42.86% annotated-turn recovery and 50% session recovery for event ordering and summarization in the BEAM development pilot.Temporal reasoning has full turn and session recall but MRR 0.3214, indicating a ranking rather than coverage failure.
  • BEAM diagnostics: The five-history BEAM run fails all three retrieval thresholds, with event ordering at 40.0% Hit@10, 14.71% turn recall, and 21.57% session recall.A Top-50 diagnostic places many gold turns at ranks 16–41, implicating candidate coverage and ranking rather than ingestion.
  • Comparative audit: Mem0 is 23.08 percentage points higher in binary QA accuracy on the fixed 91-question sample, but the history-cluster interval spans [−25.5, +25.0] points.The conclusion is limited to this sample and shared answer protocol; no population advantage or retrieval-quality ordering is inferred.
  • Comparative audit: Mem0’s revision objects lack complete source-turn lineage, making ECHO-style provenance and retrieval metrics unobservable for that baseline.The automated judge result remains provisional pending independent labeling of the frozen packet by two humans.

E. Author-Reported Results Are a Separate Estimand

The paper separates ECHO retrieval, matched QA, and author-reported product scores as different estimands, with development ablations and worker-path measurements remaining scoped to their protocols.

  • ECHO retrieval tables measure question-level Hit@10 and micro annotated-unit coverage, not QA accuracy.Turn and session recall are recomputed from the same frozen Top-10 rows.
  • The preregistered fresh-history BEAM gate required at least 80 questions, zero projection failures, Hit@10 ≥85%, turn recall ≥50%, and session recall ≥60%.
  • Author-reported managed Mem0 numbers are shown for orientation only because they are not metric-compatible with ECHO retrieval or matched sampled QA.
  • The matched QA sample establishes a question-level disadvantage for ECHO on fixed questions, while its limited history-cluster representation prevents population ranking.
  • BEAM Retrieval Ablations: Within the BEAM-100K development configuration, removing query expansion halves mean query time but reduces Hit@10 by 16.67 points and turn recall by 26.42 points.Because the audited rules contain source-specific phrases, the drop measures dependence on that particular rule set rather than generic reformulation.
  • Stateful TopicBucket saturates near four workers, whereas pure C++ ROSE reranking continues scaling; these are worker-path measurements, not QA accuracy.

B. Robustness Boundary

Robustness evidence is bounded by controlled infrastructure tests, development artifacts, and unresolved protocol limitations rather than end-to-end generalization alone.

  • The ablations validate component dependence, not universal robustness across late arrival, correction, retraction, contradictory sources, missing support, dates, or stale indexes.
  • Across 40 worker-stress configurations and 8,000 operations, no operation fails, but the deterministic harness isolates worker/state-path capacity rather than end-to-end answer latency.
  • 107.4k calls/s is the predictive-core peak at four workers, while TopicBucket/DuckDB peaks at 1,507 calls/s and ROSE core reaches 15.9k reranks/s at 16 workers.
  • Four workers are the defensible stateful operating point on this host, while production capacity still requires replication on target hardware with live model endpoints.
  • The post-merge carrier gate reaches Recall@1, Recall@10, MRR, and NDCG@10 of 1.0, but excludes embedding and LLM calls and is neither end-to-end nor cross-system.
  • Human judge calibration and reserved-history confirmation remain future work.

IX. DISCUSSION

ECHO’s discussion frames memory as governed evidence infrastructure: discovery proposes candidates, authority resolves revisions and provenance, and results remain bounded by evaluation protocol and scope.

  • A. From an Archive to a Governed Memory Plane: ECHO separates similarity-based discovery from bitemporal authority and provenance closure, making stale answers, unresolved conflict, and missing support observable failures.
  • A. From an Archive to a Governed Memory Plane: The cognitive framing motivates immutable experience, linked typed projections, append-only revision, multi-route recall, and selective realization without claiming biological equivalence.
  • B. What the Evidence Supports—and Falsifies: 78.65% Hit@10 and 40.80% turn recall on the fresh-history BEAM gate falsify generalization of the 100% pilot hit rate across BEAM histories.Top-50 diagnostics localize much of the gap to ranking and coverage, especially event ordering and summarization.
  • B. What the Evidence Supports—and Falsifies: Mem0 OSS is significantly better at the question level on 91 fixed items, while the eight-history cluster sensitivity interval remains inconclusive.
  • The paper recommends reporting question hits with micro turn/session coverage and first-relevant rank, while separating development, fresh generalization, and end-to-end QA.
  • X. LIMITATIONS AND ETHICAL CONSIDERATIONS: Visible-dataset retrieval scores are provisional because source-specific expansion phrases were audited, and the five-history BEAM gate fails.
  • X. LIMITATIONS AND ETHICAL CONSIDERATIONS: The study evaluates evidence delivery rather than autonomous behavior or human-like cognition, and judge-dependent conclusions are protocol-level results rather than population estimates.
  • X. LIMITATIONS AND ETHICAL CONSIDERATIONS: Persistent personal memory requires privacy, consent, deletion, access control, encryption, auditability, and provenance-aware revocation.

APPENDIX A EVIDENCE CONTRACT

The evidence contract binds reported results to reproducible artifacts, preserves failures and uncertainty, and reserves superiority claims for protocol-matched confirmatory evaluations.

  • All ECHO retrieval rows must be bound to dataset, source, model, prompt, and executable hashes.
  • Interrupted audits may resume only by preserving valid records and retaining truncated lines as failure evidence.
  • Retrieval and QA metrics are separate estimands, and unexposed baseline fields are reported as not observable rather than zero.
  • A future superiority claim requires a frozen answerer, human-calibrated judge, powered protocol-matched baseline, fresh source passing its preregistered gate, zero denominator drift, and preregistered non-regression gates.
  • Frozen artifacts include benchmark runs, fresh-gate summaries, question-level audits, ablation analyses, worker-stress reports, and a post-merge carrier gate.
Loading 2608.21755v1…