PaperlayerView exact revision on arXiv

Source-linked AI summary

FluctlightDB: A Memory Model of Data for AI Agents

Ganesh S

arXiv:2608.12365v1cs.DBcs.AI

TL;DR

Agent memory lacks a shared database abstraction for provenance-weighted, cue-driven recall across long sessions. FluctlightDB defines such an engine contract, reporting 99.0% LoCoMo evidence recall, 97.6% LongMemEval-S session recall@8, and 97.4% end-to-end QA.

  • Problem

    Agent memory lacks a shared database abstraction for heterogeneous, provenance-weighted, cue-driven recall across persistent sessions.

  • Method

    FluctlightDB models memory as linked engrams and implements experience() for encoding and activate() for cue-driven graph retrieval with fused lexical, semantic, and provenance scores.

  • Results

    99.0% evidence recall on LoCoMo, 97.6% session recall@8 and 97.4% end-to-end QA on LongMemEval-S, and 100% macro on FAMB are reported.

  • Takeaways & Limitations

    FluctlightDB provides a reproducible engine for agent memory whose claims can be independently reproduced and contested.

  • Takeaways & Limitations

    In a shared-brain provenance suite, performance drops to 18% (9/50) from 100% in isolated conflict pairs, and the issue remains unaddressed and unevaluated beyond the synthetic test.

Abstract

from arXiv · show

For fifty years, data systems have answered two questions. The relational model asked which records match a predicate; the vector model asked which vectors lie nearest a query. Neither was built for cue-driven, provenance-weighted recall across long sessions. We propose treating long-term agent memory as a distinct data model -- with its own write semantics (encoding, separation, consolidation, provenance) and read semantics (cue-driven activation across a linked memory graph) -- and present FluctlightDB, an embedded engine that implements this contract via experience() and activate(). We make that case carefully, not categorically: we do not claim novelty over Mem0, Zep, or HippoRAG-style memory layers, only an embedded engine contract beneath them. On LoCoMo (official evidence-recall metric; 10 conversations, 1,982 gold spans), CHORUS recalls 99.0% on an internally reproduced July 2026 run. On LongMemEval-S (500 questions, official session_recall@8), our retrieval harness scores 97.6% (488/500); end-to-end QA with our reader/judge stack scores 97.4% (487/500) -- these layers use different protocols than vendor leaderboard figures we cite for context only. On BEIR SciFact (shared MiniLM embeddings, same harness, Recall Fabric on), CHORUS/PRISM edges Chroma on nDCG@10 (0.646 vs. 0.645) and Recall@10 (0.792 vs. 0.783). We also report a small author-designed regression suite (FAMB; paraphrase n=10, other sub-tests n=1) at 100% macro -- internal validation, not peer benchmark. Strangers can verify the engine in under a minute via pip install "fluctlightdb[native]" and a minimal connect() -> experience() -> activate() script (compiled wheel, not source-only). Harnesses and frozen JSON are MIT-licensed. We claim no new neuroscience and no new transformer; we propose a missing layer of the data stack and release an engine others can reproduce and contest.

1 Introduction

The introduction argues that persistent agent memory requires a distinct database model and embedded engine, because relational rows and vector similarity do not capture cue-driven, provenance-weighted recall. FluctlightDB proposes episode-based memory with explicit write/read semantics and reports benchmark evidence alongside reproducibility resources.

  • Motivation: Agent memory currently requires manually coordinating session storage, vector indexing, deduplication, trust policies, and consistency glue across sessions.The introduction frames agents as persistent programs that act, observe, and retain information despite lacking a dedicated database abstraction.
  • Scope and reproducibility: The paper presents measurements rather than categorical proof and provides open harnesses, frozen result JSON, and a one-minute pip install "fluctlightdb[native]" smoke path.These resources are intended to let others rerun or contest the reported numbers without relying solely on maintainer scripts.
  • Contribution: FluctlightDB proposes agent memory as a first-class data model with episode-based writes and cue activation over a linked memory graph.Its write semantics include separation, encoding, consolidation, and provenance; its read semantics use cue activation rather than cosine similarity alone.
  • Contribution: FluctlightDB is an embedded Rust engine exposing experience(), activate(), and checkpoint() without a server process.It provides one durable brain directory per agent, analogous to SQLite’s embedded deployment model.

2 A Third Data Model for Agent Memory

Agent memory differs from relational and vector data because it is heterogeneous, contradictory, provenance-sensitive, and cue-dependent rather than governed by uniform truth or embedding similarity. FluctlightDB models memory as linked engrams and defines experience for writing and activate for cue-driven reading.

  • Motivation: Agent memory requires provenance-weighted, cue-dependent recall because memories may conflict, arrive out of order, or matter due to episode-level associations rather than similarity.Relational systems lack native provenance-weighted recall, while pure approximate nearest-neighbor search cannot encode association strength, salience decay, or trust.
  • Memory model: FluctlightDB represents memory as engrams carrying content, encoding context, salience, optional provenance, and edges to co-activated engrams.This representation captures both the source and episodic associations that distinguish agent memory from conventional records or vectors.
  • Operations: The experience write operation gates near-duplicates, encodes engrams, registers semantic vectors, and wires graph edges.The activate read operation takes a cue, seeds lexical and semantic indexes, and spreads activation through the graph before fusing scores.

3 System Design

FluctlightDB is an embedded episodic-memory engine exposing experience() and activate() over a versioned, WAL-backed on-disk format. Its write path encodes provenance-rich engrams and graph links, while its read path combines hybrid retrieval, graph activation, and provenance-aware ranking.

  • Persistence: Each agent brain is a directory containing versioned segments for engrams, synapses, semantic fields, neuromodulators, and metadata, plus an FTS5/HNSW recall sidecar.Segments use temporary files, fsync, and atomic rename; opening replays WAL entries after the checkpoint watermark and skips torn tail records.
  • Write path: experience() gates near-duplicates, encodes context-, salience-, and provenance-rich engrams, registers semantic vectors, and wires co-activation edges.Fast-ingest mode skips graph wiring for supplied dense vectors, providing O(1) hippocampal insertion plus sidecar update per chunk.
  • Deployment and durability: connect() provides the full episodic engine, whereas connect index() supports bulk semantic ingestion; both share one engine, WAL, and file format.checkpoint() atomically writes segments and advances the WAL checkpoint sequence, while optional WAL preserves recent experiences between checkpoints.
  • Reranking: Recall Fabric is an optional CHORUS-lane shortlist reranker, enabled for LoCoMo, BEIR, and FAMB headline results but not LongMemEval-S hybrid-index retrieval.Fabric is validated with synthetic property tests rather than standalone peer benchmarks.
  • Read path: activate() seeds candidates with BM25 and HNSW, spreads activation up to four graph hops, fuses lexical, semantic, graph, and provenance scores, and returns ranked engrams.Vector-fast mode skips graph spread; hybrid seeding avoids a full-table scan through FTS5 and HNSW.

4 Evaluation

Evaluation shows strong recall across public benchmarks, with CHORUS reaching 99.0% LoCoMo evidence recall, 97.6% LongMemEval-S retrieval recall, 97.4% end-to-end QA accuracy, and a slight BEIR SciFact edge over Chroma. Internal suites report 100% macro performance but expose a severe shared-brain provenance failure and limitations in comparability and tenant isolation.

  • LoCoMo: 99.0% mean evidence recall on full LoCoMo is achieved by CHORUS+Fabric at k=150 across 10 conversations and 1,982 gold evidence spans.The evaluation separates retrieval from generation, and the headline result uses SPECTRUM full readout rather than the index lane.
  • LoCoMo robustness: At k=50 on full LoCoMo, hybrid BM25+dense and vector-fast-only are statistically tied at approximately 90.2% and 90.3%, so the 99.0% headline depends on CHORUS+Fabric at k=150.This comparison indicates that lexical seeds do not provide a large uniform gain at moderate retrieval budgets on this benchmark.
  • Provenance conflicts: 100% isolated provenance accuracy (50/50) falls to 18% (9/50) in the shared-brain stress test because cross-case cue contamination returns unrelated engrams.No implementation mitigates this failure mode as of the preprint; scoped activation can post-filter by agent id, but ingest and benchmarks do not enforce per-tenant namespaces.
  • Internal validation: 100% macro performance is reported for both connect agent() FAMB and CHORUS, but FAMB is an author-designed internal suite with paraphrase n=10 and n=1 scenarios for its other tests.The authors explicitly keep FAMB and the provenance-conflict suite separate from LoCoMo and BEIR headline figures.
  • LongMemEval-S: 97.4% end-to-end accuracy (487/500) and 97.6% retrieval recall@8 (488/500) are reported on LongMemEval-S, with remaining errors concentrated in reader aggregation.E2E accuracy is 100% for single-session user, assistant, and preference questions, 99.3% temporal, 97.4% knowledge-update, and 92.5% multi-session; vendor rows use different stacks.

5 Discussion

The discussion frames FluctlightDB’s contribution as evidence retrieval rather than answer generation, while emphasizing operational durability and unresolved limitations in cross-case provenance and reader-dependent QA. Its strongest results are retrieval scores of 99.0% on LoCoMo and 97.6% session recall@8 on LongMemEval-S, subject to specified budgets and protocols.

  • Results: 99.0% LoCoMo evidence recall and 97.6% LongMemEval-S session recall@8 (488/500) demonstrate strong retrieval, while the engine—not the reader—is the claimed contribution.The LoCoMo headline uses CHORUS+Fabric at k=150, and LongMemEval-S reports retrieval-only performance by question type.
  • Ablations: At k=50, hybrid BM25+dense and vector-fast-only retrieval tie within 0.1 percentage points mean evidence recall, so hybrid superiority is not assumed.Dialogue evidence can depend on entities and dates, and the measured ablation is distinct from the k=150 headline configuration.
  • Limitations: 100% isolated provenance accuracy falls to 18% (9/50) when conflict pairs share one brain, reflecting cross-case cue contamination and an unaddressed multi-tenant limitation.This result comes from a synthetic stress test; the isolated score is not treated as evidence of multi-tenant readiness.
  • Limitations: 23.5% category F1 in a 50-question LoCoMo QA pilot contrasts with 99.5% retrieval on the same slice because gold answers often require inference rather than span copying.A verbatim answer-in-context proxy reaches about 38%, underscoring the reader-generation gap.
  • Results: 487/500 LongMemEval-S end-to-end QA remains weakest on multi-session aggregation at 92.5%, showing that reader and prompt choices constrain answers beyond retrieval.The end-to-end pipeline uses GPT-4o or GPT-5 readers and a GPT-4o judge; vendor figures use different stacks.

6 Future Work

Future work prioritizes resolving the shared-brain provenance gap through scoped memory operations, stricter candidate filtering, and re-benchmarking under realistic multi-tenant load. The repository does not yet implement this work, which should report before/after top-1 accuracy in Table 6.

  • 6 Future Work: 18% top-1 accuracy with shared-brain provenance versus 100% with isolated provenance is the highest-priority gap for a revised preprint.
  • 6 Future Work: Future work should add session- or agent-scoped experience/activate, stricter hybrid candidate filtering, and provenance-conflict re-benchmarking under realistic multi-tenant load.
  • 6 Future Work: These changes are not implemented in the accompanying repository, and a future version should report before/after top-1 accuracy in Table 6.

7 Related Work

FluctlightDB positions long-term agent memory as an engine-level data model with native episode, provenance, separation, and activation semantics, distinct from SQL rows and vector similarity. Its claim is deliberately narrower than prior structured-memory systems: an embedded experience/activate contract beneath application-layer approaches.

  • Vector retrieval systems: FluctlightDB distinguishes memory from approximate nearest-neighbor systems such as Chroma, Qdrant, and FAISS, which provide similarity search without native episodic or provenance-weighted recall.Its vector-fast path supports IR measurement, while episodic mode adds graph co-activation for live agents.
  • Structured memory systems: Mem0 [1], Mem0g, Zep, and HippoRAG [4] already model memory through facts, graphs, associative indices, or provenance-like policies, but primarily at the application layer.Mem0 [1] and Zep report LoCoMo using reader-LLM or LLM-as-a-judge QA, unlike the official evidence-recall retrieval protocol adopted here.
  • Positioning: The narrower contribution is an embedded store whose native contract is experience/activate, organizing episodes, cues, provenance, durability, and separation for long-term agent memory.This frames memory as a data model distinct from SQL rows and vector ANN rather than claiming that structured memory itself is new.
  • Evaluation context: LoCoMo [8] and LongMemEval [14] evaluate long-horizon memory, while BEIR anchors information-retrieval credibility for the engine’s measurements.These benchmarks support testing the proposal to treat memory as a data model alongside relational rows and vectors.

8 Conclusion

FluctlightDB argues that autonomous agents need a rigorous, SQLite-like database for memory, combining strong vector-baseline performance with high retrieval and regression-suite results.

  • Conclusion: FluctlightDB frames autonomous-agent memory as a distinct database need alongside relational facts and vector similarity search.The proposed engine is intended to be as easy to adopt and rigorous to trust as SQLite.
  • Conclusion: 100% macro performance on the authors’ agent-memory regression suites supports the engine’s internal validation claim.The passage presents these as author-designed regression suites rather than external benchmarks.
  • Conclusion: 99.0% of gold evidence was recalled on LoCoMo retrieval.The passage characterizes this as evidence of FluctlightDB’s retrieval performance while also noting that vector baselines remain strong in shared harnesses.

A Recall Fabric mechanisms

Recall Fabric is a deterministic, dependency-free mechanism layer combining prefiltering, lattice and phase addressing, consolidation, forgetting, temporal buckets, and confidence-weighted consensus. It indexes engrams and CHORUS traces on ingest, reranks hybrid and CHORUS shortlists on recall, and is supported by synthetic property tests rather than standalone peer benchmarks.

  • Recall Fabric mechanisms: Recall Fabric combines Photon SimHash/LSH prefiltering, Manifold Lattice addressing, theta–gamma phase parsing, consolidation crystallization, adaptive forgetting, chronos buckets, and confidence-weighted consensus.These mechanisms operate as deterministic, dependency-free modules behind FLUCTLIGHT FABRIC=1.
  • Recall Fabric mechanisms: On ingest, Fabric indexes engrams and CHORUS traces for lattice and phase scoring, then reranks hybrid and CHORUS shortlists before returning recall hits.The recall path applies Fabric after shortlist generation, rather than replacing hybrid or CHORUS retrieval.
  • Validation scope: Synthetic property tests cover CRT round-trip, bind/unbind invertibility, Hamming–cosine rank agreement, and decay monotonicity, but these mechanisms lack standalone peer-benchmark validation.The paper distinguishes this internal mechanism testing from the headline results reported in Section 4.

Artifacts

FluctlightDB is released under MIT licensing with benchmark harnesses, frozen metrics, a preprint DOI, and a public source repository. Figure 5 documents the Recall Fabric pipeline used in frozen CHORUS runs across LoCoMo, BEIR, and FAMB.

  • Implementation artifacts: The MIT-licensed repository includes harnesses for LoCoMo, provenance conflicts, LongMemEval, BEIR, and agent-memory benchmarks, plus frozen end-to-end metrics.The frozen metrics include a paper end-to-end script and a dated JSON artifact.
  • Release access: The release provides a Zenodo preprint DOI and a public GitHub source repository.The supplied artifact record lists DOI 10.5281/zenodo.20949890 and the FluctlightDB GitHub source.
  • Recall Fabric: Figure 5 shows Recall Fabric composing with CHORUS recall in frozen LoCoMo, BEIR, and FAMB runs.The pipeline is identified as FLUCTLIGHT FABRIC=1.
Loading 2608.12365v1…