PaperlayerView exact revision on arXiv

Source-linked AI summary

Mnemon: Raw Records, Fast Judgments, Slow Thoughts

Guangren Wang

arXiv:2609.36059v1cs.CLcs.AIcs.IR

TL;DR

Long-term memory systems commonly rewrite conversations before a question reveals which details will matter. Mnemon keeps raw dated records and divides memory work between LLM planning and decision-model judgments, with a background index linked to the records. In a public re-evaluation, it ranks first on LoCoMo and second on LongMemEval-S with gpt-4.1-mini answering, using under 4k context tokens per question.

  • Problem

    Most long-term memory systems rewrite conversations into facts, graphs, or typed memories at write time, although what matters may only become clear when a question is asked.

    Source: §1 · p. 1

  • Method

    Mnemon keeps raw dated records, assigns search planning and answer composition to an LLM and record judgments to a decision model, and consolidates records into a linked background index.

    Source: §1 · pp. 1–2, §4.1 · pp. 4–5

  • Results

    91.7% on LoCoMo and 83.8% on LongMemEval-S: with gpt-4.1-mini answering, Mnemon ranks first and second among 14 re-evaluated systems, respectively, from under 4k context tokens per question.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Because records are not interpreted at write time, the same agent can read any store that returns dated records.

    Source: §1 · p. 2

    The evaluation covers conversational memory in one Source, and the benchmarks were not held out because development and consolidation design followed results on some of them.

    Source: §7 · p. 11

Abstract

from arXiv · show

Long-term memory lets an LLM assistant use a history it can no longer reread, and most memory systems build it by rewriting conversations into facts, graphs or typed memories at write time. We argue that the work of memory divides, as thinking does, into two systems. Most of it is fast System 1 work: many small, independent yes/no judgments about records, such as whether a record is needed or no longer current, which a decision model makes by the dozen in a third of a second. Only a little is slow System 2 work: writing a few search queries, naming what the reply needs and composing the answer, which an LLM does well but slowly. We present Mnemon, a memory agent built on this division. It keeps conversations as raw, dated records; an LLM (System 2) plans searches over them, a decision model, Jev (System 1), judges what the searches return, and rules with explicit budgets turn the judgments into a small View for an unchanged answering model. A background pass consolidates each record once into topic timelines, value histories and standing instructions linked to the records, so that questions about a whole conversation reach evidence their own searches miss. Because nothing is decided about a record when it is written, the same agent can read any store that returns dated records. With gpt-4.1-mini answering, as in a public re-evaluation of 14 systems, Mnemon scores 91.7% on LoCoMo, the highest among them, and 83.8% on LongMemEval-S, from under 4k tokens of context per question, with the lowest effective cost index on LoCoMo. With a reasoning model answering, it reaches 92.2% on LoCoMo and 94.4% on LongMemEval-S, the latter on par with the best published results. From 100K to 10M tokens of history on BEAM, its cost per question grows by a factor of 1.11. On the same records, Jev separates gold evidence better than two LLMs and is 3-11 times faster.

1 Introduction

Most memory systems rewrite conversations at write time, before the question reveals what matters. Mnemon instead keeps raw records and divides read-time work between an LLM planner and a decision model, with a background index supporting whole-conversation queries.

1 Introduction

  • Raw records avoid write-time extraction that processes every message before the question is known, may lose or distort relevant information, and requires new schemas for new data types.Documents, tasks, and logs are examples of data types that would need new extraction schemas.Source: §1 · p. 1
  • A background pass consolidates records into topic timelines, value histories, and standing instructions linked back to their source records.This supports whole-conversation questions whose evidence ordinary searches may miss.Source: §1 · p. 2
  • Mnemon assigns search planning and answer composition to an LLM, while a decision model makes explicit yes/no judgments about records under budgeted rules.The judgments include whether a record is needed, outdated, or satisfies a reply need.Source: §1 · pp. 1–2
  • 91.7% on LoCoMo and 83.8% on LongMemEval-S: Mnemon ranks first and second, respectively, among 14 re-evaluated systems, using under 4k context tokens per question.It also has the lowest effective cost index on LoCoMo.Source: §1 · p. 2
  • With a reasoning model, Mnemon scores 92.2% on LoCoMo and 94.4% on LongMemEval-S, the latter on par with the best published results.From BEAM-100K to BEAM-10M, cost per question grows by a factor of 1.11.Source: §1 · p. 2
  • On the same records, Jev separates gold evidence better than two LLMs, with AUC 0.942 against 0.900 and 0.853, and runs 3–11 times faster.Figure 1’s comparison also reports Mnemon as the only system above 80% on both benchmarks that sends fewer than 4k tokens per question.Source: §1 · p. 2

2 Related Work

Prior memory systems variously page context, extract or restructure memories at write time, or retrieve and rerank raw history. Mnemon instead judges raw records at read time, while the evaluation literature motivates a shared re-evaluation protocol.

2 Related Work

  • Mnemon retains raw records as evidence, reasons at read time, and uses consolidation only for an index that points back to those records.Other systems described here extract facts, build knowledge graphs or linked notes, organize typed stores, or consolidate episodes.Source: §2 · p. 2
  • Mnemon combines BM25 and dense retrieval with reciprocal rank fusion and hypothetical-document queries, while Jev reranks by judging explicit propositions that drive actions.The passage relates these components to retrieval-augmented generation, ReAct, and Self-RAG.Source: §2 · p. 2
  • Figure 1 plots accuracy against context per question for Mnemon and 14 re-evaluated systems using gpt-4.1-mini; upper-left is better, and dashed lines mark equal effective cost index.The figure caption identifies the comparison axes and the direction of preference.Source: Figure 1 · p. 3
  • Unlike write-path fast/slow memory systems, Mnemon applies the division on the read path: System 1 judges raw records after a question is known, and System 2 plans searches and composes the answer.The related systems include D-Mem, DCPM, Engram, and Jev-Mem.Source: §2 · p. 3
  • Because protocols and LLM graders vary, several self-reported scores fell by 10–40 points when OmniMemEval re-evaluated 14 systems under one protocol.The paper follows that protocol and lists published claims separately.Source: §2 · p. 3

3 Problem Setting

The setting treats memory as selecting a bounded View from dated history in response to the current dialogue. It formalizes why write-time transformations can lose question-relevant information and separates read-time judgments from open-ended generation.

3 Problem Setting

  • The main agent receives a bounded View before replying and is otherwise unchanged; in the benchmarks, each question arrives in a fresh session and depends on that View alone.The memory agent does not edit the main agent’s reply.Source: §3 · p. 3
  • History is a sequence of dated records, while the current dialogue and desired reply determine a View constrained by a length budget.A write-time memory instead reads a transformation fixed before the question is known.Source: §3 · p. 3
  • A write-time transformation of history cannot add information about answers and may lose information about questions not yet asked; an index kept beside raw records loses none and can ease retrieval.The View still has to fit its budget, so selecting relevant records remains read-time work.Source: §3 · pp. 3–4
  • System 1 evaluates explicit yes/no propositions about individual records or index items, while System 2 handles open-ended tasks such as writing unseen-record queries and composing answers.Independent judgments can be batched in waves, so System 1 time grows with the number of waves rather than judgments.Source: §3 · p. 4
  • A Source need only expose a search route returning dated records, allowing the same Strategy to publish a View from a journal, documents, or tasks.The paper’s evaluation stores all memory in one journal Source.Source: §3 · p. 4
  • The effective cost index prices a wrong answer as one full-context repair and expresses expected cost in units of full-context cost; lower is better.The paper compares systems using this index because their read-time and write-time costs are not published.Source: §3 · p. 4

4 Design

Mnemon separates turn-time search planning and record judgments from slower background consolidation, then uses explicit rules to provide the unchanged answering agent a bounded View.

4.1 Memory: Raw Records and a Consolidated Index

  • Mnemon stores dated session records without write-time interpretation, then builds a linked background index of topic events, changing values, and standing instructions.The index points back to records rather than replacing them; background organization can lag or fail without losing the underlying evidence.Source: §4.1 · pp. 4–5

4.2 Reading with Two Systems

  • System 2 plans searches and reply needs; retrieval pools fused BM25 and embedding results, which Jev screens and judges for usefulness, currency, and need satisfaction.Jev also judges linked index items; counting, arithmetic, date comparisons, and multi-hop reasoning remain System 2 work.Source: §4.2 · pp. 5–6

4.3 Rules and Budgets

  • Rules use Jev’s yes/no decisions and rankings to trigger further searches for open needs, then assemble an index-first View while excluding records judged no longer current.Budgets include at most three needs, three rounds, a pool of 48 records, and a View of 16 records and 12,000 characters.Source: §4.3 · p. 6
  • Because rules use judgment ordering and the 1/2 threshold, monotonic score recalibration leaves the loop’s actions and resulting View unchanged.A decision model can be replaced without retuning rules if it orders and decides alike.Source: §4.3 · p. 6
  • With bounded rounds and needs, critical-path work is history-independent except for journal reads, and turn token use remains bounded.Only journal-read latency depends on the number of records; those reads spend no model tokens.Source: §4.3 · p. 6

5 Implementation

Mnemon is implemented as a replica that runs a turn-level control loop, processes a bounded record pool, and publishes a View to the main instance.

5 Implementation

  • The replica runs one job per user turn and publishes its View to the main instance, with upstream DSH unmodified.It uses a second DSH instance with the same Sources and its own Strategy and control loop; the implementation adds 115 changed lines plus new plugins.Source: §5 · p. 6
  • The control loop plans five searches and up to three needs, pools at most 48 records, judges them for up to three rounds, then composes an index-first View.The algorithm judges linked index items before composing the View within its budgets.Source: §5 · p. 7
  • Journal records are chunks of about 600 characters; a roughly 106k-token LongMemEval-S history becomes about 1,000 records in under a second.For BEAM-10M, the journal is sized for up to 200,000 records per conversation and keeps its search index between searches.Source: §5 · p. 7
  • Background consolidation reads batches of about 10,000 characters and uses DeepSeek-V4.1-Flash to fold them into an index; Jev 1.13 judges up to 40 records per call.For very long histories, each fold considers up to 120 topics, 240 values, and 60 instructions; index items are embedded as written so no View waits for them.Source: §5 · p. 7

6 Evaluation

The evaluation tests Mnemon’s accuracy, context and cost, robustness to stronger answering models and larger histories, and the contributions of record judging and consolidation. Across these tests, Mnemon performs strongly on standard benchmarks with small answering contexts, while its results vary by task and its largest-history searches expose a scaling bottleneck.

6 Evaluation; 6.1 Setup

  • The evaluation uses four public benchmarks and compares Mnemon with systems re-evaluated under a shared protocol as well as separately reported published claims.The setup specifies benchmark sizes and graders, model settings, baselines, and cost accounting, including a one-time consolidation cost per history.Source: §6 · p. 7, §6.1 · pp. 7–8

6.2 Accuracy at Small Context

  • 91.7% on LoCoMo and 83.8% on LongMemEval-S place Mnemon first and second, respectively, among 14 re-evaluated systems, using about 3.8k answering tokens per question.It has the lowest effective cost index on LoCoMo and the second lowest on LongMemEval-S; by question type, it leads on all LoCoMo types except open-domain and leads on LongMemEval-S knowledge updates.Source: Table 1 · p. 8, §6.2 · p. 8

6.3 A Stronger System 2

  • 92.2% on LoCoMo and 94.4% on LongMemEval-S are Mnemon’s scores with DeepSeek-V4.1-Flash planning and answering, reaching or matching leading published results.The stronger System 2 adds 9.6 points on LongMemEval-S under the gpt-4.1-mini grader, especially on temporal-reasoning and multi-session questions.Source: §6.3 · p. 9

6.4 Other Benchmarks and Scale

  • A 1.11-fold cost increase from BEAM-100K to BEAM-10M accompanies 80 times as many records, while critical-path work stays about the same.Mnemon ranks eighth of 13 on HaluMem and tenth of 12 on BEAM-100K; searches score every record, making reads on the largest BEAM-10M history slower.Source: §6.4 · p. 9, Table 4 · p. 10

6.5 System 1 Against LLMs

  • AUC 0.942 overall for Jev exceeds DeepSeek’s 0.900 and gpt-4.1-mini’s 0.853 on separating gold evidence from the same records.A Jev call on 24 records takes 0.34 seconds, versus 1.05–3.74 seconds for the LLMs’ one-proposition calls.Source: §6.5 · p. 10, Figure 3 · p. 11

6.6 Judging Raw Records or Organizing Them

  • 84.4% for Jev-Mem is 7.3 points below Mnemon under the shared evaluation protocol, though the systems differ in more than where System 1 works.Jev-Mem organizes turns at write time and spends 9.5 times as much per history; the authors caution that this comparison is not an ablation.Source: §6.6 · pp. 10–11

6.7 What Consolidation Adds

  • Removing the consolidated index lowers accuracy on LoCoMo, LongMemEval-S and both BEAM tiers, with gains of 2.5, 4.4, 6.2 and 5.9 points, respectively, under gpt-4.1-mini.Gains appear where evidence is not directly indicated by a question; consolidation multiplies per-question cost by 1.17–1.37.Source: §6.7 · p. 11

7 Limitations

The evaluation has limits in benchmark development, run reproducibility, and model grading, and tests only conversational memory in one Source. Mnemon also depends on hosted Jev, while latency and retention of raw records and the index raise deployment considerations.

Evaluation scope

  • Mnemon was developed on LoCoMo and LongMemEval-S, and consolidation was designed after HaluMem and BEAM-100K results were seen, so no benchmark was held out.Each configuration ran once per setting; two identical runs of an earlier configuration differed by up to 2.6 points, and graders differed by 1–2 points, or more on HaluMem.Source: §7 · p. 11
  • Evaluation covers conversational memory in one Source; stores of documents, tasks, or logs remain untested.The design admits those other store types, but the evaluation did not test them.Source: §7 · p. 11

Dependence on Jev

  • Mnemon relies on hosted, proprietary Jev; no substitute was tested in the full system.LLMs can stand in with lower discrimination at several times the latency and price.Source: §7 · p. 11

Latency and privacy

  • A question took 10–15 s end to end at the median in runs sharing one laptop and public model APIs.Raw records retain material an extractor would discard, and the index repeats some of it, so deletion and retention policies must cover both.Source: §7 · p. 12

8 Conclusion

Mnemon divides memory work between an LLM that plans searches and composes replies and a decision model that judges records, with consolidation running in the background. It reports strong benchmark results and nearly flat question cost over histories from 100K to 10M tokens.

8 Conclusion

  • Mnemon divides read-time memory work between an LLM for search planning and reply composition and a decision model for many small record judgments.A background pass consolidates each record once; nothing about a record is decided when it is written.Source: §8 · p. 12
  • The same agent can read any store that returns dated records, and its raw records support longer histories and stronger answering models without rewriting.The paper concludes that judging raw records once the question is known was more accurate than organizing them in advance with the same model.Source: §8 · p. 12
  • With a small answering model, Mnemon ranked first on LoCoMo and second on LongMemEval-S in a public re-evaluation; reasoning-model results were on par with the best published LongMemEval-S results.Question cost grew by a factor of 1.11 from 100K to 10M tokens of history.Source: §8 · p. 12

Related papers

Loading 2609.36059v1…