PaperlayerView exact revision on arXiv

Source-linked AI summary

Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects

Chris Latimer, Nicoló Boschi, Andrew Neeser, Chris Bartholomew, Gaurav Srivastava, Xuan Wang, Naren Ramakrishnan

arXiv:2512.12818v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Agent-memory systems often retrieve snippets for otherwise stateless models, yet struggle to distinguish evidence from inference and organize information over long horizons. HINDSIGHT structures memory into four networks and three operations for retaining, recalling, and reflecting. It improves results across long-horizon conversational benchmarks, including a rise from 39.0% to 83.6% LongMemEval overall accuracy with an open-source 20B backbone.

  • Problem

    Current agent-memory systems use retrieved conversational snippets but still struggle to distinguish evidence from inference and organize information over long horizons.

    Source: §1 · p. 1

  • Method

    HINDSIGHT organizes agent memory into world, experience, opinion, and observation networks, with retain, recall, and reflect operations for structured memory and preference-conditioned reasoning.

    Source: §1 · p. 2

  • Results

    HINDSIGHT improves overall accuracy on LongMemEval and LoCoMo; with an open-source 20B backbone, LongMemEval rises from 39.0% to 83.6%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported experiments indicate that this structured memory architecture improves performance in practice while separating evidence from synthesized summaries and beliefs.

    Source: §9 · p. 21

    The evaluation focuses on conversational settings; broader tool-use and workflow integration and more diverse benchmarks remain future directions.

    Source: §9 · p. 21

Abstract

from arXiv · show

Agent memory has been touted as a dimension of growth for LLM-based applications, enabling agents that can accumulate experience, adapt across sessions, and move beyond single-shot question answering. The current generation of agent memory systems treats memory as an external layer that extracts salient snippets from conversations, stores them in vector or graph-based stores, and retrieves top-k items into the prompt of an otherwise stateless model. While these systems improve personalization and context carry-over, they still blur the line between evidence and inference, struggle to organize information over long horizons, and offer limited support for agents that must explain their reasoning. We present Hindsight, a memory architecture that treats agent memory as a structured, first-class substrate for reasoning by organizing it into four logical networks that distinguish world facts, agent experiences, synthesized entity summaries, and evolving beliefs. This framework supports three core operations -- retain, recall, and reflect -- that govern how information is added, accessed, and updated. Under this abstraction, a temporal, entity aware memory layer incrementally turns conversational streams into a structured, queryable memory bank, while a reflection layer reasons over this bank to produce answers and to update information in a traceable way. On key long-horizon conversational memory benchmarks like LongMemEval and LoCoMo, Hindsight with an open-source 20B model lifts overall accuracy from 39% to 83.6% over a full-context baseline with the same backbone and outperforms full context GPT-4o. Scaling the backbone further pushes Hindsight to 91.4% on LongMemEval and up to 89.61% on LoCoMo (vs. 75.78% for the strongest prior open system), consistently outperforming existing memory architectures on multi-session and open-domain questions.

1 INTRODUCTION

Existing agent-memory systems struggle to preserve long-term information, distinguish observations from beliefs, and maintain consistent preferences. HINDSIGHT organizes memory and reasoning into explicit networks and operations, with strong results on long-horizon conversational benchmarks.

Motivation

  • Current memory systems struggle to preserve long-term information across sessions, distinguish observed facts from beliefs, and sustain preference consistency.Although dedicated architectures improve personalization and context carry-over, they can still blur evidence and inference, struggle to organize information over long horizons, and provide limited support for explaining answers.Source: §1 · pp. 1–2

Architecture

  • HINDSIGHT combines a structured memory bank that accumulates what an agent has seen, done, and decided with reasoning that uses memory to answer questions and update beliefs.TEMPR implements retain and recall over a temporal, entity-aware memory graph; CARA implements reflect and maintains an evolving opinion network using configurable behavioral parameters.Source: §1 · p. 2

Contributions

  • Separate memory networks and retain, recall, and reflect operations support epistemic clarity, traceability, and stable, auditable belief updates.The architecture turns conversational transcripts into a structured, queryable memory bank and supports reasoning over it.Source: §1 · p. 2

Empirical evaluation

  • With an open-source 20B backbone, HINDSIGHT raises LongMemEval overall accuracy from 39.0% to 83.6% and LoCoMo overall accuracy from 75.78% to 85.67%.Larger backbones reach 91.4% and 89.61%, respectively, matching or surpassing prior memory systems and frontier-backed full-context baselines.Source: §1 · p. 2

2 RELATED WORK

Related work spans context management, structured memory architectures, evaluation benchmarks, and cognitive frameworks. The reviewed systems and frameworks leave open challenges around distinguishing evidence from beliefs, organizing memory over long horizons, and evolving subjective beliefs.

2.1 MEMORY ARCHITECTURES AND SYSTEMS

  • Memory architectures range from tiered context systems to temporal knowledge graphs, atomic notes, production-oriented retrieval, reinforcement-trained operations, multimodal memory, and spatial scene graphs.The cited systems differ in whether they separate objective facts from beliefs, model behavioral profiles, or support belief evolution; Table 1 compares architectural features.Source: §2.1 · p. 3, Table 1 · p. 3

2.2 BENCHMARKS AND COGNITIVE FOUNDATIONS

  • LoCoMo evaluates dialogues spanning up to 35 sessions, LongMemEval covers conversations up to 1.5 million tokens, and MemoryBench tests continual learning from feedback.The passage reports that existing systems struggle with long-range temporal and causal reasoning and with using feedback without forgetting.Source: §2.2 · p. 3
  • Cognitive frameworks organize memory by source, form, operations, or facets such as storage and persistence, while reviewed work highlights limits in implicit memory and knowledge integration.The section frames HINDSIGHT against gaps in distinguishing observations from beliefs, maintaining stable behavioral profiles, and evolving subjective beliefs over time.Source: §2.2 · pp. 3–4

3 HINDSIGHT OVERVIEW

HINDSIGHT combines a four-network memory bank with retain, recall, and reflect operations to support factual recall and preference-conditioned reasoning. Its design separates evidence, summaries, and beliefs while organizing memories for temporal and entity-aware retrieval.

3 HINDSIGHT OVERVIEW

  • HINDSIGHT combines long-term factual recall with preference-conditioned reasoning through a memory bank that accumulates interactions and supports answering questions, forming opinions, and updating beliefs.Source: §3 · p. 4

3.1 FOUR-NETWORK MEMORY ORGANIZATION

  • The four networks distinguish objective world facts, the agent’s own experiences, confidence-scored subjective opinions, and preference-neutral entity summaries derived from facts.Together, they represent world knowledge, personal history, beliefs, and synthesized entity profiles.Source: §3.1 · p. 4

3.2 THREE CORE OPERATIONS

  • Retain converts input into temporally ranged facts, canonical entities, and graph links; recall retrieves relevant memories within a token budget; reflect generates a profile-shaped response and may update opinions.Recall combines semantic, keyword, graph, and temporal search; new evidence can also update existing opinions through reinforcement.Source: §3.2 · pp. 4–5

3.3 COMPONENT ARCHITECTURE

  • TEMPR builds the four-network graph and provides agent-oriented retrieval, while CARA applies behavioral parameters to reasoning and maintains the dynamic opinion network.The architecture overview depicts input passing through retain and recall before CARA’s preference-conditioned reflect operation.Source: §3.3 · pp. 5–6, Figure 2 · p. 5

3.4 DESIGN PRINCIPLES

  • Design principles include separating evidence, beliefs, and summaries; storing occurrence and mention times; and linking memories by temporal, semantic, entity, and causal relations.Behavioral parameters and confidence updates are intended to support stable perspectives while allowing beliefs to evolve with new evidence.Source: §3.4 · p. 6

4 TEMPR: RETAIN AND RECALL

TEMPR turns conversations into structured, temporal, entity-aware memories, then retrieves variable-sized context through complementary search channels, fusion, reranking, and token-budget filtering. Its design combines narrative facts and graph links to support multi-hop retrieval and entity summaries.

4.1.1 MEMORY ORGANIZATION

  • Each extracted fact is assigned to one of four networks and stored as a self-contained node with narrative text, an embedding, temporal metadata, and optional opinion confidence.Source: §4.1.1 · pp. 6–7

4.1.2 LLM-BASED NARRATIVE FACT EXTRACTION

  • LLM extraction produces 2–5 comprehensive facts per conversation, each intended to cover an entire exchange and preserve cross-turn context and reasoning.Figure 3 contrasts fragmented facts with narrative extraction.Source: §4.1.2 · p. 7, Figure 3 · p. 7

4.1.3 ENTITY RESOLUTION AND LINKING

  • Entity resolution maps mentions to canonical entities using string similarity, co-occurrence, and temporal proximity, enabling links that support multi-hop discovery across distant conversations.Source: §4.1.3 · p. 8

4.1.4 LINK TYPES AND GRAPH STRUCTURE

  • The graph links memories by identity, temporal proximity, semantic similarity, and LLM-extracted causality, enabling discovery by more than surface form.Causal links are upweighted during traversal to favor explanatory connections.Source: §4.1.4 · p. 8

4.1.5 THE OBSERVATION PARADIGM

  • Entity observations summarize underlying facts into concise, preference-neutral profiles; unlike opinions, they have no confidence scores and are regenerated when facts change.Source: §4.1.5 · p. 9

4.2 RECALL: AGENT-OPTIMIZED RETRIEVAL ARCHITECTURE

  • TEMPR combines complementary retrieval channels with reciprocal-rank fusion and neural reranking to surface direct and indirect memories within a specified token budget.Callers can trade off retrieval latency and coverage.Source: §4.2 · p. 9

4.2.1 AGENT-OPTIMIZED RETRIEVAL INTERFACE

  • Callers can specify retrieval effort and context size, while four parallel channels capture semantic, lexical, graph-based, and temporal relevance.Temporal retrieval normalizes query dates and matches them against memory occurrence intervals.Source: §4.2.1 · p. 9, §4.2 · p. 9, §4.2.2 · pp. 9–10

4.2.3 RECIPROCAL RANK FUSION (RRF)

5 CARA: COHERENT ADAPTIVE REASONING AGENTS

CARA implements Hindsight’s reflect operation by using bank-specific behavioral profiles to form and update opinions from retrieved memories. Its mechanisms keep opinions responsive to new evidence and background information coherent over time.

5 CARA: COHERENT ADAPTIVE REASONING AGENTS

  • CARA implements reflect with four capabilities: disposition-profile integration, Hindsight memory integration, opinion formation and reinforcement, and background merging with conflict resolution.The agent uses its profile when forming and updating opinions over world and experience networks.Source: §5 · p. 11

5.1 MOTIVATION

  • With identical remote-work facts, different behavioral profiles weight flexibility versus structure differently and can produce contrasting opinions.CARA provides a mechanism to specify profiles and shape opinion formation and updating accordingly.Source: §5.1 · p. 12

5.2.1 DISPOSITION PARAMETERS

  • The disposition space uses skepticism, literalism, and empathy dimensions, while β controls how strongly preferences shape opinion formation.The dimensions are ordered from trusting to skeptical, flexible to literal, and detached to empathetic; β ranges from 0 to 1.Source: §5.2.1 · p. 12

5.3 BANK PROFILE STRUCTURE

  • Each bank profile combines the agent’s name, behavioral profile, and first-person background, with numeric preferences verbalized for system messages.The verbalization explicitly describes the intended reasoning and communication style.Source: §5.3 · p. 12, §5.3.1 · p. 13

5.4.2 OPINION FORMATION PROCESS

  • For subjective queries, CARA retrieves relevant facts, experiences, and opinions, conditions generation on the bank profile, then stores candidate opinion updates with confidence scores.An opinion records its judgment and formation context; confidence represents belief strength and supports later reinforcement or revision.Source: §5.4.2 · pp. 13–14, §5.4.1 · p. 13

5.5 OPINION REINFORCEMENT

  • As new facts arrive, CARA assesses their relationship to existing opinions and adjusts confidence so small evidence changes yield small shifts while repeated reinforcement or strong contradictions can move beliefs substantially.Contradictions can also prompt a more nuanced opinion text; the profile may influence update speed.Source: §5.5 · pp. 14–15

5.6 BACKGROUND MERGING

  • CARA merges new biographical snippets into a concise first-person background, resolving direct conflicts in favor of new information when appropriate and appending non-conflicting details.Incoming snippets are normalized into first person before merging.Source: §5.6 · p. 15

5.7.1 EXAMPLE: OPINION EVOLUTION

  • In the example, opinions form trajectories: new evidence can raise confidence from 0.70 to 0.85, then lower it to 0.55 and qualify the belief.CARA’s opinions begin with preference-conditioned formation and are subsequently adjusted as evidence accumulates.Source: §5.7.1 · p. 16

6 PUTTING IT ALL TOGETHER: UNIFIED HINDSIGHT ARCHITECTURE

Hindsight composes TEMPR’s retain and recall operations with CARA’s reflect operation in one system. The resulting loop structures conversational input into memory, retrieves relevant content, and uses it for preference-conditioned reasoning and belief updates.

6.1 INTEGRATION: RETAIN, RECALL, REFLECT

  • Hindsight integrates TEMPR and CARA into a three-operation loop: experiences are retained, recalled for queries, and reflected on to update beliefs and identity configuration.The system turns conversational input into a structured memory bank for preference-conditioned reasoning over time.Source: §6 · p. 16, §6.1 · pp. 16–17
  • Retain extracts temporally ranged facts, resolves entities, builds graph links, reinforces related opinions, and merges background information into the updated four-network memory.The pipeline uses LLM-powered fact extraction and entity resolution.Source: §6.1 · p. 16
  • Recall combines semantic, keyword, graph, and temporal retrieval, then fuses and reranks results while filtering them to the token budget.Returned facts are ordered by relevance and constrained by the query’s token budget.Source: §6.1 · p. 17
  • Reflect retrieves memories, loads the bank’s preferences and background, generates a conditioned response, and stores any new opinions in the opinion network.The operation returns both a response and an updated opinion network.Source: §6.1 · p. 17

7 EXPERIMENTS

The experiments test whether HINDSIGHT retains and reasons over long conversational histories, using LongMemEval and LoCoMo with a shared judging setup. Results report strong performance on both benchmarks, including gains over same-backbone full-context and prior memory-system comparisons.

7.1 DATASETS

  • The evaluation uses two benchmarks designed to test long-term memory in conversational agents.The experiments focus on memory coherence across many sessions and preference-conditioned reasoning.Source: §7.1 · p. 17, §7 · p. 17

7.1.1 LONGMEMEVAL

  • LongMemEval contains 500 questions testing information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention across two conversation scales.Its S setting has about 115,000 tokens and 50 sessions; its M setting has about 1.5 million tokens and 500 sessions.Source: §7.1.1 · p. 17

7.1.2 LOCOMO

  • LoCoMo uses 50 multi-session human-human conversations averaging 304.9 turns, 9,209.2 tokens, and 19.3 sessions, with some reaching 35 sessions.It includes images and questions about personal details, preferences, events, and distant-session context.Source: §7.1.2 · p. 18

7.2 EVALUATION METRICS

  • An LLM judge scores each response for correctness and completeness, including abstention when LongMemEval lacks the requested information.For multi-hop or temporal questions, the judge also checks appropriate use of retrieved memories and temporal context.Source: §7.2 · p. 18

7.3 EXPERIMENTAL SETUP

  • GPT-OSS-20b powers fact extraction and reflection, while GPT-OSS-120b serves as the judge with temperature 0.0 for consistent scoring.Retrieval uses semantic, keyword, graph, and temporal recall fused with Reciprocal Rank Fusion and neural reranking; experiments use neutral behavioral profiles and low bias strength.Source: §7.3 · p. 18

7.4 RESULTS ON LONGMEMEVAL

  • 91.4% overall accuracy on LongMemEval with Gemini-3 Pro is the best reported result, while OSS-20B reaches 83.6% versus 39.0% for same-backbone full context.HINDSIGHT with OSS-120B reaches 89.0%; OSS-20B also surpasses Full-context GPT-4o (60.2%), Zep+GPT-4o (71.2%), and Supermemory+GPT-4o (81.6%).Source: Table 3 · p. 19, §7.4 · pp. 19–20

7.5 RESULTS ON LOCOMO

  • On LoCoMo, HINDSIGHT reaches 89.61% overall accuracy with Gemini-3 Pro, close to Backboard’s reported 90.00%, and has the highest Open Domain score at 95.12%.OSS-20B and OSS-120B reach 83.18% and 85.67%, respectively; the paper describes these results as transferring to realistic, multi-session human conversations.Source: §7.5 · p. 20, Table 4 · p. 20

8 CODE AVAILABILITY

The paper releases HINDSIGHT’s implementation and materials for running and inspecting its benchmark evaluations.

8 CODE AVAILABILITY

  • The repository includes the memory architecture, benchmark scripts and configurations, and utilities for fact extraction, graph construction, and retrieved-memory analysis.An interactive viewer provides per-question results, retrieved contexts, model and judge configurations, and aggregate metrics for reported variants.Source: §8 · p. 20

9 CONCLUSION

The conclusion presents HINDSIGHT as a first-class memory substrate with explicit networks and operations, and identifies extensions for broader settings and memory management.

9 CONCLUSION

  • HINDSIGHT separates world, experience, observation, and opinion networks and uses retain, recall, and reflect to distinguish evidence, summaries, and beliefs.The authors report that this structure matters in practice and improves performance.Source: §9 · pp. 20–21
  • Jointly optimizing fact extraction, graph construction, and retrieval is proposed as future work, particularly for robustness and efficiency in noisy, open-domain settings.The authors also identify reinforcement learning as a way to explore the interplay among retain, recall, and reflect.Source: §9 · p. 21
  • The authors plan to extend evaluation beyond conversational benchmarks and integrate richer tool-use and workflow orchestration.They also identify controlled forgetting, time-aware belief revision, and privacy-aware memory management as future directions.Source: §9 · p. 21

A SYSTEM PROMPTS

The appendix supplies system prompts for extracting facts, forming opinions, synthesizing entity observations, and evaluating responses.

A SYSTEM PROMPTS

  • The appendix contains complete prompt templates for fact extraction, opinion formation, observation generation, and evaluation.The fact-extraction prompt turns transcripts into structured narrative facts with temporal ranges, entities, and causal relationships.Source: A SYSTEM PROMPTS · p. 23, §A.1 · p. 23

FACT FORMAT - ALL FIVE DIMENSIONS REQUIRED - MAXIMUM VERBOSITY

  • Fact extraction instructions require detailed event descriptions and temporal information, including the day of the week and a specified date format.The prompt says not to summarize or omit details.Source: FACT FORMAT - ALL FIVE DIMENSIONS… · p. 23
  • Each fact records where and why, who was involved, plus structured type, kind, entity, date-range, and location fields.Assistant facts must also include the user request that triggered them.Source: FACT FORMAT - ALL FIVE DIMENSIONS… · p. 23

A.2 OPINION FORMATION PROMPT (CARA)

  • Opinion formation extracts new perspectives during reflection and writes them in first person with reasoning and a confidence score from 0.0 to 1.0.It excludes statements that merely report insufficient information or inability to answer.Source: §A.2 · p. 24

A.3 OBSERVATION GENERATION PROMPT (TEMPR)

  • Observation generation synthesizes factual, third-person entity summaries from underlying facts without behavioral-profile influence or added opinions.The instructions say to combine related facts and avoid judgments or interpretations.Source: §A.3 · pp. 24–25
  • Observations cover an entity’s identity, characteristics, roles, relationships, and activities; conflicting facts should be resolved using the most recent or best-supported account.The prompt requests three to seven observations, or fewer when facts are sparse.Source: §A.3 · p. 25

A.4 LONGMEMEVAL JUDGE PROMPTS

LongMemEval uses different judge prompts for different question types. Each prompt applies a type-specific correctness rule and requests a yes-or-no judgment.

A.4.1 SINGLE-SESSION AND MULTI-SESSION QUESTIONS

  • For single- and multi-session questions, equivalent answers and responses containing all intermediate steps count as correct, but incomplete subsets do not.The prompt provides question, correct answer, and model response as inputs.Source: §A.4.1 · p. 25

A.4 LONGMEMEVAL JUDGE PROMPTS

  • Each judge prompt ends by requiring a boxed yes-or-no judgment, with the question and response supplied in a type-specific template.The introductory passage says different prompts are used for different question types.Source: §A.4.1 · p. 25, §A.4 · p. 25, §A.4.2 · p. 26

A.4.2 TEMPORAL REASONING QUESTIONS

  • For temporal questions, off-by-one errors in durations such as days, weeks, or months are not penalized.The judge compares the question, correct answer, and model response.Source: §A.4.2 · p. 26

A.4.3 KNOWLEDGE UPDATE QUESTIONS

  • For knowledge-update questions, a response remains correct if it includes previous information alongside the required updated answer.The prompt supplies the question, correct answer, and model response.Source: §A.4.3 · p. 26

A.4.4 PREFERENCE QUESTIONS

  • For preference questions, a response is correct when it recalls and uses the user’s personal information correctly; it need not address every rubric point.The judge receives a question, rubric, and model response.Source: §A.4.4 · p. 26

A.4.5 ABSTENTION QUESTIONS

  • For abstention questions, a response is correct if it identifies the question as unanswerable, including by noting incomplete information.The prompt supplies the question, explanation, and model response.Source: §A.4.5 · p. 27

A.5 STRUCTURED OUTPUT SCHEMAS

Hindsight uses Pydantic models to structure and validate information extracted through LLM calls. The schemas specify fields for facts, opinions, and entity observations.

A.5 STRUCTURED OUTPUT SCHEMAS

  • Pydantic models enforce structured output from LLM calls to support reliable parsing and validation of extracted information.Source: §A.5 · p. 27

A.5.1 FACT SCHEMA

  • The fact schema requires a complete description, temporal and location details, people and relationships, and motivations, and classifies facts as world, experience, or opinion.Source: §A.5.1 · p. 27
  • Optional fact fields capture start and end times, mention time, entities, and causal relations; relations identify a target fact, type, and strength from 0.0 to 1.0.Entities are represented by their named text; relation types include causes, caused_by, enables, and prevents.Source: §A.5.1 · p. 27

A.5.2 OPINION SCHEMA

  • The opinion schema stores a first-person opinion statement, confidence from 0.0 to 1.0, and reasoning for how the opinion was formed.Source: §A.5.2 · p. 27

A.5.3 OBSERVATION SCHEMA

  • The observation schema stores factual statements about an entity in a list of observations.Source: §A.5.3 · p. 28
Loading 2512.12818v1…