Source-linked AI summary
REMem: Reasoning with Episodic Memory in Language Agent
Yiheng Shu, Saisri Padmaja Jonnalagedda, Xiang Gao, Bernal Jiménez Gutiérrez, Weijian Qi, Kamalika Das, Huan Sun, Yu Su
TL;DR
Language-agent memory remains largely semantic, limiting precise recollection and reasoning over spatiotemporally situated interaction histories. REMem builds a hybrid graph of time-aware gists and facts and uses agentic iterative retrieval; across four benchmarks, it improves episodic recollection and reasoning over existing systems while refusing unanswerable questions more robustly.
Problem
Current language-agent memory remains mainly semantic, while episodic recollection and multi-step reasoning over situated interaction histories remain challenging.
Method
REMem uses offline indexing to organize time-aware event gists and time-scoped facts in a hybrid memory graph, followed by agentic inference with iterative retrieval and graph exploration.
Results
Across four episodic memory benchmarks, REMem improves episodic recollection by 3.4% and episodic reasoning by 13.4% in absolute terms over current state-of-the-art systems.
Takeaways & Limitations
REMem provides a promising step toward more reliable long-horizon language agents through stronger episodic recollection, reasoning, and refusal behavior.
Takeaways & Limitations
Building memories in streaming settings remains an engineering challenge compared with offline batch indexing.
Abstract
from arXiv · showhide
Humans excel at remembering concrete experiences along spatiotemporal contexts and performing reasoning across those events, i.e., the capacity for episodic memory. In contrast, memory in language agents remains mainly semantic, and current agents are not yet capable of effectively recollecting and reasoning over interaction histories. We identify and formalize the core challenges of episodic recollection and reasoning from this gap, and observe that existing work often overlooks episodicity, lacks explicit event modeling, or overemphasizes simple retrieval rather than complex reasoning. We present REMem, a two-phase framework for constructing and reasoning with episodic memory: 1) Offline indexing, where REMem converts experiences into a hybrid memory graph that flexibly links time-aware gists and facts. 2) Online inference, where REMem employs an agentic retriever with carefully curated tools for iterative retrieval over the memory graph. Comprehensive evaluation across four episodic memory benchmarks shows that REMem substantially outperforms state-of-the-art memory systems such as Mem0 and HippoRAG 2, showing 3.4% and 13.4% absolute improvements on episodic recollection and reasoning tasks, respectively. Moreover, REMem also demonstrates more robust refusal behavior for unanswerable questions.
1 INTRODUCTION
Current language-agent memory remains largely semantic and struggles to preserve concrete experiences in spatiotemporal context. REMem frames episodic memory around recollection and multi-step reasoning, then addresses both with a time-aware hybrid memory graph and agentic retrieval.
- Motivation: Semantic memory paradigms leave language-agent memories insufficiently adaptable and grounded in specific experiences.Parametric memory, model editing, and embedding-based retrieval each remain limited in contextual or spatiotemporal grounding.
- Motivation: Existing episodic approaches lose coherent event contexts, lack explicit event modeling, or fail to integrate time, location, and participants.
- Challenges: Episodic recollection binds situational elements such as time, location, participants, and emotion to specific events.
- Challenges: Episodic reasoning performs multi-step reasoning over recollected episodes, including inter-event relations, ordinal constraints, and superlatives.
- Approach: REMem stores time-aware gists and time-scoped facts in a hybrid graph, then uses agentic retrieval and graph exploration for complex logical composition.The supported operations include time-range filtering, neighbor exploration, and ordinal constraints.
- Results: 3.4% and 13.4% absolute improvements on episodic recollection and reasoning tasks, respectively, were achieved across four benchmarks.REMem also exceeded 90% exact match on Test of Time and showed more robust refusal behavior for unanswerable questions.
2 RELATED WORK
Prior language-agent memory systems range from saved conversational memories and flat embedding retrieval to graph-based and temporal memory structures. Episodic benchmarks increasingly test multi-session and temporal reasoning, but many tasks and systems still center on retrieving relevant segments rather than reasoning across events.
- Memory systems: Non-parametric memory systems augment language models with saved memories, embedding retrieval, graphs, or memory layers for personalization and cross-session access.
- Memory systems: Flat embedding spaces provide competitive retrieval but do not explicitly encode episodic or temporal structure.
- Memory systems: HippoRAG, Graphiti, and Mem0 use associative retrieval, temporal knowledge graphs, context assembly, or conversational extraction and consolidation.
- Benchmarks: LoCoMo and REALTALK evaluate multi-session conversational memory, while Complex-TR and related temporal benchmarks test temporal reading comprehension.
- Benchmarks: Many long-horizon memory tasks remain effectively single-retrieval problems requiring agents to retrieve correct segments rather than reason across linked events.
3 METHODOLOGY
REMem uses offline indexing to build a hybrid graph of time-aware event gists and facts, followed by agentic inference that iteratively retrieves and explores this graph. The design preserves event context while supporting temporal and relational reasoning.
- Framework: REMem separates memory use into indexing and agentic inference phases.
- Indexing: Indexing extracts concise event gists and time-scoped subject-predicate-object facts, preserving temporal qualifiers and potentially contradictory historical records.
- Indexing: Gists capture participants, actions, objects, locations, intentions, and quantities in atomic event descriptions with resolved timestamps.
- Agentic inference: Agentic inference combines semantic or lexical retrieval, graph exploration, and flow control in an iterative ReAct-style process.
- Agentic inference: Retrieval and graph exploration return both gists and facts, providing complementary event-level and structured evidence.
- Agentic inference: The agent decomposes complex questions into sub-queries, then retrieves episode narratives or entity contexts filtered by temporal conditions.
4 EXPERIMENTAL SETUP
The evaluation tests recollection on synthetic and real conversational benchmarks and reasoning on temporal reading-comprehension benchmarks. It compares REMem with embedding, structure-augmented, and other memory baselines using task-specific metrics and controlled implementation settings.
- Datasets: LoCoMo and REALTALK measure episodic recollection from synthetic and real human conversations, respectively.Both include temporal and non-temporal situational questions.
- Datasets: Complex-TR and Test of Time measure episodic reasoning through temporal reading comprehension.The evaluation samples 1,000 Complex-TR queries and uses all semantic Test of Time samples.
- Baselines: Comparisons include strong embedding-model RAG systems and structure-augmented methods such as Mem0, Graphiti, and HippoRAG 2.
- Metrics: Test of Time uses exact match, while other benchmarks use token F1, BLEU-1, and LLM-as-a-judge scores.
- Implementation: REMem-I uses multi-step agentic inference, whereas REMem-S uses single-step embedding retrieval followed by generation.The selected maximum inference steps are three for recollection and five for reasoning tasks.
5 RESULTS
Across episodic recollection and reasoning evaluations, REMem generally outperforms competing memory approaches, while also offering a stronger balance for refusing unanswerable questions.
- Episodic recollection: REMem consumes 9k tokens with REMem-I and 0.9k with REMem-S per LoCoMo query, compared with 26k for Full-Context.These figures describe average inference-phase input tokens.
- Episodic recollection: Structure-augmented methods perform particularly poorly on REALTALK, where spontaneous, noisy, and less information-dense human utterances challenge memory systems.Graphiti loses coherent situational context, while HippoRAG 2 does not model temporal dimensions or events.
- Episodic reasoning: REMem shows absolute superiority on complex episodic reasoning tasks, with REMem-I exceeding REMem-S by LLM-J +7.0 and EM +20.6.The advantage is attributed to multi-step retrieval and flexible tool use.
- Episodic reasoning: REMem-I improves over Full-Context by LLM-J +8.0 and EM +13.4 on challenging reasoning tasks.It is the only method reported to exceed 90% EM on these tasks.
6 DISCUSSIONS
Ablations show that gists, facts, graph structure, and retrieval tools contribute differently to REMem, while refusal evaluation and error analysis identify both strengths and remaining failure modes.
- Ablation study: Removing gists causes the largest degradation, with LoCoMo LLM-J falling from 76.2 to 48.9.This supports gists as carriers of the main situational elements.
- Ablation study: Removing facts produces a smaller but consistent drop, including a Complex-TR LLM-J decrease from 89.6 to 87.2.Facts provide concrete anchors for connecting concepts across sessions during multi-hop reasoning.
- Ablation study: Removing synonymy edges reduces F1 and BLEU-1 on both datasets, indicating improved lexical robustness and recall from synonymic relationships.LLM-J remains almost unchanged, suggesting the core reasoning path is largely preserved.
- Refusal performance: REMem’s refusal behavior reaches 63.96% F1 by combining 73.3% precision with 56.8% recall.It produces fewer unnecessary rejections than Graphiti while identifying more unanswerable cases than Mem0.
- Error analysis: In sampled LoCoMo errors, 46% are selection or grounding errors, 19% temporal or numerical reasoning mistakes, and 18% incorrect abstentions despite retrieved evidence.Complex-TR errors most frequently involve temporal window mismatch at 42%.
- Qualitative comparison: REMem outperforms NV-Embed-v2 on disambiguation and timestamp reconciliation, whereas the embedding baseline performs better on straightforward interval calculations.The comparison highlights different strengths across episodic question types.
7 CONCLUSION
REMem combines a time-aware hybrid memory graph with agentic retrieval and reasoning, yielding broader episodic-memory performance while leaving streaming construction as an engineering challenge.
- Conclusion: REMem unifies concept-level and context-level information through a hybrid memory graph with flexible temporal awareness.The graph contains gist and phrase nodes connected by relation, context, and synonymy edges.
- Conclusion: The framework’s agentic retriever integrates retrieval and reasoning for episodic recollection and multi-step inference.Its procedures use semantic or lexical retrieval followed by targeted graph exploration and answer synthesis.
- Limitations and future work: The conclusion identifies streaming memory construction as an engineering challenge compared with offline batch indexing.Future work is framed around long-term memory in more complex environments.
- Memory construction: Gists are natural-language, timestamped atomic event descriptions that capture participants, actions, objects, locations, intentions, and quantities.Relative temporal expressions are resolved to absolute dates when applicable.
- Memory construction: Facts provide structured, time-scoped relations that connect concrete concepts to the episode-level gist representations.Context edges link each gist to phrase nodes extracted from the same source chunk.
C.1 LOCOMO
On LoCoMo and REALTALK, embedding models remain strong baselines, but REMem improves performance across relevant settings and shows different trade-offs between its inference variants.
- LoCoMo results: REMem-I outperforms NV-Embed-v2 on LoCoMo by F1 +2.8 and J +3.2, while REMem-S attains the highest J scores.REMem-S often performs better in single-session settings, whereas REMem-I is strongest on multi-session questions.
- LoCoMo scope: LoCoMo contains only 14.2% multi-session queries, so many questions may not require complex multi-step retrieval.The authors note that multi-step retrieval can instead introduce contextual noise in single-session settings.
- Baseline comparison: NV-Embed-v2 remains a strong baseline even with only the top-3 chunks, surpassing Qwen3-Embedding-8B and some structure-augmented systems.HippoRAG 2 is the stated exception among the compared configurations.
- REALTALK results: REALTALK is more challenging than LoCoMo, with consistently lower performance across methods because it contains noisier, more casual human interactions.REMem is the only method reported to surpass Full-Context, particularly on temporal-reasoning tasks.
C.3 COMPLEX-TR
On Complex-TR, REMem is competitive with strong retrieval baselines and benefits from agentic retrieval and improved reasoning prompts, although temporal prompt methods remain better on some time-to-event cases.
- Comparison with baselines: NV-Embed-v2 w/ TISER outperforms REMem-I on time-to-event questions by J +1.4 but trails REMem by J −1.3 on average.Time-to-event questions require more direct temporal resolution, which better suits TISER.
- Comparison with baselines: HippoRAG 2 slightly outperforms the embedding model by J +1.1, likely because its entity-centric design aligns with Complex-TR.Embedding models remain strong baselines on this benchmark.
- Reasoning and retrieval: REMem’s agentic retrieval is important for complex reasoning, as REMem-S lacks multi-hop reasoning capabilities (J −6.0).The comparison attributes the gap to the need for agentic retrieval for autonomous reasoning.
- Reasoning and retrieval: REMem with TISER achieves the highest overall performance, improving over REMem by F1 +7.3 and J +2.4.The improvement comes from using TISER as the final-step reasoning prompt.
C.6 ABLATION STUDY
The ablation study finds that gist nodes provide most of REMem’s context, while facts supply complementary support that is especially important for multi-hop questions.
- Memory components: Removing gists reduces J by 27.3% on LoCoMo and 8.7% on Complex-TR, showing that gists provide the primary context.The ablation compares versions without gist information across both benchmarks.
- Memory components: Removing facts causes a J −7.1% drop on LoCoMo multi-hop questions, despite comparable performance on most other tasks.Facts therefore provide supplementary support that is particularly useful for multi-hop questions.
- Memory components: Phrase nodes help bridge concepts across sessions and facilitate effective graph exploration.This interpretation is linked to the multi-hop degradation observed without facts.
- Temporal analysis: REMem classifies queries into temporal categories and reports separate temporal, non-temporal, and overall performance averages on LoCoMo.The temporal average aggregates all temporal categories, while the overall average combines temporal and none categories.
- Temporal analysis: REMem-I performs on par on non-temporal and temporal LoCoMo questions.The reported comparison uses the category-based results in Table 18.
E IMPLEMENTATION DETAILS
The implementation compares open-source memory baselines under closely aligned settings and documents the tables used for category performance, runtime, and resource usage.
- Baseline implementation: Mem0 and Graphiti were reproduced with open-source implementations while aligning backbone LLM, embedding model, and context scale as closely as feasible.The reproduction also used message-level memory additions after observing Mem0 often rejected adding input text.
- Evaluation reporting: The implementation reports LoCoMo performance by temporal category, including temporal, non-temporal, and overall aggregates.These results are presented in Table 18.
- Evaluation reporting: Running time and memory usage are reported in Table 19.The table documents the resource measurements for the implementation.
F.1 COMPARATIVE ANALYSIS: REMEM VS. TISER
The comparative analysis highlights REMem’s multi-hop temporal reasoning and extraction coverage while also documenting efficiency, token-use, graph-scale, and tool-design considerations.
- REMem versus TISER: REMem typically outperforms TISER on Complex-TR by handling multi-hop temporal reasoning more comprehensively and recovering the full set of required entities.A cited example shows REMem recovering both the gold answer and an additional answer for a before-ASU question.
- REMem versus TISER: REMem can misinterpret before/after questions when tool calls do not align with whether temporal inequalities include equality.The passage notes that retrieved context can often clarify the intended temporal relationship.
- Efficiency: Graphiti is two orders of magnitude less efficient than Mem0 and REMem during indexing, while REMem’s multi-step inference runtime grows linearly with the number of steps.REMem’s single-step inference runtime is comparable to Mem0 and Graphiti.
- Efficiency: REMem-I’s token consumption increases with the number of inference iterations, whereas REMem-S is nearly equivalent to the embedding baseline.These measurements are reported for LoCoMo inference.
- Memory construction: REMem’s extraction is more comprehensive than Mem0’s on the MuSiQue example, including an inferred June 2003 season closing date.The comparison associates Mem0’s extraction with weaker factual coverage and incomplete or decontextualized content.
- Memory construction: The evaluated graph structures vary substantially: conversational and Complex-TR datasets produce large dense graphs, while Test of Time produces a smaller graph without gist annotations.Test of Time uses anonymous entities and formalized statements.