PaperlayerView exact revision on arXiv

Source-linked AI summary

LMEB: Long-horizon Memory Embedding Benchmark

Xinping Zhao, Xinshuo Hu, Jiaxin Xu, Danyu Tang, Xin Zhang, Mengjia Zhou, Yan Zhong, Yao Zhou, Zifei Shan, Meishan Zhang, Baotian Hu, Min Zhang

arXiv:2603.12572v6cs.CL

TL;DR

Existing text embedding benchmarks underassess long-horizon memory retrieval involving fragmented, context-dependent, and temporally distant information. LMEB introduces a standardized evaluation framework spanning diverse memory types and tasks, and finds that benchmark performance is not determined by model size or traditional passage-retrieval performance. It also identifies LMEB's text-only scope as a limitation.

  • Problem

    Existing text embedding benchmarks focus mainly on traditional passage retrieval and do not adequately evaluate long-horizon memory retrieval.

  • Method

    LMEB evaluates embedding models through 22 datasets and 193 zero-shot retrieval tasks spanning episodic, dialogue, semantic, and procedural memory.

  • Results

    LMEB provides a reasonable challenge, larger models do not always perform better, and LMEB and MTEB assess substantially different capabilities.

  • Takeaways & Limitations

    Strong traditional passage-retrieval performance does not necessarily transfer to long-horizon memory retrieval, supporting LMEB as a distinct evaluation benchmark.

  • Takeaways & Limitations

    LMEB focuses primarily on text-only retrieval and does not yet cover multimodal long-horizon memories.

Abstract

from arXiv · show

Memory embeddings are crucial for memory-augmented systems, such as OpenClaw, but their evaluation is underexplored in current text embedding benchmarks, which narrowly focus on traditional passage retrieval and fail to assess models' ability to handle long-horizon memory retrieval tasks involving fragmented, context-dependent, and temporally distant information. To address this gap, we introduce the Long-horizon Memory Embedding Benchmark (LMEB), a comprehensive framework for evaluating embedding models on complex, long-horizon memory retrieval. LMEB comprises 22 datasets and 193 zero-shot retrieval tasks spanning four memory types: episodic, dialogue, semantic, and procedural. These memory types differ in terms of level of abstraction and temporal dependency, capturing distinct aspects of memory retrieval that reflect the diverse challenges of the real world. We evaluate 15 widely used embedding models, ranging from hundreds of millions to ten billion parameters. The results reveal that (1) LMEB provides a reasonable level of difficulty; (2) Larger models do not always perform better; (3) LMEB and MTEB measure orthogonal capabilities. This suggests that the field has yet to converge on a universal model capable of excelling across all memory retrieval tasks, and that strong performance on traditional passage retrieval does not necessarily transfer to long-horizon memory retrieval. LMEB provides a standardized and reproducible framework that fills a key gap in memory embedding evaluation and supports future advances in long-term, context-dependent retrieval.

1 Introduction

LMEB addresses the underexplored evaluation of embedding models for long-horizon memory retrieval, extending beyond traditional passage retrieval to diverse memory types and demands. It standardizes evaluation across 22 datasets, 193 tasks, and 15 embedding models.

  • Current text embedding benchmarks mainly focus on traditional passage retrieval, leaving long-horizon, context-rich memory retrieval underexplored.
  • LMEB extends embedding evaluation to episodic, dialogue, semantic, and procedural memory, each reflecting distinct real-world retrieval demands.
  • LMEB evaluates 15 embedding models across 22 datasets and 193 retrieval tasks spanning four memory types.
  • The benchmark is designed to clarify embedding-model performance across long-horizon memory retrieval demands and support more capable memory embeddings.

2 The LMEB Benchmark

LMEB is a standardized benchmark for fragmented, context-dependent, and temporally distant memory retrieval. Its taxonomy, diverse datasets, unified protocol, and extensibility support reproducible evaluation across multiple memory settings.

  • 2.1 LMEB Overview and Taxonomy: LMEB targets long-horizon memory retrieval rather than only passage retrieval, covering fragmented, context-dependent, and temporally distant information.
  • 2.1 LMEB Overview and Taxonomy: The benchmark includes 22 English zero-shot datasets spanning episodic, dialogue, semantic, and procedural memory across 193 retrieval tasks.
  • 2.1 LMEB Overview and Taxonomy: LMEB organizes memory types by abstraction and temporal dependency, distinguishing event order, fragmented conversations, stable knowledge, and generalized skills or action sequences.
  • 2.2 Dataset and Diversity Analysis: Datasets span event-, turn-, round-, session-, sentence-, paragraph-, tool-, and experience-level retrieval settings.
  • 2.2 Dataset and Diversity Analysis: Dialogue datasets have relatively high lexical similarity, while procedural datasets show low similarity because they focus on domain-specific tasks such as coding, planning, and tool use.
  • 2.3 Evaluation Protocol and Extensibility: LMEB uses unified evaluation pipelines, model wrappers, and standardized files to support reproducible benchmarking and low-effort integration of models and datasets.

3 Experimental Setup

The paper places detailed experimental settings, benchmarked models, implementation details, and task instructions in the appendices.

  • Experimental settings, benchmarked models, implementation details, and task instructions are provided in Appendices D and C.

4 Main Results

LMEB presents a meaningful challenge across four memory-retrieval types, while results show that model size and task instructions affect performance inconsistently.

  • Difficulty: 61.41: bge-multilingual-gemma2 achieves the highest Mean (Dataset) score on N@10 under the w/ inst. setting.This indicates that LMEB is challenging without being excessively difficult.
  • Model scale: Larger models do not consistently outperform smaller models under the w/o inst. setting.KaLM-Embedding-Gemma3 and bge-multilingual-gemma2 underperform EmbeddingGemma-300M and bge-m3 (Dense).
  • Model scale: Model architecture, training data, and task adaptation also play important roles beyond model size.
  • Task instructions: Task instructions affect embedding models differently: some benefit, some are insensitive, and some perform better without instructions.Training data and training objectives likely influence these differences.

5 Correlation Analysis

LMEB evaluates capabilities that differ substantially from traditional passage retrieval: overall LMEB scores correlate weakly with MTEB, with especially limited transfer for dialogue and episodic memory.

  • Overall correlation: -0.115 and -0.130: LMEB and MTEB show low Pearson and Spearman correlations, respectively.The benchmarks therefore assess substantially different capabilities.
  • Evaluation settings: Tables 2 and 3 compare embedding-model performance across LMEB memory types with and without task instructions.Table 2 uses only the query, whereas Table 3 uses the query and task instruction.
  • Episodic and dialogue memory: -0.496 and -0.364: LMEB-Dialogue and MTEB show Pearson and Spearman correlations, respectively.Strong traditional passage-retrieval performance does not readily transfer to fragmented, redundant, context-dependent dialogue memory retrieval.
  • Semantic and procedural memory: 0.103 and 0.061: LMEB-Semantic and MTEB show Pearson and Spearman correlations, respectively.Their weak relationship suggests substantial differences despite both evaluating semantic passage retrieval.
  • Semantic and procedural memory: 0.291 and 0.429: LMEB-Procedural and MTEB show Pearson and Spearman correlations, respectively.Procedural memory exhibits somewhat stronger, but still limited, correspondence with MTEB.

6 Related Work

Existing embedding benchmarks and models primarily develop around text representation and traditional retrieval, while LMEB examines their relationship to long-horizon memory retrieval.

  • Embedding benchmarks: Traditional embedding benchmarks cover retrieval, semantic textual similarity, and clustering, with retrieval-focused benchmarks consolidating heterogeneous information-retrieval datasets.
  • Embedding models: Transformer-based contextual embeddings have become the dominant backbone for text embedding models.
  • LMEB comparison: LMEB compares its Mean (Dataset) N@10 scores with MTEB retrieval scores, selecting instruction settings according to each model's better performance.Point size in the correlation figure represents model size.
  • Embedding models: Recent work adapts large language models into embedding models, including GTE, Qwen3-Embedding, BGE, Jina, NV-Embed, and KaLM.

7 Conclusion

LMEB addresses the limited evaluation of long-horizon memory retrieval with a standardized benchmark, finding that memory retrieval differs from traditional passage retrieval.

  • Benchmark: LMEB comprises 22 datasets spanning four memory types and 193 retrieval tasks for long-horizon memory retrieval.
  • Findings: The benchmark finds reasonable difficulty, inconsistent advantages for larger models, and largely orthogonal capabilities between LMEB and MTEB.
  • Framework: Open-sourcing standardized data and adaptable code provides a unified, reproducible framework for evaluating embedding models on real-world memory retrieval tasks.

A Datasets

LMEB assembles datasets for episodic, dialogue, semantic, and procedural memory retrieval, covering long-horizon contexts and varied retrieval settings.

  • Memory types: LMEB includes episodic, dialogue, semantic, and procedural memory datasets spanning distinct long-horizon retrieval demands.The benchmark covers event, dialogue, knowledge, tool-use, trajectory, and planning-oriented resources.
  • Episodic memory: EPBench and KnowMeBench provide episodic resources based on structured events and time-anchored autobiographical memory streams.EPBench uses temporal, spatial, entity, and descriptive fields, while KnowMeBench evaluates evidence-linked queries over reconstructed narratives.
  • Dialogue memory: Dialogue datasets cover long-term conversations, multi-session chat, authentic interactions, temporal-event queries, and evolving user memory.LoCoMo contains up to 35 sessions and approximately 9k tokens on average; other datasets probe reasoning, updates, preferences, and memory growth.
  • Semantic memory: Semantic datasets retrieve answer-containing passages from research papers, peer-reviewed scientific QA, long documents, API documentation, and other knowledge sources.QASPER retains 1,335 valid queries, PeerQA retains 136 questions, and LooGLE uses chunked documents for long-context questions.
  • Procedural memory: Procedural datasets evaluate retrieval of functionally equivalent procedures, distilled experiences, and task-relevant items for planning and tool use.Proced_mem_bench treats trajectories with relevance scores of 7.0 or higher as relevant memories, while DeepPlanning focuses on shopping tasks.

B Tasks

LMEB organizes evaluation around dataset-specific retrieval tasks, with task types instantiated across subsets and queries augmented with explicit instructions.

  • Task organization: Each LMEB task is characterized by a task type and the abilities it assesses within a dataset.A subset partitions a dataset by topic, scenario, or source while preserving consistent evaluation criteria.
  • Task organization: The number of tasks generally equals the number of subsets multiplied by the number of task types, unless otherwise specified.
  • Instructions: LMEB prepends dataset- and task-specific instructions to each original query to form an instructed query for embedding evaluation.The original query is denoted q, and the instructed query is denoted qinst.

D.1 Benchmarked Models

The benchmark evaluates a diverse collection of embedding models across a broad parameter range using NDCG@k as the main retrieval metric and Recall@k as a complementary metric.

  • Models: LMEB evaluates embedding models ranging from several hundred million to 10 billion parameters.The evaluated set includes Jina, Qwen, EmbeddingGemma, KaLM, and other widely used embedding model families.
  • Metrics: NDCG@k is the main metric and measures ranking quality with logarithmic discounting over the top-k results.
  • Metrics: Recall@k measures the proportion of gold memories retrieved within the top-k results, using capped recall in LMEB.Capped Recall@k is used because standard Recall@k can produce unintuitive results when the number of relevant memories varies.

E Dataset Licenses

LMEB documents dataset licensing while using weighted Jaccard similarity to quantify lexical overlap between dataset corpora.

  • Licenses: REALTALK and TMD lack specified dataset licenses in the paper or repository.
  • Licenses: The remaining datasets use varied licenses, including MIT, Apache 2.0, CC BY-NC, CC BY-NC-SA, CC BY-SA, and copyrighted archives.
  • Dataset similarity: Weighted Jaccard similarity quantifies lexical overlap between source and target datasets using normalized unigram frequencies.The normalized frequency of each word is its count divided by the total frequency of all words in the dataset.

G.1 Limitations.

LMEB’s main limitation is its focus on text-only long-horizon memory retrieval. The benchmark does not yet cover multimodal memories distributed across text, images, audio, video, and other modalities, which the paper identifies as future work.

  • LMEB focuses primarily on text-only long-horizon memory retrieval.
  • The benchmark does not yet cover multimodal retrieval involving text, images, audio, video, and other modalities.
  • Extending LMEB to multimodal settings is identified as an important direction for future work.
Loading 2603.12572v6…