PaperlayerView exact revision on arXiv

Source-linked AI summary

EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory

Chang Nie, Chaoyou Fu, Junlan Feng, Caifeng Shan

arXiv:2606.21649v2cs.CL

TL;DR

Static embeddings overlook evolving context in long-context retrieval, where information is sequential and requires state tracking. EvoEmbedding maintains a rolling latent memory to generate context-aware representations, achieving state-of-the-art retrieval performance and strong generalization across long-context and agentic-memory tasks.

  • Problem

    Existing retrieval systems rely on static representations that struggle with dynamic, sequential long-context information requiring continuous state tracking.

  • Method

    EvoEmbedding recurrently updates a latent memory and jointly encodes it with incoming content to produce context-aware embeddings.

  • Results

    +11.1% over Qwen3-Embedding-8B, with state-of-the-art performance across eight long-context retrieval benchmarks and generalization to 128K contexts.

  • Takeaways & Limitations

    EvoEmbedding enables context-aware retrieval in long-context and agentic workflows, with naive RAG reaching 77.6% on LongMemEval.

  • Takeaways & Limitations

    The model may degrade out of domain and currently lacks native multimodal retrieval support.

Abstract

from arXiv · show

Existing embedding models are inherently static: they encode text segments in isolation, ignoring their surrounding context and temporal order. This paper introduces EvoEmbedding, a novel embedding model that generates evolvable representations for retrieval. It is tailored for long-context scenarios, where information is dynamic, sequential, and requires continuous state tracking. Our design is simple: EvoEmbedding maintains a continuously updated latent memory as it sequentially processes inputs, and uses it alongside the raw content to jointly generate evolvable embeddings. Consequently, for the same query, our model adapts its representation to retrieve distinct targets based on the evolving context, going beyond static semantic search. To equip the model with this capability, we construct EvoTrain-180K, a diverse dataset for the joint optimization of latent memory and retrieval. Furthermore, we introduce a memory queue to prevent representation collapse during recurrent encoding, alongside segment-batching techniques that tackle significant length variance and accelerate training by 3.8$\times$. Extensive experiments show that our model not only outperforms larger-scale specialists (e.g., Qwen3-Embedding-8B and KaLM-Embedding-Gemma3-12B) across a range of long-context retrieval benchmarks, but also generalizes well to downstream tasks (e.g., personalization) with contexts 10$\times$ longer than its training window. Notably, EvoEmbedding seamlessly integrates into agentic workflows to boost performance. For instance, a naive RAG pipeline equipped with our model surpasses dedicated agentic memory systems. Project Page: https://clare-nie.github.io/EvoEmbedding/.

1 Introduction

EvoEmbedding addresses static retrieval’s loss of temporal continuity and context by maintaining an evolving latent memory that generates context-dependent representations. Across long-context benchmarks and agentic-memory settings, it improves retrieval performance, scalability, and temporal targeting while avoiding explicit memory-construction overhead.

  • Motivation: Existing retrieval pipelines encode segments in isolation and train embeddings mainly on short, static samples, disrupting temporal continuity and contextual associations.These design choices limit retrieval in dynamic long-context scenarios.
  • Approach: EvoEmbedding sequentially updates a fixed-capacity latent memory to preserve evolving contextual states and generate contextually evolvable embeddings.The unified architecture is instantiated across 0.8B, 2B, and 4B parameter base models.
  • Results: +6.4% versus KaLM-Embedding-Gemma3-12B and +11.1% versus Qwen3-Embedding-8B, while achieving the highest overall accuracy across eight long-context benchmarks.These results are reported across 10 benchmarks covering diverse tasks, domains, and context scales.
  • Results: 77.6% versus A-MEM’s 65.2% and LightMem’s 70.2%, with naive RAG using EvoEmbedding-4B achieving the highest LongMemEval accuracy and zero token overhead for explicit memory construction.Figure 2 describes this as SOTA performance with minimal token consumption.
  • Results: EvoEmbedding’s query-context similarities peak at targeted historical stages for temporal keywords such as ‘firstly’ and ‘lastly’, decoupling temporal intent from coarse textual semantics.The result supports its suitability for temporal RAG.

2 Related Work

Related work spans static text embedding models, agentic RAG systems that optimize retrieval for long-context settings, and latent-memory approaches that retain information internally. These directions motivate context-aware retrieval beyond isolated embeddings and external retrieval alone.

  • Text Embedding Models: Text embedding models encode variable-length text into dense semantic vectors and assess relevance through similarity, with LLM-based representation learning becoming the dominant recent trend.Sentence-BERT is identified as an early milestone in this line of work.
  • Agentic RAG for Long-Context: Agentic RAG optimizes retrieval during information storage and query execution to address static-embedding limitations in long-context scenarios.Existing directions include retrieval-friendly databases, knowledge graphs, memory architectures, and post-processing.
  • Latent Memory and Retrieval: Latent-memory methods maintain long-term information internally, including RMT, which exceeds 1M tokens while preserving linear computational complexity through continuous in-place memory updates.M+ scales latent memory by using a co-trained retriever to dynamically recall compressed hidden states.

3 Methodology

EvoEmbedding sequentially evolves latent memory while jointly encoding that memory with each segment to produce context-dependent retrieval representations. Its methodology combines queue-based memory control, dynamic segment batching, joint memory-retrieval training, and EvoTrain-180K data synthesis.

  • Evolving representations: EvoEmbedding sequentially processes segments while parallelizing memory evolution and representation generation, using latent memory as global semantic context for evolvable embeddings.During querying, only representation generation is executed to obtain the retrieval vector.
  • Latent memory queue: The FIFO latent-memory queue stores representations from the most recent L steps, limiting historical memory loop-encoding to at most L times and preventing recurrent collapse.Its capacity is C = L×K, with a projector mapping newly generated tokens into shared memory space.
  • Dynamic segment-batching: 3.8× speedup results from dynamically batching k consecutive segments while keeping concatenated inputs below a threshold such as 2048 tokens.Memory evolution, queue updates, and embedding generation remain consistent with the sequential formulation, while performance also improves.
  • Joint optimization: Training jointly optimizes memory generation and contrastive representation losses, with the frozen backbone forcing latent memory into the base language model’s native semantic space.The memory loss uses cross-entropy to predict target answers from the query and generated memory, while LoRA adapters are deactivated during prediction.
  • Joint optimization: The contrastive objective uses dynamically partitioned positive and negative segments, applying length weighting to stabilize optimization across variable context lengths.The query embedding is normalized, the temperature is τ = 0.1, and log(N + 1) calibrates loss scale as negative distractors vary.
  • Training data: EvoTrain-180K contains 184,137 samples generated through raw-context construction, dynamic QA generation, and retrieval labeling with verification across diverse domains, formats, and lengths.The pipeline uses over 40 QA templates and constrains each sample to at most 12K tokens and 256 segments.

4 Experiments

Across retrieval, generation, and memory-system evaluations, EvoEmbedding achieves the strongest overall results, including against larger embedding and specialized agentic-memory baselines. Ablations further show that its latent-memory queue is essential, while segment batching substantially accelerates training.

  • Overall performance: EvoEmbedding achieves the best overall performance across retrieval and downstream generation tasks on long-context and conversational benchmarks.The evaluation spans 10 benchmarks covering retrieval and generation tasks.
  • Retrieval and generation: 80.5 Overall Recall@10 and 65.2 NDCG@10 are the highest retrieval scores for EvoEmbedding-4B, exceeding KaLM-Embedding-Gemma3 by 7.8% and 8.1%.The reported margins are absolute improvements over the runner-up baseline.
  • Agentic memory comparison: 77.6% LongMemEval accuracy from naive Top-8 RAG with EvoEmbedding-4B surpasses LightMem’s 70.2%, while Single-User and Single-Assistant reach 98.6% and 100.0%.EvoEmbedding nevertheless trails LightMem on LoCoMo’s Temporal and Open-domain subtasks.
  • Agentic memory comparison: Naive RAG with EvoEmbedding surpasses the full-context LongMemEval baseline by +22.8% and approaches LoCoMo’s full-context upper bound within 0.6%.The LoCoMo result can reach 77.5% as the retrieval budget increases, while the method incurs zero additional token cost for separate memory construction.
  • Plug-and-play integration: +19.2% on A-MEM and +20.5% on MemoryOS are the reported gains when EvoEmbedding is integrated into existing memory pipelines.It outperforms reasoning and reranking strategies across all three evaluated frameworks, with comparable GPU memory overhead to Qwen3-Reranker-4B.
  • Ablations and efficiency: 3.8× training speedup reduces training time from 101.4 to 26.6 hours, while segment batching yields an overall performance gain of 1.9%.Removing the memory queue or memory loss causes representation collapse and performance degradation exceeding 50% on LoCoMo and LongMemEval.

5 Discussion and Limitations

EvoEmbedding addresses the weaknesses of context-free retrieval by maintaining a finite rolling latent memory, while its design emphasizes controllability, factuality, and flexible capability decoupling. The framework remains limited by out-of-domain degradation and the lack of native multimodal retrieval support.

  • Why EvoEmbedding?: Static retrieval can be deceived by keyword traps or paraphrased evidence requiring historical grounding, motivating a finite rolling latent memory queue.The queue is intended to help the model query past events rather than memorize everything.
  • Why latent memory for retrieval instead of generation?: Latent memory is preferred over generative memory because direct parameter, activation, or prompt modification can cause unpredictable behavior, while generative recall struggles with factual verification under limited capacity.The passage identifies controllability and factuality as key considerations.
  • Why multi-LoRA design?: Multi-LoRA unifies memory, retrieval, and generation in one general-purpose language model while decoupling capability-specific training and keeping the backbone frozen.Updating only task-specific LoRA adapters is described as a way to avoid catastrophic forgetting in generation.
  • Limitations: EvoEmbedding may degrade in out-of-domain scenarios because of limited computational resources and training-data scale, and it currently lacks native multimodal retrieval.Visual or audio extensions are possible, but long-horizon memories in omni-modal settings would require additional management.

6 Conclusion · Appendix

EvoEmbedding addresses the limitations of static representations in long-context and agentic settings by combining sequential text encoding with continuously updated latent memory. The approach enables context-aware matching while introducing mechanisms to improve training efficiency and prevent representation collapse.

  • 6 Conclusion: EvoEmbedding is introduced as a family of embedding models for long-context scenarios and agentic workflows.
  • 6 Conclusion: Continuously updated latent memory is integrated with sequential text encoding to move beyond static semantic search.
  • 6 Conclusion: The combined representation supports precise, context-aware matching.
  • 6 Conclusion: The model is designed to overcome limitations of traditional static representations.
  • 6 Conclusion: A memory queue is proposed to resolve representation collapse during training.
  • 6 Conclusion: Dynamic segment-batching is proposed to resolve training inefficiency.

A Statistics of EvoTrain-180K

EvoTrain-180K contains 184,137 training instances designed around dynamic state transitions and temporal reasoning. Its highly varied context and segment distributions support learning across mixed-length scenarios, including long-context inference.

  • 184,137 training instances comprise EvoTrain-180K, which was constructed to capture dynamic state transitions and complex temporal reasoning.
  • Data Distributions: 52.9% of contexts are under 512 tokens, while the long tail extends to 12K tokens.
  • Data Distributions: 246 segments span the maximum per sample, with 44.6% of samples containing 1–8 segments.
  • Data Distributions: 128K testing contexts are supported by the dataset’s mixed-length design, which combines short-sequence semantic matching with temporal dynamics from longer contexts.
  • Statistical Summary: 1.3K tokens and 21 segments are the approximate average context length and segment count, respectively.Each instance is paired with an average of 19.45 contrastive negative samples.

B More Details about EvoEmbedding · B.1 Runtime Pipeline

EvoEmbedding’s runtime pipeline uses parallel processes for latent-memory evolution and evolvable-embedding generation. Memory evolution activates a dedicated adapter, combines projected prior memory with current input and learnable tokens, then updates a memory queue.

  • B.1 Runtime Pipeline: The runtime pipeline details forward passes for both memory evolution and embedding generation.These processes are presented in Algorithms 1 and 2.
  • B.1 Runtime Pipeline: Memory evolution and embedding generation share a highly symmetrical architecture.They primarily differ in LoRA adapters, appended tokens, and output projection mechanisms.
  • B.1 Runtime Pipeline: The shared design enables switching between dynamically updating latent state and generating evolvable embeddings.The two operations use corresponding forward-pass structures while serving different outputs.
  • B.1 Runtime Pipeline: Memory evolution begins by activating the memory LoRA adapter θ_m and projecting the previous memory M_t−1.The projected memory is denoted m_in before concatenation with the current input.
  • B.1 Runtime Pipeline: The model concatenates projected memory, current input x_t, and K learnable tokens r_l into ˜x_t.The algorithm specifies ˜x_t = [m_in; x_t; r_l].
  • B.1 Runtime Pipeline: An LLM processes ˜x_t, and the final K outputs ˜M_t are transformed into updated memory.The update applies f_m before queueing the result with M_t−1.
  • B.1 Runtime Pipeline: The memory queue combines prior memory M_t−1 with the transformed outputs to produce updated memory queue M_t.Algorithm 1 returns the updated memory queue after this operation.

B.2 Training Details

EvoEmbedding generates representations by recurrently combining latent memory with the current input through retrieval-specific projection modules. Its model variants are initialized from Qwen backbones selected to support fair comparison with Qwen3-Embedding baselines.

  • Embedding Generation: The embedding-generation procedure activates a retrieval LoRA adapter, projects the previous latent memory, and concatenates it with the current input and <EOS>.This produces the input sequence used for recurrent embedding generation.
  • Embedding Generation: The model applies an LLM to the combined sequence, projects its final hidden state, and returns the resulting representation.The representation is computed as v_t from the EOS-position hidden state.
  • Model Initialization: EvoEmbedding-0.8B and -2B derive from Qwen3.5-0.8B and Qwen3.5-2B, while EvoEmbedding-4B is initialized with Qwen3-4B.These initialization choices are intended to ensure fair comparison against Qwen3-Embedding baselines.

B.3 Sensitivity Analysis on Memory Capacity and LoRA Rank

EvoEmbedding benefits from increasing memory capacity up to C = 512, where retrieval performance saturates and NDCG peaks. Its generation performance remains stable across LoRA ranks from 16 to 128 and different Top-k settings, indicating parameter-efficient adaptation.

  • Memory Capacity: 16 to 512: both Overall R@10 and N@10 steadily improve as memory queue capacity C increases.A larger queue broadens the historical horizon and helps capture long-range temporal dependencies.
  • Memory Capacity: C = 512: NDCG peaks and retrieval performance saturates, while larger queues yield diminishing returns and higher memory consumption.The authors therefore use C = 512 as the default configuration.
  • Impact of LoRA Rank: r ∈{16, 32, 64, 128}: generation performance remains exceptionally stable across all tested LoRA ranks and Top-k settings.The result indicates that a lightweight adapter can activate evolvable representations without massive parameter updates.

B.4 More Experimental Results

The section reports complete generation-task results for a naive RAG pipeline across five benchmarks and retrieval budgets k ∈ {1, 2, 4, 8, 16}. These results show EvoEmbedding consistently outperforms across all evaluated retrieval budgets.

  • Generation Tasks: The evaluation varies the number of retrieved contexts across k ∈ {1, 2, 4, 8, 16}.Results are provided for all evaluated retrieval budgets.
  • Generation Tasks: EvoEmbedding consistently outperforms across all evaluated retrieval budgets.The reported numerical results establish this superiority across the five benchmarks.
  • Generation Tasks: Table 9 reports exact accuracy (%) for the naive RAG pipeline across five generation benchmarks.The table also includes an overall average.
Loading 2606.21649v2…