PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence

Han Luo, Zhan Zhuang, Xu He, Jujia Zhao, Yu Li, Suqin Yuan, Dong Li, Lei Feng

arXiv:2610.13277v1cs.IRcs.CL

TL;DR

Single-vector retrieval can miss local evidence in memories, despite encoders already computing contextual token representations. The paper combines sparse token matching with whole-memory similarity, optionally adapting only the query branch. It reports retrieval gains across models and benchmarks, alongside higher question-answering accuracy on long-term memory benchmarks.

  • Problem

    Single-vector retrieval scores query–memory relevance through whole-memory similarity without directly matching queries to fine-grained information within memories.

    Source: §1 · p. 1

  • Method

    The method aggregates sparse matches between query tokens and frozen memory token states, then fuses local scores with whole-memory similarity, with optional query-only adaptation.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    Across eight models and eight benchmarks, retrieval improves; on Qwen3-Embedding-4B, training-free fusion raises average Recall@10/NDCG@10 from 62.5/48.0 to 73.8/59.9.

    Source: §5.1 · pp. 6–7

  • Takeaways & Limitations

    The retrieval gains accompany higher question-answering accuracy on long-term memory benchmarks, while compression retains most retrieval gains with reduced storage and latency.

    Source: §6 · p. 9

Abstract

from arXiv · show

Memory-augmented assistants retrieve evidence from stored conversations and documents to answer queries. Most embedding-based retrievers compare single-vector representations of queries and memories, even when relevance depends on a small part of the content. Such local evidence can be matched to queries using the contextual token representations already computed by the same embedding models. Our method uses sparse token matching to improve retrieval without additional training. Each query token retrieves a small set of matching tokens across candidate memories. The matching scores are aggregated for each memory and combined with whole-memory similarity to rank memories. An optional low-rank query adapter can be trained with supervision while keeping the encoder and stored memory representations fixed. Our method improves retrieval across eight benchmarks and multiple model families without additional training. Query adaptation further increases the gains to 13.3 and 14.3 percentage points in average Recall@10 and NDCG@10 on Qwen3-Embedding-4B. These retrieval gains translate into higher question-answering accuracy on long-term memory benchmarks.

1 INTRODUCTION

The introduction argues that single-vector retrieval misses fine-grained evidence already represented in an encoder’s contextual token states. It presents sparse token matching fused with whole-memory similarity, with optional query adaptation, and reports retrieval and downstream question-answering gains.

1 INTRODUCTION

  • Single-vector retrieval scores whole-memory similarity without explicitly matching queries to fine-grained evidence that may occupy only a small part of a memory.The same encoder already computes contextual token representations that can match such local evidence.Source: §1 · p. 1
  • The method uses contextual token representations from a frozen embedding model for sparse query-to-memory matching, then combines local scores with whole-memory similarity.With supervision, a lightweight query adapter adjusts token-level matching while leaving the encoder and memory index fixed.Source: §1 · pp. 1–2
  • Figure 1 reports that training-free fusion moves the relevant memory from rank 28 to rank 1 and yields 523 net Top-10 hits on LoCoMo.This illustration accompanies the motivation for combining local evidence with whole-memory similarity.Source: §1 · p. 1, Figure 1 · p. 2
  • Across eight retrieval benchmarks and eight embedding backbones, token representations improve retrieval without training, with higher question-answering accuracy on long-term memory benchmarks.The introduction also identifies component contributions and quality–cost trade-offs as evaluation targets.Source: §1 · p. 2

2 RELATED WORK

Related work spans long-term memory systems, single-vector retrieval, contextual and multi-vector matching, and retrieval adaptation. This paper positions its approach as reuse of frozen token states alongside whole-memory scoring, with optional query-side adaptation.

2 RELATED WORK

  • Long-term memory systems organize past interactions for question answering, while prior retrieval approaches combine links, relevance, temporal signals, evolving context, or memory management.The paper instead improves retrieval within a fixed memory bank using token representations from a frozen encoder.Source: §2 · p. 2
  • Single-vector methods learn a shared embedding space for efficient query–candidate comparison; this method also uses an existing encoder’s contextual token states without changing it.The passage contrasts using token states alongside the final embedding with improving the final embedding through representation learning.Source: §2 · p. 2
  • Contextual and multi-vector retrieval includes chunk representations, query–passage scoring, token-level late interaction, compact multi-vectors, and frozen-state methods such as SMART and XTR.This method applies matching to frozen token states and combines local scores with whole-memory similarity.Source: §2 · p. 2
  • Retrieval adaptation ranges from final-embedding mappings to query-only and token-representation updates; the optional low-rank adapter updates only the local query branch.The whole-memory scoring branch and stored memory representations remain unchanged.Source: §2 · p. 3

3 METHOD

The method ranks original memories by combining sparse token-level evidence with whole-memory similarity, using a fixed encoder and memory bank. An optional supervised query adapter modifies only the local matching branch.

3.1 PROBLEM FORMULATION AND THE FIXED MEMORY BANK

  • The fixed encoder stores pooled embeddings and contextual token states with source-memory IDs, allowing one reusable bank to support both whole-memory scoring and token-level retrieval.The token-state index adds O(Nd + |Mq|d) representation payload, compared with O(|Mq|d) for a single-vector index.Source: §3.1 · pp. 3–4

3.2 GLOBAL SPARSE TOKEN MATCHING

  • For each query token, approximate global top-k search finds matching memory tokens, and their scores are aggregated by source memory with a fallback floor for missing matches.Only retrieved matches above the shared fallback score add to a memory’s local score; when a memory has no retrieved token, only the fallback contributes.Source: §3.2 · p. 4

3.3 LOCAL–GLOBAL FUSION

  • Query-wise standardization makes pooled and sparse scores comparable before their weighted combination ranks candidate memories.The pooled score keeps every candidate eligible for final ranking, including memories with no retrieved tokens.Source: §3.3 · p. 4

3.4 OPTIONAL QUERY ADAPTATION

  • With source supervision, a low-rank residual adapts query token states while leaving the encoder, memory states, and pooled query embedding unchanged.The adapter is applied only to the local query branch; MaxSim provides a differentiable training surrogate for discrete sparse-neighbor retrieval.Source: §3.4 · pp. 4–5

4 EXPERIMENTAL SETUP

The evaluation spans retrieval benchmarks in several domains and downstream question answering on conversational-memory tasks. It compares same-backbone and standalone retrieval baselines under a shared evaluation and training protocol.

Benchmarks

  • Eight retrieval benchmarks cover conversational memory, financial reports, scientific papers, biomedical text, and long documents; downstream question answering uses LoCoMo and LongMemEval.The question-answering evaluation assesses whether retrieval improvements translate into better answers using past conversations.Source: §4 · p. 5

Baselines

  • Comparisons include same-backbone embedding and token-interaction baselines, standalone retrievers, and the method with and without query adaptation.Eight frozen backbones span Qwen3-Embedding, NVIDIA Llama-based models, EmbeddingGemma, and Jina-Embeddings-v5-text-nano.Source: §4 · p. 5

Implementation Details

  • Retrieval quality is measured by equally averaging capped Recall@10 and NDCG@10 across eight benchmarks, with uncompressed token indexes and HNSW search.The main comparison retrieves 10 memories; trainable retrieval results average three seeds, with model selection on source validation data.Source: §4 · p. 5

5 RESULTS

The method improves retrieval across benchmarks, backbones, and memory budgets, and the retrieval gains accompany higher question-answering accuracy. Component analyses and sensitivity tests identify contributions from sparse matching and local–global fusion, while compression reduces storage with limited quality change for SQ4.

5.1 MAIN RETRIEVAL RESULTS

  • 73.8/59.9 and 75.7/62.3 average Recall@10/NDCG@10 on Qwen3-Embedding-4B exceed the evaluated alternatives for training-free fusion and query adaptation, respectively.The corresponding Original embedding scores are 62.5/48.0; direct MaxSim reaches 67.8/54.3 and ColBERTv2 70.8/59.5.Source: §5.1 · pp. 5–6, Table 1 · p. 6
  • Average Recall@10 and NDCG@10 improve over Original embedding on all eight backbones, with and without query adaptation.The backbones span Qwen3-Embedding, Llama-based models, EmbeddingGemma, and Jina-Embeddings.Source: Table 2 · p. 6, §5.1 · p. 7
  • On Qwen3-Embedding-4B, LoCoMo Recall@10/NDCG@10 rises from 41.6/29.9 to 67.8/53.6, while LongMemEval rises from 59.2/47.1 to 95.9/88.2.The method leads both metrics on LoCoMo and LongMemEval, while ColBERTv2 leads both on PeerQA; BM25 leads MLDR Recall@10 and XTR leads MLDR NDCG@10.Source: §5.1 · p. 7

5.2 DOWNSTREAM QUESTION ANSWERING

  • On LoCoMo, adapted retrieval improves answer accuracy and F1 across all eight backbones; on LongMemEval, overall accuracy improves on seven of eight.For Qwen3-Embedding-4B, accuracy rises from 63.44% to 67.86% on LoCoMo and from 67.80% to 72.60% on LongMemEval.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 COMPONENT ANALYSIS

  • Removing local–global fusion reduces mean Recall@10/NDCG@10 by 1.0/1.5 points on Qwen-0.6B and 1.2/1.7 points on Qwen-4B.Replacing sparse matching with exhaustive MaxSim also lowers mean Recall@10/NDCG@10 by 1.9/2.0 points on Qwen-0.6B and 3.4/2.2 points on Qwen-4B.Source: §5.3 · p. 7, Table 4 · p. 8
  • Query adaptation adds 0.3/0.5 mean Recall@10/NDCG@10 points on Qwen-0.6B and 2.0/2.4 on Qwen-4B, but its gains vary by backbone.At k = 64, replacing the tail floor with zero reduces adapted Recall@10 by 1.61–7.14 points and NDCG@10 by 2.57–7.98 points; estimating unreturned matches matters more than eliminating approximate-search errors in these settings.Source: §5.3 · p. 8

5.4 SENSITIVITY ANALYSIS

  • Increasing token retrieval budget k does not consistently improve Recall@10 or NDCG@10, and LoCoMo metrics decline for training-free Qwen-4B beyond k = 32.For LongMemEval as k varies from 16 to 128, each Recall curve spans less than 1.6 points and each NDCG curve less than 0.9.Source: §5.4 · p. 8, Figure 4 · p. 9

5.5 EFFICIENCY AND COMPRESSION

  • Across two models and eight tasks, SQ4 reduces storage by 81.4–85.9% relative to Full, with mean Recall@10/NDCG@10 changing by +0.33/+0.19 points.On Qwen-0.6B LoCoMo, SQ4 retains gains over Original embedding at about 5 ms additional latency; RP128+SQ4 reduces storage further at a larger quality cost.Source: §5.5 · p. 9, Table 5 · p. 9

6 CONCLUSION

Single-vector scoring does not directly use contextual token representations, so the method combines sparse token matching with whole-memory similarity. Across eight models and benchmarks, it improves retrieval without additional training, and query adaptation also improves downstream question-answering accuracy.

6 CONCLUSION

  • Across eight models and benchmarks, sparse token matching combined with whole-memory similarity improves Recall@10 and NDCG@10 without additional training.Optional query adaptation keeps the encoder and memory index fixed and also improves downstream question-answering accuracy; compression reduces storage and latency while retaining most retrieval gains.Source: §6 · p. 9

A EVALUATION DETAILS

Evaluation follows the LMEB protocol across eight benchmarks, with capped Recall@10 and NDCG@10 averaged equally across subtasks within each benchmark. The appendix also specifies backbone sources and token-search implementation settings.

A EVALUATION DETAILS

  • Retrieval is evaluated on 5,873 queries using capped Recall@10 and NDCG@10, with subtasks averaged equally within each benchmark before reporting the eight-benchmark mean.Metrics are computed from each system’s returned rankings.Source: A EVALUATION DETAILS · p. 13
  • Table 6 lists backbone encoding settings.Source: Table 6 · p. 13
  • The evaluation uses Llama, EmbeddingGemma, and Jina embedding backbones, with model names linked to their official model cards.The appendix identifies the specific Llama variants and their model-card sources.Source: A EVALUATION DETAILS · p. 13
  • Local matching uses content-token states, while whole-memory vectors retain native pooling; normalized token states are stored in FP16 and searched with an inner-product HNSW index.The setup excludes prefixes, padding, and special tokens from local matching; EmbeddingGemma token states are taken before its pooled projection.Source: A EVALUATION DETAILS · p. 13

B TRAINING DETAILS

The query adapter is trained and selected using a fixed EvoTrain-180K split and validation schedule. Table 7 presents the adapter training settings.

B TRAINING DETAILS

  • The adapter and Search-Adaptor use 1,024 training and 256 validation examples from EvoTrain-180K.Source: B TRAINING DETAILS · p. 13
  • Adapter checkpoints are selected by mean source-validation Recall@10 and NDCG@10 at epochs 0, 1, 5, 10, 15, 20, and 25.Source: B TRAINING DETAILS · p. 13
  • Trainable retrieval results average three seeds, and all candidates in each query’s source bank contribute to the training loss.Table 7 lists query-adapter training settings.Source: B TRAINING DETAILS · p. 13, Table 7 · p. 13

C PER-TASK BACKBONE RESULTS

The appendix provides per-task retrieval results for six additional backbones, reporting Recall@10 and NDCG@10 separately.

C PER-TASK BACKBONE RESULTS

  • Table 8 reports per-task capped Recall@10 for six additional backbones, with original-embedding rows and mean ± SD for Ours.Source: Table 8 · p. 14
  • Table 9 reports per-task NDCG@10 for six additional backbones, with original-embedding rows and mean ± SD for Ours.Source: Table 9 · p. 14

D ADDITIONAL SCORING ANALYSIS

The scoring analysis examines query-level recoveries, the effect of removing local token-score contributions, and whether shorter independent chunks close the gap to token matching. It also reports the scope of exact-search, missing-score, and scoring-control analyses.

  • Removing six highlighted query tokens’ excess-over-floor contributions shifts the relevant memory from fused rank 1 to rank 32; removing only “progress” leaves it at rank 1.The intervention removes score contributions, not text, and leaves original local normalization fixed.Source: D ADDITIONAL SCORING ANALYSIS · p. 14
  • 607 recoveries versus 84 reverse transitions yield 523 net hit queries across 1,976 LoCoMo queries; open-domain queries have 9 recoveries versus 10 reverse transitions.These are gross query-level counts, not Recall@10 or category-normalized rates.Source: D ADDITIONAL SCORING ANALYSIS · p. 15
  • 67.98/53.42 Recall@10/NDCG@10 for the method without query adaptation exceeds original embedding’s 46.87/34.24, while shorter windows do not close the gap.Independent 128-, 64-, and 32-token windows yield Recall@10 of 44.99–46.87 and NDCG@10 of 33.42–34.23.Source: D ADDITIONAL SCORING ANALYSIS · p. 15
  • The scoring analysis includes exact-search and missing-score-estimation effects at k = 64, plus separate scoring controls.Source: Table 10 · p. 15, Table 11 · p. 15

E COMPRESSION AND COST MEASUREMENT

This section reports retrieval quality alongside storage and latency comparisons, and specifies the measurement settings. It also describes the downstream question-answering protocol and metrics.

E COMPRESSION AND COST MEASUREMENT

F DOWNSTREAM QUESTION-ANSWERING DETAILS

  • Downstream QA uses Qwen3-30B-A3B-Instruct-2507 with GPT-4o-mini-2024-07-18 as judge, retrieving K = 10 memories in the main comparison.The budget analysis varies K across 4, 8, 10, and 16.Source: F DOWNSTREAM QUESTION-ANSWERING DETAILS · p. 16
  • Native retrieval metrics use 1,533 LoCoMo and 470 non-abstention LongMemEval questions, while QA retains all 1,540 and 500 questions, respectively.QA reports question-weighted overall accuracy and F1; LongMemEval F1 is a supplemental lexical metric.Source: F DOWNSTREAM QUESTION-ANSWERING DETAILS · p. 16

Related papers

Loading 2610.13277v1…