PaperlayerView exact revision on arXiv

Source-linked AI summary

NEAR: Neighborhood-Aware Residuals for Memory Retrieval

Han Luo, Zhan Zhuang, Xu He, Jujia Zhao, Yu Li, Suqin Yuan, Dong Li, Lei Feng

arXiv:2610.13314v1cs.IR

TL;DR

Records sharing topics, entities, or events can outrank the correct evidence when retrieval relies only on query–memory similarity. NEAR uses local references and residual representations alongside original embeddings, and improves retrieval across benchmarks and embedding models.

  • Problem

    Records with different factual details may share topics, entities, or events, making records without the required evidence more likely to outrank correct evidence under similarity-only ranking.

    Source: §1 · p. 1

  • Method

    NEAR derives local references from memory neighborhoods and a precomputed query-reference table, then combines residual representations with original embeddings for a single ANN search over a fixed index.

    Source: §1 · pp. 1–2, §3.3 · p. 4

  • Results

    Across all nine frozen embedding models, NEAR improves average Recall@10 and NDCG@10 over Base; on Qwen-4B, average Recall@10 rises from 62.67 to 69.76.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    Shared-content neighborhoods can serve as matching references, and the resulting retrieval uses a fixed index with one ANN search.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

Memory retrieval aims to identify evidence from stored records to answer a query. Stored records with different factual details may share topics, entities, or events. Such shared contents would make records without the required evidence more likely to receive high similarity scores and outrank the correct evidence. In this paper, we use the information shared by these records to construct local references for query-memory matching. We propose NEighborhood-Aware Residuals (NEAR), a training-free framework that uses these references to construct residual representations that complement the original embeddings. Specifically, each memory derives a reference direction from its neighbors, and each query obtains its reference from a precomputed table. Using these references, NEAR constructs projection residuals for both queries and memories. These residuals are combined with the original embeddings for a single approximate nearest-neighbor search over a fixed memory index. Experiments on eight long-context retrieval benchmarks show consistent improvements across nine embedding models. Downstream evaluations show that NEAR improves question-answering performance.

1 INTRODUCTION

NEAR treats similar memory records as local references for query–memory matching rather than only as competing candidates. It complements original embeddings with residual representations, supports one ANN search over a fixed index, and improves retrieval results across the evaluated benchmarks and models.

1 INTRODUCTION

  • Records sharing topics, entities, or events can outrank evidence when ranking uses only query–memory similarity, but their relationships can provide local references.NEAR uses those shared contents as matching information.Source: §1 · p. 1
  • NEAR constructs memory-specific local references from shared contents and uses residual representations to complement original embeddings for query–memory matching.Each memory derives a reference from its nearest neighbors; a precomputed table supplies query references.Source: §1 · pp. 1–2
  • A compact query-reference table and fixed memory index enable retrieval with a single ANN search and no preliminary candidate retrieval.Memory representations are indexed offline, while new queries use the table to form residuals.Source: §1 · pp. 1–2
  • On eight long-context retrieval benchmarks, across nine frozen embedding models, NEAR improves mean Recall@10 and NDCG@10 over original embeddings and evaluated same-backbone baselines, and raises LoCoMo question-answering accuracy by 9.6 percentage points using Qwen3-Embedding-4B as the retrieval model.Source: §1 · pp. 1–2

2 RELATED WORK

Related work covers systems for organizing and retrieving long-term memories, embedding adaptation and similarity calibration, and residual-based matching. NEAR focuses on scoring queries against a given memory bank without changing its content or the pretrained encoder.

2 RELATED WORK

  • Long-term memory systems organize, link, or compress stored information, while retrieval methods combine relevance with importance and temporal signals or learn context-dependent representations.NEAR instead scores queries against a given memory bank without changing its contents or updating the pretrained encoder.Source: §2 · p. 2
  • Embedding adaptation ranges from supervised mappings to training-free corrections of similarity scores or vectors, including local-affinity correction and reference-based transformations.NEAR retains original similarity alongside residual matching and combines them in a single-search index.Source: §2 · p. 3, Figure 2 · p. 3
  • Residual-based methods remove reference components, but first-stage memory retrieval also needs a query reference before any candidates have been retrieved.NEAR combines each memory’s neighborhood reference with a precomputed query-reference mapping.Source: §2 · p. 3

3 METHOD

NEAR builds residual representations from neighborhood-derived memory references and precomputed query references, then combines original and residual matching scores. Its concatenated representations express the fused score as one inner product for a single ANN search.

3 METHOD

  • NEAR complements cosine similarity between frozen query and memory embeddings with neighborhood-derived residual representations, building memory representations and a query-reference table offline.A new query uses the table to form residuals before a single ANN search.Source: §3 · p. 3

3.1 NEIGHBORHOOD-BASED MEMORY REPRESENTATIONS

  • Each memory’s nearest neighbors define a local reference direction; NEAR removes the embedding component along it and repeats the procedure using normalized residuals in subsequent layers.The matching representation either retains residual length or is normalized, with the choice shared by memories and queries and selected on development data.Source: §3.1 · pp. 3–4

3.2 PRECOMPUTED QUERY REFERENCES

  • Spherical K-means clusters memory states into anchors, and a query combines the stored directions of its most similar anchors using similarity-based softmax weights.The resulting query reference supports the same projection and normalization rules used to form its matching representation.Source: §3.2 · p. 4

3.3 SCORE FUSION AND SINGLE-INDEX RETRIEVAL

  • NEAR fuses original and residual scores, scales the residual contributions, and concatenates query and memory representations so their inner product equals the fused score.Memory keys are constructed and indexed offline; each new query forms its vector using the reference tables and performs one inner-product ANN search.Source: §3.3 · p. 4

4 EXPERIMENTS

NEAR improves retrieval and downstream question answering across the evaluated benchmarks and models. Ablations, sensitivity analyses, and cost comparisons examine how the gains depend on reference construction and retrieval settings.

4.1 EXPERIMENTAL SETUP

  • Evaluation covers eight conversation and document retrieval benchmarks and 5,873 queries, comparing frozen-backbone methods with lexical and multi-vector retrievers.Nine frozen embedding models are evaluated, and downstream question answering is assessed on LoCoMo and LongMemEval.Source: §4.1 · pp. 4–5

4.2 MAIN RETRIEVAL RESULTS

  • Across nine frozen models, NEAR improves mean Recall@10 and NDCG@10 over Base; for Qwen-4B, Recall@10 rises 62.67 to 69.76 and NDCG@10 47.88 to 55.33.It also tops same-backbone methods on LoCoMo at all three Qwen sizes, improves both metrics over Base on LongMemEval, and leads on specified ESG-Reports and PeerQA comparisons.Source: §4.2 · p. 5, Table 1 · p. 6, Table 2 · p. 6

4.3 DOWNSTREAM QUESTION ANSWERING

  • On LoCoMo, NEAR improves Accuracy and F1 across nine backbones; on LongMemEval, Accuracy improves on seven of nine, with small decreases for Llama-3B and EmbeddingGemma.For Qwen-4B on LoCoMo, Accuracy rises from 62.7 to 72.3 and F1 from 30.9 to 35.5.Source: §4.3 · pp. 5–6, Table 3 · p. 7

4.4 ABLATION STUDY

  • Full NEAR beats global-mean and shuffled-reference variants and outperforms one-sided query- or memory-only matching across all four backbones.In the LoCoMo case study, fusion moves the evidence from rank 75 to rank 1.Source: §4.4 · p. 7, Table 4 · p. 7

4.5 SENSITIVITY ANALYSIS

  • NEAR beats Base across tested weights and constructions on both tasks, while Recall@K and NDCG@K gains persist for K=4, 8, 10, and 16 across five models.Unnormalized residuals perform better on LoCoMo, whereas normalized residuals achieve higher NDCG@10 at moderate and large weights on LongMemEval.Source: §4.5 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

4.6 RETRIEVAL COST ANALYSIS

  • Mean Recall@10/NDCG@10 reach 71.82/59.21 for NEAR versus 57.91/45.88 for Base, while median latency rises from 0.242ms to 0.579ms.SQ4 reduces NEAR's total assets by a median 82.7%; mean Recall@10 changes from 71.82 to 71.97 and NDCG@10 from 59.21 to 59.16.Source: §4.6 · p. 9, Table 5 · p. 9

5 CONCLUSION

NEAR uses local references from similar memory records to complement original embeddings in a single approximate nearest-neighbor search. Across eight benchmarks, it improves retrieval metrics across nine embedding models, with further gains in downstream question answering.

5 CONCLUSION

  • Across eight long-context benchmarks, NEAR improves mean Recall@10 and NDCG@10 across nine embedding models, with further gains in downstream question answering.The paper also reports a 9.6-percentage-point improvement in LoCoMo question-answering accuracy with Qwen3-Embedding-4B as the retrieval model.Source: §5 · p. 9
  • NEAR turns similar records from competing candidates into local references and combines residual representations with original embeddings for retrieval.Memory references come from neighbors, while query references come from a precomputed table; matching uses one approximate nearest-neighbor search over a fixed index.Source: §5 · p. 9
  • Future work will extend NEAR to frequently updated memory banks and explore more compact reference representations.Source: §5 · p. 9

A EVALUATION DETAILS

The evaluation covers eight benchmarks and nine frozen embedding models, using established retrieval metrics and aligned development and held-out evaluation procedures. It also specifies model and baseline configurations, fixed settings, and downstream question-answering evaluation.

A EVALUATION DETAILS

  • 5,873 queries across eight benchmarks are evaluated with capped Recall@10 and NDCG@10 using linear relevance gains.Source: A EVALUATION DETAILS · p. 13
  • Encoder inputs and pooling follow released formats, while same-encoder comparisons use identical inputs and cached embeddings.The configurations identify specific KaLM, ModernBERT, Llama-Nemotron, and Llama-NV checkpoints.Source: A EVALUATION DETAILS · p. 13
  • Baselines include same-frozen-embedding methods, a trained residual MLP, native-encoder late-interaction systems, and BM25.Development configurations are selected on aligned splits; ColBERTv2 uses PLAID and XTR uses WARP.Source: A EVALUATION DETAILS · p. 13
  • Neighborhood width, query-reference table size, routing entries, and inverse temperature are fixed at 50, 16, four, and 10, respectively.Depth, residual normalization, and total residual weight are selected on development data.Source: A EVALUATION DETAILS · p. 13
  • Question-answering evaluation restores retrieved memories to chronological order and uses temperature-zero generation with at most 1,024 output tokens.Correctness uses fixed task-specific judge prompts; LoCoMo and LongMemEval use different stated F1 procedures, and base and NEAR use the same questions.Source: A EVALUATION DETAILS · p. 13

B DEVELOPMENT SELECTION

  • Grouped five-fold development evaluation keeps queries with shared or overlapping candidate banks together, while MLDR uses disjoint Con-TEB development queries.Configurations selected by development NDCG@10 are reused for held-out Top-10 and Top-5 evaluation; seven grouped benchmarks are averaged over three repetitions.Source: B DEVELOPMENT SELECTION · p. 13

C IMPLEMENTATION DETAILS

NEAR computes fixed score scales from the memory bank offline and shares them across queries in the same bank. Its query-reference table and combined inner-product representation support a single ANN search over a fixed index.

C IMPLEMENTATION DETAILS

  • For each matching representation, NEAR computes a fixed score scale from the memory bank.The scale uses population variance over all memories, including each memory itself, and is computed offline without queries or relevance labels.Source: C IMPLEMENTATION DETAILS · p. 13
  • The score scales are shared by all queries in a bank, and the implementation uses memory-representation covariance rather than constructing all pairwise scores.Source: C IMPLEMENTATION DETAILS · p. 13
  • A query-reference table of at most 16 entries is built from memory embeddings with spherical K-means, and matching uses one FAISS HNSW inner-product ANN search.The HNSW settings are M = 32, efConstruction 200, efSearch 128, and random seed 42.Source: C IMPLEMENTATION DETAILS · p. 14

D COMPLETE RETRIEVAL RESULTS

The complete retrieval-results appendix provides task-level and aggregate comparisons across encoders and same-backbone methods. Its tables cover capped Recall@10 and NDCG@10, including additional results for two smaller Llama-derived encoders.

D COMPLETE RETRIEVAL RESULTS

  • Table A1 presents eight-task mean capped Recall@10 and NDCG@10 for all same-backbone methods, with bold marking row bests and ties.Source: Table A1 · p. 14
  • The comparison tables report task-level Recall@10 and NDCG@10 for nine encoders and six same-encoder methods.The two smaller Llama-derived encoders are grouped for legibility.Source: D COMPLETE RETRIEVAL RESULTS · p. 14
  • Tables A2–A4 provide additional Llama results and complete capped Recall@10 and NDCG@10 results, with Overall reporting mean and sample SD across three splits.Table A4 uses the evaluation settings of Table 1.Source: Table A2 · p. 14, Table A3 · p. 15, Table A4 · p. 16

E TOP-5 RESULTS

The appendix reports Top-5 retrieval results using the configurations selected for Top-10 evaluation, with Table A5 defining the metrics and aggregation.

  • The reported Top-5 results use configurations selected for Top-10 evaluation.Source: E TOP-5 RESULTS · p. 17
  • Table A5 reports capped Recall@5 and NDCG@5 across eight benchmarks as mean ± SD over three splits.Each cell places capped Recall@5 above NDCG@5.Source: Table A5 · p. 17
  • Bold in Table A5 marks the best mean, including ties.Source: Table A5 · p. 17

F ADDITIONAL MECHANISM CONTROLS

The mechanism controls retain NEAR’s selected depth, residual normalization, and fusion weight; shuffled-reference results are averaged over three fixed permutations.

G EFFICIENCY DETAILS

The appendix specifies measurement scope for retrieval costs and offline construction, and describes the quality and cost coverage of Qwen3-Embedding tables.

  • Storage and latency use representative memory banks and one CPU core, with storage including the index and query-side state.Query latency includes vector preparation and ANN search but excludes encoding.Source: G EFFICIENCY DETAILS · p. 17
  • Offline construction starts from cached embeddings, and reported times are medians over five runs.Source: G EFFICIENCY DETAILS · p. 17
  • Table A6 reports offline bank-construction times for LoCoMo and LongMemEval banks containing 629 and 476 memories, respectively.Source: Table A6 · p. 17
  • Tables A7 and A8 cover Qwen3-Embedding-0.6B and Qwen3-Embedding-4B, respectively, with quality on complete tasks and costs on representative banks.Aux. denotes query-side state, and Table A8 uses the preceding table’s settings.Source: Table A7 · p. 18, Table A8 · p. 18

Related papers

Loading 2610.13314v1…