PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Memory Construction

Zijun Wang, Yifan Ji, Haidong Xin, Zhenghao Liu, Yukun Yan, Zulong Chen, Yu Gu, Ge Yu, Maosong Sun

arXiv:2610.13348v1cs.CL

TL;DR

LLMs struggle to answer questions over long interaction histories, and offline memory construction can be costly and poorly matched to query-specific information needs. T2Mem builds memory progressively during inference using lightweight episode metadata and linked raw interactions. Across LoCoMo and LongMemEval, it consistently outperforms strong baselines across two backbone LLMs while reducing construction overhead.

  • Problem

    Chunk-based retrieval can break temporal and cross-turn dependencies in long histories, while offline memory construction incurs overhead and must anticipate future query needs.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    T2Mem extracts lightweight episode-level metadata offline, retains links to raw interactions, and progressively retrieves and organizes both during inference.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    T2Mem consistently outperforms strong baselines across LoCoMo and LongMemEval with two backbone LLMs while substantially reducing memory construction overhead.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Deferring memory construction to inference lets the process adapt to individual queries' information needs.

    Source: §6 · p. 10

Abstract

from arXiv · show

Large Language Models (LLMs) struggle to answer questions over long interaction histories, where relevant evidence may be scattered across distant sessions. Existing memory systems typically construct and maintain sophisticated memory representations offline, before user queries are observed, incurring substantial overhead and limiting their ability to adapt memory construction to query-specific information needs. In this paper, we propose T$^2$Mem, a Test-Time Memory construction framework that defers memory construction to inference and progressively builds memory along the reasoning process. Specifically, T$^2$Mem extracts lightweight episode-level metadata offline to capture key events and temporal associations as retrieval cues, while retaining links to the original interactions to facilitate the reconstruction of dependencies across episodes. At test time, the LLM adaptively retrieves and organizes metadata and raw interactions, progressively constructing query-specific memory as reasoning unfolds. Experiments on LoCoMo and LongMemEval show the effectiveness of T$^2$Mem by consistently outperforming strong baselines across different backbone LLMs while substantially reducing the overhead of memory construction. Further analysis demonstrates that deferring memory construction to inference enables the memory construction process to dynamically adapt to the information needs of individual queries. All code is available at https://github.com/OpenBMB/T2Mem.

1 INTRODUCTION

Long interaction histories challenge LLMs, while chunk retrieval can break temporal and cross-turn dependencies and offline memory construction incurs substantial overhead. T2Mem instead progressively constructs query-specific memory during inference, and experiments report consistent gains over strong baselines across two benchmarks and backbone LLMs.

1 INTRODUCTION

  • Chunk-level retrieval can disrupt temporal context and cross-turn dependencies, making interconnected information in long histories harder to exploit.RAG segments historical interactions into chunks and retrieves relevant ones at inference time.Source: §1 · p. 1
  • Offline memory systems must preserve information for unknown future queries, creating substantial preprocessing and maintenance costs.These costs can hinder scalability and practical deployment.Source: §1 · p. 1
  • Earlier test-time approaches construct memory from an initially retrieved candidate set, which can omit evidence needed as construction unfolds.The paper frames inference-time reasoning as a way to dynamically determine which historical information to activate and organize.Source: §1 · p. 2
  • T2Mem extracts lightweight episode-level metadata and uses backlinks to activate raw interactions as reasoning progressively shapes query-specific memory.Each activated piece of evidence can guide subsequent reasoning and memory construction.Source: §1 · p. 2
  • T2Mem consistently outperforms strong memory and retrieval baselines across LoCoMo and LongMemEval with two backbone LLMs.The reported advantage is especially evident on multi-hop and temporal questions requiring evidence across episodes.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses long-history access through memory maintenance and through varied memory representations, but sophisticated representations can be costly to build and maintain. Recent query-time construction remains largely one-shot, whereas T2Mem uses an evolving memory state to guide further retrieval and integration.

2 RELATED WORK

  • Memory-management systems organize, retain, and update historical interactions, while the effectiveness of memory-unit structure and representation remains less explored.Examples draw on operating-system-inspired memory states or human-memory-inspired retention mechanisms.Source: §2 · p. 2
  • Memory representations range from event-based episodes and topic-based compression to semantic or temporal hierarchies and links back to source interactions.Building and maintaining these sophisticated representations can impose substantial overhead that hinders deployment and scalability.Source: §2 · pp. 2–3
  • Prior query-time methods retrieve broad candidate facts and reorganize them, while T2Mem lets evolving memory guide what evidence to retrieve and integrate next.The paper describes this closed loop as supporting information seeking and progressive memory refinement during reasoning.Source: §2 · p. 3

3 METHODOLOGY

T2Mem defers memory orchestration to inference, progressively acquiring and organizing relevant information from interaction history as the LLM reasons. It uses episode metadata as retrieval cues and links them to raw sessions for further evidence.

3.1 DISENTANGLING MEMORY ORCHESTRATION FROM OFFLINE TO TEST TIME

  • Unlike offline systems that organize query-agnostic memory before inference, T2Mem progressively acquires relevant information during inference and tailors construction to the query and evolving reasoning.Source: §3.1 · pp. 3–4
  • T2Mem extracts episode-level metadata containing key events, facts, entities, and temporal cues, while linking each entry to its source dialogue session.Source: §3.1 · p. 4

3.2 DYNAMIC MEMORY CONSTRUCTION DURING LLM REASONING

  • At each reasoning step, the LLM distills relevant evidence into memory, then either stops with an answer or selects a tool for more information.Source: §3.2 · p. 5
  • Each step produces a reasoning trace, a new memory cache, and an action; caches are concatenated, and construction stops when the query’s information need is met or Tmax is reached.Source: §3.2 · p. 5
  • SearchMeta retrieves unseen metadata when more events or temporal cues are needed, while ReadRaw follows a metadata entry’s source link when finer session detail is required.Source: §3.2 · p. 5

4 EXPERIMENTAL METHODOLOGY

The experiments evaluate long-term memory on LoCoMo and LongMemEval, comparing retrieval, offline-memory, and test-time methods with accuracy as the primary metric. They use two backbone models and a shared embedding model for metadata retrieval.

Datasets and metrics

  • LoCoMo and LongMemEval are used to assess memory systems’ ability to retain, retrieve, and use information from long-term interaction histories.The experiments use accuracy as the primary metric, judged for consistency with the ground-truth answer.Source: §4 · p. 6

Baselines

  • The comparison spans retrieval-based methods, systems that construct structured memory offline, and approaches that defer memory construction to inference.Baselines include Naive RAG and Search-o1, several structured-memory systems, and LazyMem as a test-time baseline.Source: §4 · p. 6

Evaluation metrics

  • All methods use Qwen3.5-Plus as the LLM-based judge to assess whether answers match the ground truth.The paper identifies accuracy as the primary evaluation metric.Source: §4 · p. 6

Implementation details

  • Qwen3.5-27B and Gemma4-31B-Instruct serve as backbones, with BGE-M3 used to retrieve metadata.T2Mem uses k = 5 for initial retrieval, k = 1 in subsequent SearchMeta calls, and at most five memory-construction steps in the main experiments.Source: §4 · p. 7

5 EXPERIMENTAL RESULTS

T2Mem outperforms the evaluated baselines across both benchmarks and backbones, while analyses of ablations, memory quality, and token use examine how its components and test-time computation relate to performance.

5.1 OVERALL PERFORMANCE

  • T2Mem outperforms all baselines on LoCoMo and LongMemEval, with consistent gains across backbone models.The paper reports particularly evident gains over retrieval-based methods on Multi-Hop and Temporal questions, and more consistent performance across task categories than offline-memory methods.Source: §5.1 · p. 7
  • Offline memory can vary substantially across task categories; on LongMemEval, LightMem has strong results in several categories but substantially lower accuracy on Single-Assistant questions.The paper relates this variation to preconstructed memories not aligning equally well with different information needs.Source: §5.1 · p. 8
  • T2Mem outperforms LazyMem, whose one-pass retrieval set makes later construction dependent on initial retrieval quality and coverage.T2Mem progressively retrieves evidence as reasoning unfolds, allowing it to acquire missing evidence during construction.Source: §5.1 · p. 8

5.2 ABLATION STUDIES

  • T2Mem consistently outperforms its Summary and Graph variants, while removing metadata, raw interactions, or rollout construction degrades performance.The paper describes metadata as cues for locating interactions, raw sessions as sources of fine-grained evidence, and rollout as progressively integrating retrieved knowledge according to evolving information needs.Source: §5.2 · p. 8

5.3 EFFECTIVENESS OF TEST-TIME COMPUTATION FOR MEMORY CONSTRUCTION

  • T2Mem scores highest across five memory-quality dimensions, including temporal accuracy, entity coverage, completeness, factual correctness, and detail preservation.The paper attributes detail preservation to iterative construction that can read more metadata or access linked source sessions when accumulated memory is insufficient.Source: §5.3 · p. 9
  • T2Mem achieves the highest QA accuracy with token consumption comparable to three offline-memory systems, and its performance improves with more test-time computation.Search-o1 scales more weakly, while Naive RAG shows no consistent benefit and can decline as the token budget increases; Figure 2 covers memory quality, token consumption, and QA performance across budgets.Source: §5.3 · p. 9, Figure 2 · p. 9

6 CONCLUSION

T2Mem constructs query-specific memory progressively at inference using lightweight offline metadata linked to original interactions. The paper reports consistent gains over strong baselines on two benchmarks and reduced memory-construction overhead.

6 CONCLUSION

  • T2Mem uses episode-level metadata as retrieval cues and links it to original interactions to progressively construct query-specific memory at test time.It adaptively retrieves and organizes metadata and raw interactions as reasoning unfolds.Source: §6 · p. 9
  • On LoCoMo and LongMemEval, T2Mem consistently outperforms strong baselines across two backbone models while substantially reducing memory-construction overhead.The conclusion also states that deferring construction to inference enables adaptation to individual queries’ information needs.Source: §6 · p. 10
  • Generative AI tools were used only for proofreading; the authors state that they prepared and verified the methodological designs, experiments, interpretations, manuscript, references, and citations.The authors take full responsibility for the final content.Source: AI USE STATEMENT · p. 10

A.2 ADDITIONAL DATASET DETAILS

The experiments use LoCoMo and LongMemEval, with specified evaluation splits for each benchmark.

A.2 ADDITIONAL DATASET DETAILS

  • The benchmarks are LoCoMo and LongMemEval, both used for long-term memory evaluation.The paper refers to Table 4 for their statistics.Source: §A.2 · p. 16
  • The evaluation excludes LoCoMo’s adversarial category, following prior studies.The passage cites Fang et al. (2026) and Hu et al. (2026a) for this choice.Source: §A.2 · p. 16
  • LongMemEval uses its official cleaned split1 for evaluation.Source: §A.2 · p. 16

A.3 ADDITIONAL EXPERIMENTAL DETAILS

Experiments used vLLM for backbone and retrieval models, fixed generation settings, and prompt-based reasoning-length control with a fixed answer allowance.

  • vLLM served both the backbone and retrieval models.Source: §A.3 · p. 16
  • Generation used temperature 0.0, top-p 0.95, top-k 20, and a maximum answer length of 1024 tokens.Source: §A.3 · p. 16
  • HiMem used bge-reranker-v2-m3, while LazyMem replaced its unavailable LazyMem-4B processor with the corresponding backbone model.The reported LazyMem results therefore use this backbone-substituted implementation.Source: §A.3 · p. 16
  • Reasoning length was controlled by prompting for at most N reasoning tokens and setting max tokens to N + 64, but the limit covered the full response rather than reasoning alone.The extra 64 tokens were reserved for the final answer and formatting; native thinking mode was disabled.Source: §A.3 · p. 16

A.4 ABLATION STUDIES ON THE LONGMEMEVAL DATASET

On LongMemEval, T2Mem led both backbone models in average accuracy; ablations indicate contributions from its metadata, raw-interaction access, and rollout construction.

  • Replacing episode-level metadata with summaries or graphs reduces average performance.Source: §A.4 · p. 16
  • Removing metadata or raw-interaction access lowers average accuracy, while removing rollout construction clearly declines Temporal Reasoning and Single Preference performance.Source: §A.4 · pp. 16–17
  • T2Mem achieves the highest average accuracy on LongMemEval with both backbone models.Source: §A.4 · p. 16, Table 5 · p. 17

A.5 EFFECTIVENESS OF T2MEM AT DIFFERENT MEMORY CONSTRUCTION STEPS

The analyses examine construction depth and comparisons with Search-o1, illustrate iterative evidence recovery and query-specific memory, and describe the experiment prompts.

A.5 EFFECTIVENESS OF T2MEM AT DIFFERENT MEMORY CONSTRUCTION STEPS

  • Accuracy generally rises with construction depth: Single-Hop gains plateau after three steps, while Temporal and Open-Domain gains continue through seven before diminishing.Source: §A.5 · p. 17
  • T2Mem's gains over Search-o1 grow at two and three steps, as metadata cues and linked source sessions help connect evidence across sessions.Each construction step generates a reasoning trace, memory cache, and next action, then appends the cache to accumulated memory.Source: §A.5 · pp. 17–18, Figure 4 · p. 18

A.6 CASE STUDIES

  • In the case study, retrieving a June 27 session supplied timing absent from the initial metadata, which T2Mem added to memory for the final answer.Source: §A.6 · p. 18

A.7 SALIENCY ANALYSIS OF ANSWER-SUPPORTING EVIDENCE

  • Token saliency averages output-to-input attention across 16 layers, 24 query heads, and output positions, with scores normalized separately for each input.Source: §A.7 · p. 18
  • LightMem's salient relocation description left Caroline's country unspecified, whereas T2Mem's query-specific metadata explicitly linked her to Sweden and supported the answer.Source: §A.7 · pp. 18–19, Figure 6 · p. 20

A.8 PROMPT TEMPLATES

  • The prompts cover episode-level metadata generation, iterative reasoning and information access, final answering, answer-accuracy judging, and answer-supporting-evidence quality.Source: §A.8 · pp. 19–20

Related papers

Loading 2610.13348v1…