PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-T1: Reinforcement Learning for Temporal Reasoning in Multi-session Agents

Yiming Du, Baojun Wang, Yifan Xiang, Zhaowei Wang, Wenyu Huang, Boyang Xue, Bin Liang, Xingshan Zeng, Fei Mi, Haoli Bai, Lifeng Shang, Jeff Z. Pan, Yuxin Jiang, Kam-Fai Wong

arXiv:2512.20092v1cs.CL

TL;DR

Long, noisy multi-session dialogues challenge agents to identify temporally pertinent information and reason accurately across sessions. Memory-T1 uses coarse-to-fine memory retrieval and multi-level RL rewards for accuracy, grounding, and temporal consistency. It achieves state-of-the-art temporal reasoning results and maintains robustness in dialogue histories up to 128k tokens.

  • Problem

    Existing approaches struggle to identify temporally pertinent information as dialogue histories grow longer and noisier, limiting temporal reasoning across sessions.

    Source: §1 · p. 1

  • Method

    Memory-T1 filters sessions using predicted query time scope and relevance, then uses an RL agent trained with answer-accuracy, evidence-grounding, and temporal-consistency rewards to select evidence sessions.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    Memory-T1 achieves state-of-the-art temporal reasoning performance, with a 3B model outperforming a 14B baseline and robustness maintained up to 128k tokens.

    Source: §1 · p. 2, §5 · pp. 10–11

  • Takeaways & Limitations

    Selecting temporally consistent memory evidence is a critical step toward more reliable and factually consistent long-term conversational agents.

    Source: §5 · p. 11

Abstract

from arXiv · show

Temporal reasoning over long, multi-session dialogues is a critical capability for conversational agents. However, existing works and our pilot study have shown that as dialogue histories grow in length and accumulate noise, current long-context models struggle to accurately identify temporally pertinent information, significantly impairing reasoning performance. To address this, we introduce Memory-T1, a framework that learns a time-aware memory selection policy using reinforcement learning (RL). It employs a coarse-to-fine strategy, first pruning the dialogue history into a candidate set using temporal and relevance filters, followed by an RL agent that selects the precise evidence sessions. The RL training is guided by a multi-level reward function optimizing (i) answer accuracy, (ii) evidence grounding, and (iii) temporal consistency. In particular, the temporal consistency reward provides a dense signal by evaluating alignment with the query time scope at both the session-level (chronological proximity) and the utterance-level (chronological fidelity), enabling the agent to resolve subtle chronological ambiguities. On the Time-Dialog benchmark, Memory-T1 boosts a 7B model to an overall score of 67.0\%, establishing a new state-of-the-art performance for open-source models and outperforming a 14B baseline by 10.2\%. Ablation studies show temporal consistency and evidence grounding rewards jointly contribute to a 15.0\% performance gain. Moreover, Memory-T1 maintains robustness up to 128k tokens, where baseline models collapse, proving effectiveness against noise in extensive dialogue histories. The code and datasets are publicly available at https://github.com/Elvin-Yiming-Du/Memory-T1/

1 INTRODUCTION

Long, noisy dialogue histories make it difficult for agents to locate temporally pertinent information and reason consistently across sessions. Memory-T1 combines coarse-to-fine retrieval with multi-level rewards, and the paper reports state-of-the-art temporal reasoning performance with robustness in histories up to 128k tokens.

1 INTRODUCTION

  • Temporal reasoning matters because confusing session dates or event order can produce inconsistent or inaccurate answers.The introduction illustrates this with the inference from “last night” and a session date to the event date January 9, 2024.Source: §1 · p. 1
  • Existing approaches struggle with ambiguous temporal expressions, noisy long histories, or unstructured multi-session dialogue, leaving robust scalable temporal reasoning unresolved.The introduction contrasts flat-text long-context models, time-aware frameworks, and RL methods dependent on structured metadata.Source: §1 · pp. 1–2
  • Figure 1 marks event or query time ranges and highlights key evidence spans within utterances.A time range can represent a duration or an instantaneous point.Source: Figure 1 · p. 2
  • The framework first predicts query temporal scope and filters irrelevant sessions, then retrieves high-recall candidates before an RL agent selects evidence sessions.Its multi-level reward combines answer accuracy and evidence grounding with temporal consistency at session and utterance levels.Source: §1 · p. 2
  • Memory-T1 achieves state-of-the-art temporal reasoning performance, with a 3B model outperforming a 14B baseline and robustness maintained up to 128k tokens.The paper describes these results on Time-Dialog and emphasizes robustness in long, noisy conversational contexts.Source: §1 · p. 2
  • Figure 2 depicts Memory-T1 as a coarse-to-fine cascade for selecting time-consistent memories in multi-session temporal reasoning.Source: Figure 2 · p. 3

2 RELATED WORK

Prior temporal-reasoning methods use temporal alignment, specialized training, or RL, but often depend on explicit supervision or handcrafted structures. Memory-T1 is positioned as learning implicit memory selection and temporal alignment through multi-level time-consistency rewards.

Temporal Reasoning in LLMs

  • Temporal-reasoning methods include temporal-context alignment, specialized training, and RL, while some approaches depend on explicit supervision or handcrafted structures that limit multi-session dialogue applicability.TReMu uses timeline summaries but depends on timestamp accuracy; Memory-T1 is presented as learning implicit memory selection and temporal alignment without external tools.Source: §2 · p. 3

Reinforcement Learning in Agents

  • RL has been applied to LLM reasoning, tool use, planning, and long-term interaction, but temporal reasoning over multi-session dialogue remains underexplored.The passage identifies robust memory retrieval, chronological event alignment, and reasoning with ambiguous supervision as needs in this setting.Source: §2 · p. 3

3 MEMORY-T1

Memory-T1 frames multi-session temporal reasoning as retrieving relevant, chronologically accurate evidence from noisy dialogue histories. Its coarse-to-fine retrieval and multi-level rewards train an agent to select evidence and answer queries with temporal alignment.

3.1 PROBLEM FORMULATION

  • Memory histories are timestamped sessions of utterances paired with referenced events; event annotations support reward computation during training but are unavailable at inference.The task is formulated as answering a query with an answer grounded in dialogue history.Source: §3.1 · p. 4

3.2 MEMORY-T1: TEMPORAL-AWARE MEMORY RETRIEVAL

  • Candidate Generation filters sessions first by a predicted query-time window, then by textual relevance to form a manageable, high-recall candidate pool.The remaining pool is sized to fit the agent’s context budget while preserving sessions that are temporally and textually pertinent.Source: §3.2 · p. 4
  • An RL-finetuned agent selects evidence sessions from the candidate pool and generates the answer in one output, explicitly citing the selected sessions.The policy is trained with GRPO; its objective includes a KL-divergence penalty against a frozen reference policy.Source: §3.2 · pp. 4–5

3.3 REWARD DESIGN

  • The multi-level reward combines answer accuracy, evidence grounding against gold sessions, and temporal consistency; failed output parsing receives a fixed −0.5 penalty.Accuracy rewards are adapted to answer formats, while evidence grounding compares cited and gold session sets using Jaccard similarity.Source: §3.3 · p. 5
  • Temporal consistency scores session-level proximity to the query’s time range with a soft logistic penalty and utterance-level event overlap, rewarding fully aligned events and penalizing non-overlap.Utterance-level event scores are averaged within relevant utterances and then across the session; the design targets temporally incorrect evidence within an otherwise relevant session.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate Memory-T1 on Time-Dialog and LoCoMo against diverse baselines, then analyze reward components, candidate generation, context length, noise, and efficiency. The results report top overall Time-Dialog scores and robustness across several evaluation conditions.

4.1 DATASETS

  • Time-Dialog contains 4,716 multi-session dialogue QA examples, with fine-grained time and evidence annotations used for reward computation but withheld at inference.The held-out split has 200 examples; LoCoMo is used to assess out-of-domain generalization.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTS SETUP

  • Experiments use Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, BM25 retrieval, and GRPO training within VERL.Training uses 32 examples per batch, a 1×10^-6 learning rate, eight rollouts per prompt, and a maximum sequence length of 16k tokens.Source: §4.2 · p. 7

4.3 RESULTS

  • 66.9% and 67.0% are the top overall scores for Memory-T1 3B and 7B, respectively, establishing state-of-the-art performance in the reported comparison.The 3B model exceeds Qwen2.5-14B (60.7%); a gap remains to GPT-4 (Oracle) at 86.2%.Source: §4.3 · p. 7, Table 1 · p. 8

4.4 ABLATION STUDY

  • Removing reward components harms performance: task-accuracy-only training drops overall score by 22.4%, while removing evidence grounding drops it by 9.1%.Removing only sequence consistency improves Category A by 23.4% but reduces Category B by 56.2%, illustrating different effects across task categories.Source: §4.4 · pp. 7–8, Table 2 · p. 9

4.5 MODEL ANALYSIS

  • On LoCoMo, Memory-T1 scores 37.7% versus 33.5% for Qwen-2.5-3B; its Non-RAG score rises from 31.9% to 36.7%.The Adversarial subset is an exception to the Non-RAG advantage: RAG improves unanswerable detection from 26.0 to 29.8.Source: §4.5 · pp. 8–9, Table 3 · p. 9
  • With top-k set to 10, candidate retrieval reaches about 90% evidence recall; temporal filtering removes distracting context without sacrificing recall.On the same unfiltered context, Memory-T1 scores 54.0% versus 44.2% for the base model.Source: Figure 3 · p. 8, §4.5 · p. 9
  • In the 64k–128k bracket, Memory-T1 (7B) leads by 25.0 F1 points, while the Qwen2.5-7B baseline drops by over 30 F1 points as context grows.At 5% time-label noise, overall F1 remains 67.0; at 20% noise, it is 60.0, with the main decline concentrated in time-span tasks.Source: §4.5 · pp. 9–10, Table 4 · p. 9, Figure 4 · p. 10
  • Average inference latency is 1.26 seconds per query, comparable to Time-R1 at 1.24 seconds and Qwen2.5-3B at 1.36 seconds.Qualitative analysis associates gains on six subtasks with time-range filtering and RL selection addressing errors in event order, co-temporality, and time constraints.Source: §4.5 · p. 10, Table 5 · p. 10

5 CONCLUSION

Memory-T1 combines coarse-to-fine retrieval with rewards for answer accuracy, evidence grounding, and temporal consistency. It achieves state-of-the-art Time-Dialog performance, including a 3B model outperforming a 14B baseline and robustness in histories up to 128k tokens.

5 CONCLUSION

  • Memory-T1 uses a coarse-to-fine strategy and rewards answer accuracy, evidence grounding, and temporal consistency to provide dense supervision for temporal ambiguities and noise.Source: §5 · p. 10
  • A 3B model achieves state-of-the-art Time-Dialog performance, outperforming a 14B baseline and maintaining strong robustness in dialogue histories up to 128k tokens.The framework uses coarse-to-fine retrieval and a multi-level reward incorporating answer accuracy, evidence grounding, and temporal consistency.Source: §5 · pp. 10–11
  • Selecting temporally consistent memory evidence is presented as a critical step toward more reliable and factually consistent long-term conversational agents.Source: §5 · p. 11

REPRODUCIBILITY STATEMENT

The authors describe planned public release of the dataset and source code and point to materials documenting annotation, prompts, and algorithms.

REPRODUCIBILITY STATEMENT

  • The authors state that they will publicly release the annotated dataset and source code to support replication and community research.Source: REPRODUCIBILITY STATEMENT · p. 11
  • Annotation processes and prompts are illustrated in Appendix A, Figures 20–22, while training and evaluation prompts appear in Figures 23 and 24.Source: REPRODUCIBILITY STATEMENT · p. 11
  • Detailed algorithmic procedures are provided in Appendix B, and the authors say releasing these assets can lower replication barriers and enable fair comparisons.Source: REPRODUCIBILITY STATEMENT · p. 11

ETHICS STATEMENT

The annotated Time-Dialog dataset is the paper’s main artifact, with human verification and screening procedures described for its preparation.

ETHICS STATEMENT

  • The authors identify the annotated Time-Dialog dataset as the work’s main artifact and report approximately 200 human hours for verification and protocol refinement.Three experienced NLP researchers reviewed GPT-4-assisted annotations, categorized error types, and iterated on the protocol.Source: ETHICS STATEMENT · p. 11
  • Three trained research assistants conducted model evaluations, each compensated at $20 per hour, above the local average.Source: ETHICS STATEMENT · p. 11
  • Before release, the data were screened to exclude personally identifiable information and offensive content; the dataset and code are to be released under an MIT license.Source: ETHICS STATEMENT · p. 11

A DATASET AND ANNOTATIONS

The Time dataset supplies multi-level temporal QA tasks, augmented with annotations for query time ranges, evidence, and utterance event times. Memory-T1 then filters candidate sessions and trains a policy with accuracy, grounding, and temporal rewards.

A DATASET AND ANNOTATIONS

  • The Time dataset covers three reasoning difficulty levels and 11 QA subtasks, and the paper adds three annotation layers to support reward mechanisms and analysis.The iterative GPT-4 annotation and human-verification process achieved over 95% overall accuracy; query temporal ranges use an all-history default when no discernible temporal focus exists.Source: A DATASET AND ANNOTATIONS · p. 15, Table 6 · p. 15

B ALGORITHM

  • The framework first generates candidate sessions, then fine-tunes a GRPO policy that samples evidence sets and answers and receives rewards against ground-truth evidence, answers, and query time ranges.The policy update includes a KL-divergence term weighted by λ to regularize deviation from a frozen reference policy.Source: B ALGORITHM · p. 16

B.1 CANDIDATE GENERATION (ALGORITHM 2)

  • Candidate generation filters sessions first by overlap with an LLM-predicted query time window, then ranks the survivors with BM25 and keeps the top-ranked sessions.This temporal-then-lexical cascade is intended to reduce the search space without large-scale expensive semantic models.Source: §B.1 · p. 17

B.2 MULTI-LEVEL REWARD CALCULATION (ALGORITHM 3)

  • The weighted reward combines binary answer correctness, session-level evidence-set F1, and temporal consistency to score each policy output.Evidence annotations report over 95% session-level accuracy and over 85% utterance-level accuracy; the paper uses the more reliable session-level labels for the evidence grounding reward.Source: A DATASET AND ANNOTATIONS · p. 15, §B.2 · p. 17
  • Temporal consistency averages session rewards combining chronological proximity to the gold range with utterance-level fidelity to that range.Relevant utterances inside the range receive positive reward, boundary utterances a smaller positive reward, and those outside it a penalty.Source: §B.2 · p. 17

C.1 HYBERPARAMETER DESIGN

  • The tolerance margin and scale factor set temporal-alignment softness, incentive parameters control reward and penalty magnitude, and component weights balance proximity against fidelity.Source: §C.1 · p. 18

C.2 SENSITIVE ANALYSIS

The sensitivity analysis finds that reward performance depends on balancing answer accuracy, evidence grounding, and temporal consistency. The best tested configuration scores 67.0%.

C.2 SENSITIVE ANALYSIS

  • 67.0% is the optimal score, achieved with reward weights (wa, wg, wt) = (0.6, 0.2, 0.2).Figure 6 presents the sensitivity analysis across four weight configurations.Source: §C.2 · p. 18
  • Candidate generation first filters sessions by overlap with the query’s predicted temporal window, then ranks the remaining sessions by BM25 relevance.The algorithm returns the top-ranked sessions as the candidate pool.Source: §C.2 · p. 19
  • 64.0% is achieved with accuracy-skewed weights (0.8, 0.1, 0.1), while uniform weights (1/3, 1/3, 1/3) score 62.2%.The authors interpret these results as showing that peak performance requires balancing the reward components rather than maximizing one objective alone.Source: §C.2 · p. 19

C.3 ACCURACY REWARD (Ra) METRICS

The accuracy reward scores final answers according to answer type, while the overall reward also incorporates evidence grounding and temporal consistency.

C.3 ACCURACY REWARD (Ra) METRICS

  • Categorical answers receive strict exact-match scores, while the accuracy reward maps fully wrong answers to -1 and partially or fully correct answers to their score.For example, predicting “B” for a gold answer of “B” scores 1, while “C” scores 0.Source: §C.3 · pp. 19–21
  • The final reward combines answer accuracy, evidence grounding measured by session-level F1, and temporal consistency.The temporal component combines chronological proximity and chronological fidelity; the total reward uses weights wa, wg, and wt.Source: §C.3 · p. 20
  • Timestamp answers are compared after normalization to canonical representations, making equivalent date and time formats match.“2025-09-24” matches “September 24, 2025” after normalization.Source: §C.3 · p. 20
  • Numerical duration answers use ϵ-Exact Match, which accepts predictions within tolerance ϵ of the gold duration.With a gold answer of 13 days and ϵ = 1, predictions from 12 to 14 days count as correct.Source: §C.3 · p. 20
  • Ordered-list answers receive Hamming Accuracy, the fraction of items appearing in the correct positions.The example prediction earns 1/2 because only the second and fourth items are correctly positioned.Source: §C.3 · p. 21

C.4 COMPARATIVE PERFORMANCE OF GRPO AND PPO

The section compares GRPO and PPO, tests performance across longer contexts, and examines the text-similarity reward’s contribution to temporal reasoning.

C.4 COMPARATIVE PERFORMANCE OF GRPO AND PPO

  • PPO lowers overall performance relative to GRPO by 18.5% for 3B models and 17.0% for 7B models.For 7B, PPO improves Category A by 9.2% but declines by 28.4% in Category B and 30.6% in Category C; the comparison is reported across model sizes and categories.Source: §C.4 · pp. 21–22

C.5 TEMPORAL REASONING UNDER CONTROLLED CONTEXT WINDOWS.

  • Memory-T1 maintains high, stable performance beyond 128k tokens, while the baseline peaks around 32k tokens and then collapses as context grows.The authors attribute the framework’s resilience to coarse-to-fine candidate generation filtering noisy history; Memory-T1 (7B) exceeds 0.9 F1 on Order Reasoning, Range Reasoning, and Contextual Temporal Filtering, but scores near zero on Comparison and Timeline.Source: §C.5 · p. 22

C.6 IMPACT ON TEXT SIMILARITY IN REWARD

  • Removing the text-similarity reward sharply reduces performance on duration computation and compositional reasoning, despite slight gains on localization and extraction.The authors describe similarity filtering as a noise-reduction mechanism that helps keep reasoning grounded in relevant context.Source: §C.6 · p. 22

C.7 SUPPLEMENTARY EXPERIMENTS: OUT-OF-DOMAIN GENERALIZATION

The supplementary experiments report out-of-domain performance, and the appendix documents LLM assistance and prompts used for annotation, training, and evaluation.

C.7 SUPPLEMENTARY EXPERIMENTS: OUT-OF-DOMAIN GENERALIZATION

  • 37.7% (Non-RAG) is achieved by Memory-T1 (3B), compared with 29.2% for Time-R1 and 40.2% for MemAgent (7B).The authors characterize this result as strong out-of-domain generalization.Source: §C.7 · p. 23

D LLM USAGE

  • GPT-4o assisted with Time-Dialog data annotation, and LLMs were also used for manuscript polishing.Further details of the annotation process are assigned to Appendix A.Source: D LLM USAGE · p. 23
  • The appendix includes prompts for event extraction and temporal-coverage annotation, and for predicting time ranges in user questions.The event-extraction prompt asks for events described in individual utterances.Source: Figure 20 · p. 28, D LLM USAGE · p. 28, Figure 21 · p. 29
  • Other prompts cover fact–evidence alignment, Memory-T1 training, and temporal-reasoning evaluation.The training prompt requests selected memory sessions and an answer in JSON; the evaluation prompt specifies output formats for answer types.Source: Figure 22 · p. 30, D LLM USAGE · pp. 30–31, Figure 23 · p. 31
Loading 2512.20092v1…