PaperlayerView exact revision on arXiv

Source-linked AI summary

TReMu: Towards Neuro-Symbolic Temporal Reasoning for LLM-Agents with Memory in Multi-Session Dialogues

Yubin Ge, Salvatore Romeo, Jason Cai, Raphael Shu, Monica Sunkara, Yassine Benajiba, Yi Zhang

arXiv:2502.01630v2cs.AI

TL;DR

Temporal reasoning in multi-session dialogues is challenging and existing benchmarks do not explicitly address relative time and cross-session dependency. The paper constructs a benchmark from LoCoMo and introduces TReMu, which combines time-aware memory with Python-based temporal reasoning; TReMu outperforms baseline methods on the benchmark.

  • Problem

    Existing temporal benchmarks do not explicitly address relative time and cross-session dependency in multi-session dialogues, where relative expressions require inferring event times and events may link across sessions.

    Source: §1 · p. 1

  • Method

    The paper augments LoCoMo dialogues with temporal questions and introduces TReMu, which combines timeline-based memories linked to inferred dates with Python-code temporal reasoning.

    Source: §1 · p. 2

  • Results

    TReMu outperforms baseline methods including CoT, while the benchmark reveals suboptimal performance for evaluated LLMs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark and framework address temporal reasoning in multi-session dialogues, including relative time and cross-session dependency.

    Source: §1 · p. 2, §6 · p. 9

    The evaluation uses multiple-choice QA rather than generative dialogue evaluation, which the authors leave for future work because temporal-expression variation makes exact match and token F1 unreliable.

    Source: §7 · p. 9

Abstract

from arXiv · show

Temporal reasoning in multi-session dialogues presents a significant challenge which has been under-studied in previous temporal reasoning benchmarks. To bridge this gap, we propose a new evaluation task for temporal reasoning in multi-session dialogues and introduce an approach to construct a new benchmark by augmenting dialogues from LoCoMo and creating multi-choice QAs. Furthermore, we present TReMu, a new framework aimed at enhancing the temporal reasoning capabilities of LLM-agents in this context. Specifically, the framework employs time-aware memorization through timeline summarization, generating retrievable memory by summarizing events in each dialogue session with their inferred dates. Additionally, we integrate neuro-symbolic temporal reasoning, where LLMs generate Python code to perform temporal calculations and select answers. Experimental evaluations on popular LLMs demonstrate that our benchmark is challenging, and the proposed framework significantly improves temporal reasoning performance compared to baseline methods, raising from 29.83 on GPT-4o via standard prompting to 77.67 via our approach and highlighting its effectiveness in addressing temporal reasoning in multi-session dialogues.

1 Introduction

Temporal reasoning in multi-session dialogues is difficult for LLM-agents, while existing benchmarks do not explicitly capture relative time and cross-session dependency. The paper introduces a benchmark and TReMu, which combines time-aware memory with neuro-symbolic reasoning.

1 Introduction

  • Existing temporal benchmarks typically use shorter texts or do not explicitly test relative time and cross-session dependency in multi-session dialogues.Relative expressions require inferring event times, while recalled events can link sessions and reflect changes over time.Source: §1 · pp. 1–2
  • TReMu creates retrievable session memories by summarizing events with inferred dates, distinguishing when an event occurred from when it was mentioned.The summaries link events to session times or inferred event dates.Source: §1 · p. 2
  • For temporal questions, TReMu retrieves relevant memory and has an LLM generate executable Python code for temporal calculations and answer selection.The code acts as an intermediate rationale and is executed step by step.Source: §1 · p. 2
  • The benchmark augments LoCoMo dialogues with multiple-choice questions spanning three types of temporal reasoning.It targets relative time and cross-session dependency.Source: §1 · p. 2
  • The benchmark reveals suboptimal LLM performance, while TReMu outperforms baseline methods including CoT on multi-session temporal reasoning.The evaluation covers GPT-4o, GPT-4o-mini, and GPT-3.5-Turbo.Source: §1 · p. 2

2 Benchmark Construction

The benchmark is built by augmenting LoCoMo dialogues with temporal questions focused on relative time and cross-session dependency. Its questions are designed around single or linked events and pass through extraction, linking, QA generation, and manual quality control.

2.1 Benchmark Design

  • LoCoMo was selected for its long dialogues, averaging 600 turns and 16,000 tokens across up to 32 sessions, and its greater length and session count than compared datasets.The authors describe it as presenting a greater challenge.Source: §2.1 · p. 3
  • Questions cover Temporal Anchoring for a single relative-time event, plus Temporal Precedence and Temporal Interval for related events across sessions.Relative time can also add complexity to two-event questions.Source: §2.1 · p. 3

2.2 Construction Pipeline

  • GPT-4o extracts temporal events and relative-time expressions, links related cross-session events, and generates multiple-choice temporal questions, including unanswerable ones.Questions are manually checked for design alignment and grounding, with incorrect answers revised and unreasonable questions removed.Source: §2.2 · pp. 3–4
  • The resulting benchmark covers intervals from days to months and includes more temporal QAs than LoCoMo, as well as unanswerable questions.Benchmark statistics and details are presented in Table 2.Source: Table 2 · p. 3, §2.2 · p. 4

3 Methodology

TReMu builds on the memorization, retrieval, and response stages of memory-augmented LLM-agents. It adds timeline-based temporal memories and executable-code reasoning for temporal questions.

3.1 Preliminary: Memory-Augmented LLM-Agents

  • Memory-augmented LLM-agents summarize and store dialogue sessions, retrieve relevant memories, and combine them with ongoing dialogue to generate a response.TReMu builds on MemoChat's prompted implementation of this pipeline.Source: §3.1 · p. 4

3.2 TReMu

  • TReMu adds time-aware memorization and neuro-symbolic temporal reasoning to the memory-augmented LLM-agent pipeline.These are its two key components.Source: §3.2 · p. 4

3.2.1 Time-aware Memorization

  • Time-aware memorization creates fine-grained memory pieces with events linked to inferred dates, then organizes events on a timeline indexed by inferred timesteps.The organization groups events occurring simultaneously and distinguishes event occurrence from mention, reducing temporal ambiguity and improving time-based retrieval.Source: §3.2.1 · p. 4

3.2.2 Neuro-symbolic Temporal Reasoning

  • For temporal questions, the LLM generates Python code from retrieved memory, and executing it provides an intermediate rationale for answering.Python libraries support temporal calculations, and function-calling extends the reasoning beyond simple precedence relations.Source: §3.2.2 · pp. 4–5

4 Experiments

The experiments find that TReMu outperforms the baselines across three LLMs, while component comparisons associate gains with time-aware memorization and neuro-symbolic reasoning. The study also examines code execution failures and case examples of temporal errors.

4.1 Experimental Setup

  • The setup compared direct prompting, CoT, and memory-based baselines across three LLMs; GPT-3.5-Turbo dialogues exceeding its input limit omitted earlier sessions.The component comparisons contrast MemoChat + CoT with Timeline + CoT, and Timeline + CoT with TReMu; most tested open-source LLMs could not handle LoCoMo's long inputs.Source: §4.1 · pp. 5–6
  • The setup evaluated accuracy overall and precision, recall, and F1 for unanswerable questions.GPT-3.5-Turbo's earlier dialogue sessions were omitted when dialogues exceeded its input length.Source: §4.1 · pp. 5–6

4.2 Experimental Results

  • Accuracy rose from 29.83 with SP to 77.67 with our framework using GPT-4o, and our framework outperformed all baseline methods across three LLMs on both accuracy and F1.Source: §4.2 · p. 7

4.3 Ablation Study

  • Replacing standard memory with timeline summarization improved accuracy by helping models infer relative times, while replacing CoT with symbolic reasoning added gains.The authors attribute the latter gains to Python code generation while retaining step-by-step reasoning.Source: §4.3 · p. 7

4.4 Execution Failure Study

  • Code execution failures were generally low across the three LLMs; GPT-4o had the lowest rate and GPT-3.5-Turbo the highest.The study regenerates code when execution errors occur; Figure 2 reports execution-failure percentages.Source: §4.4 · p. 7, Figure 2 · p. 7

4.5 Case Study

  • In the first case, CoT confused a course's end date with its start, while TReMu distinguished the event date from the date it was mentioned.In the second, CoT misread the range for “last week,” while TReMu used dateutil.relativedelta and a custom weekRange function to infer it.Source: §4.5 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

5 Related Work

Prior work addresses temporal reasoning through language-modeling methods and develops multi-session dialogue datasets. This paper focuses on temporal reasoning involving relative time and dependencies across sessions.

  • Prior temporal-reasoning work uses fine-tuning or auxiliary objectives, while prompting-based LLM performance remains suboptimal (Chu et al., 2024).The paper distinguishes its approach through memory-augmented agents, timeline summaries, and neuro-symbolic reasoning.Source: §5 · p. 9
  • Multi-session dialogue datasets include MSC, a Korean dataset, CONVERSATION CHRONICLES, and LoCoMo, with LoCoMo featuring long-term multimodal dialogues.These works address time intervals, memory management, or speaker relationships, among other dataset characteristics.Source: §5 · p. 9
  • The paper targets relative time and cross-session dependency, temporal characteristics not explicitly explored in the prior multi-session dialogue work it describes.Source: §5 · p. 9

6 Conclusion

The paper constructs a temporal-reasoning benchmark from LoCoMo dialogues and introduces TReMu, which combines timeline-based memory with executable Python reasoning. The benchmark is challenging, and TReMu substantially outperforms baseline methods.

  • The benchmark augments LoCoMo dialogues to cover relative time and cross-session dependency.Source: §6 · p. 9
  • TReMu combines time-aware memorization through timeline summarization with Python code generation for neuro-symbolic temporal reasoning.Source: §6 · p. 9
  • Evaluations find the benchmark challenging and TReMu substantially better than baseline methods on temporal reasoning in multi-session dialogues.Source: §6 · p. 9

7 Limitations

The paper reports two scope limitations: evaluation uses multiple-choice QA rather than generative dialogue QA, and experiments mainly focus on closed-source models.

  • The benchmark uses multiple-choice assessment for reliability; generative dialogue QA is deferred because temporal-expression variation makes exact match and token-level F1 unreliable.Source: §7 · p. 9
  • Experiments mainly use closed-source models because most tested open-source LLMs could not handle LoCoMo's long dialogues or reliably follow instructions and output formats.The authors leave adaptation to open-source models as future work.Source: §7 · p. 9

A Prompts for Benchmark Construction

The benchmark is constructed with GPT-4o through temporal event extraction, event linking, and question creation.

A Prompts for Benchmark Construction

B Examples of Temporal QAs in Constructed Benchmark

The section presents examples of temporal QAs and identifies their ground-truth answers and supporting events.

B Examples of Temporal QAs in Constructed Benchmark

C Prompt for Time-aware Memorization

The section presents the designed prompt for time-aware memorization.

C Prompt for Time-aware Memorization

D Prompt for Neuro-symbolic Temporal Reasoning

The neuro-symbolic temporal reasoning prompts describe retrieval, Python-code generation, and answer selection using the code and its execution result.

D Prompt for Neuro-symbolic Temporal Reasoning

Loading 2502.01630v2…