PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Memory Learning for Long-Term Conversations

Cai Ke, Jiangyue Yan, Han Zhang, Xin Liu, Zike Yuan, Yue Yu, Hui Wang, Ruifeng Xu

arXiv:2609.17088v1cs.AIcs.CL

TL;DR

Long-term conversation agents need memory that adapts to changing user expectations instead of relying on static archival heuristics. ICML combines synthesized expert data with interactive online reinforcement learning, using Planner and Trigger agents whose memory policies co-evolve through feedback. The framework significantly outperforms strong baselines and continuously improves response quality as interactions accumulate.

  • Problem

    Existing approaches passively archive information with static heuristics, limiting adaptive memory management for changing user expectations and preferences.

  • Method

    ICML uses session synthesis and online reinforcement learning with Planner and Trigger agents to learn what to memorize and when to retrieve it.

  • Results

    ICML significantly outperforms strong baselines in generation quality and continuously improves response quality through online interaction.

  • Takeaways & Limitations

    Interactive memory learning offers a robust and efficient framework for lifelong personalized assistants that retain valuable information and self-evolve.

  • Takeaways & Limitations

    The evaluation prioritizes open-domain engagement and personalized alignment, not mathematics, coding, or standard question-answering benchmarks.

Abstract

from arXiv · show

Recent advancements in large language models have significantly enhanced the capabilities of agents in modeling long-term conversations. Despite these successes, existing approaches typically adopt a static heuristic paradigm, where information is passively archived without adaptive memory valuation. Consequently, these methods fail to self-evolve or align their memory management with evolving user needs. To address this, we propose ICML (InteraCtive Memory Learning), a multi-agent framework that transforms the memory mechanism from a passive archive into a learnable, interactive memory policy. Specifically, we first employ a session synthesis pipeline to generate expert data, facilitating rapid test-time adaptation in unseen scenarios. Building on this, ICML utilizes an online reinforcement learning mechanism where a Planner agent selectively encodes high-value information and a Trigger agent dynamically retrieves it to optimize response quality, whereby the two agents co-evolve through continuous interaction feedback. Crucially, both agents are synchronized through a delayed reward mechanism that propagates future feedback back to earlier storage decisions, ensuring memory policies are precisely aligned with user expectations. Experimental results demonstrate that ICML significantly outperforms strong baselines, exhibiting the unique capability to continuously improve response quality as interactions accumulate.

1 Introduction

Long-term conversation agents need memory policies that adapt to changing user expectations rather than passively archiving information. ICML introduces interactive memory learning with multi-agent online reinforcement learning and reports stronger, continually improving personalized responses.

  • Existing memory methods use static heuristics, treating memory as a database and failing to decide what to memorize or when to trigger.This leaves conflicts between outdated preferences and new user requirements unresolved.
  • A retrospective session-synthesis pipeline generates interconnected storylines and forward-annotated expert data for autonomous test-time adaptation in new environments.It begins with a seed session representing an initial user interaction.
  • ICML frames memory as a learnable process that uses continuous environmental feedback to optimize what information is memorized and when it is retrieved.The framework follows the distinction between valuable memories and irrelevant noise described in human memory research.
  • The framework is presented as a plug-and-play online RL approach enabling memory policies to self-evolve and align with user expectations without human intervention.Its evaluations cover three long-term open-domain datasets.
  • ICML outperforms state-of-the-art baselines while response quality and personalization improve consistently as interactions accumulate.

2 Related Work

Related work spans long-term open-domain conversation generation and management-centric memory architectures. Prior approaches target personalized interaction or sophisticated memory lifecycle handling, providing context for ICML’s interactive learning paradigm.

  • Long-Term Open-Domain Conversation: Long-term open-domain conversation research targets lifelong companionship and personalized experiences rather than long-term question answering.A major trend is generation-centric dialogue agents for LLMs.
  • These related-work directions frame conversation generation and memory management as distinct approaches to supporting long-term personalized interaction.
  • Agentic Memory Architectures and Management: Agentic memory research develops management-centric architectures for handling the full memory lifecycle.Examples use graph-based representations or interconnected structured notes that evolve dynamically.

3 Methodology

ICML formulates long-term conversation memory management as a POMDP and combines retrospective expert-data synthesis with collaboratively trained Planner and Trigger agents. Cross-Session Truth Rewards connect future retrieval quality to earlier storage decisions, while online optimization enables policy adaptation during interaction.

  • 3.1 Problem Formulation: ICML models interactive memory learning as a POMDP with latent user intent and history, while observations contain the current query, recent dialogue, and external memory.The tuple is ⟨S, A, O, R, γ⟩.
  • 3.2 Retrospective Session Synthesis: Retrospective Session Synthesis reversely generates logically consistent histories from a seed session, then annotates valuable memories and required retrievals for expert-data initialization.Planner labels identify high-value turns; Trigger labels identify relevant historical fragments and include hard and soft negatives.
  • 3.3.1 Planner: The Planner saves or discards current interactions to retain high-value information and filter redundant or irrelevant content.An LLM-based proxy reward supplies dense early guidance, with a miss penalty for discarding high-value turns.
  • 3.3.2 Trigger: The Trigger selects a memory index or no memory from the current query, dialogue history, and candidate memories, then concatenates the selected memory with context for response generation.Its quality reward scores final responses across multiple dimensions.
  • 3.3.4 Cross-Session Truth Reward: Cross-Session Truth Rewards propagate future retrieval quality to historical Planner storage decisions, aligning storage with long-term memory utility and enabling Planner–Trigger co-adaptation.A Pending Reward Buffer stores latent memories awaiting verification.
  • 3.3.5 On-Policy Optimization: PPO optimizes both Planner and Trigger policies from online rollout trajectories, using cumulative returns and shared actor–critic training for the collaborative agents.The online phase collects observation, action, and reward trajectories from real-world scenarios.

4 Experiments

Across three long-term conversation datasets, ICML consistently outperforms diverse baselines and shows that interactive memory policies can improve quality, coordination, and efficiency over time.

  • Main Results: ICML achieves state-of-the-art results across all datasets, outperforming management-centric memory agents, generation-centric dialogue agents, and long-context baselines.The comparison covers automatic, personalized, and human-oriented evaluation settings.
  • Main Results: ICML aligns automated and human judgments, improving all LLM cross-evaluation dimensions while receiving dominant preference in human evaluation.The results indicate stronger balance between accurate recall, contextual precision, and natural conversation.
  • Ablation Study: Removing Cross-Session Truth Reward causes the sharpest ablation decline, while removing Evolution, Synthetic Data, Planner, or Trigger also harms performance.These ablations support the roles of long-term feedback, cold-start initialization, continuous adaptation, and agent cooperation.
  • Collaborative Agents: Matched Planner and Trigger model sizes usually maximize performance, while smaller paired models can outperform mismatched larger configurations.The diagonal scaling pattern appears on CC and remains consistent on MSC and GC.
  • Collaborative Agents: Over 10 sessions, Planner precision and Trigger recall rise consistently as the agents refine their coordination through environmental feedback.This trend demonstrates online co-evolution during continued interaction.
  • Analysis: A moderate warm-up of 0.25K–0.5K synthetic episodes provides the best performance balance, avoiding both cold-start failure and overfitting to fixed patterns.The analysis identifies moderate initialization as preferable to insufficient or excessive static supervision.
  • Efficiency: ICML keeps inference usage stable and storage growth minimal while also achieving faster memory construction than methods with more complex processing.The reported efficiency supports real-time long-term interaction without context bloating.

5 Conclusions

ICML uses a collaborative Planner–Trigger framework with delayed feedback to optimize long-term memory. It outperforms strong baselines while maintaining efficient resource use and continuous self-evolution through online interaction.

  • Contributions: ICML combines co-evolving Planner and Trigger agents with delayed feedback to retain information according to conversational utility.The framework aligns memory operations with later interaction outcomes.
  • Results: ICML significantly outperforms strong baselines in generation quality while maintaining millisecond-level latency and stable token consumption.The conclusion presents the method as an efficient solution for lifelong personalized assistants.

Limitations

The evaluation focuses on personal conversational assistants in deep, long-term open-domain dialogue rather than general reasoning or fact-retrieval tasks.

  • Scope: The method is validated in dialogue, while mathematics, coding, and standard question-answering benchmarks remain outside the study’s scope.The stated scope prioritizes open-domain engagement and personalized alignment.

A Dataset Information

The paper evaluates ICML on three long-term, multi-session conversation datasets and compares it with management-centric and generation-centric memory agents.

  • Datasets: Three datasets—CC, MSC, and GC—are used to evaluate long-term multi-session conversation performance.CC emphasizes temporal dynamics and speaker relationships; MSC models relationship-building through repeated interactions; GC includes realistic gaps from minutes to years.
  • Dataset construction: The experiments use human-verified datasets constructed through a crowdsourcing pipeline for context-dependent long-term conversation evaluation.The study samples episodes from the test sets for its experiments.
  • Baselines: The evaluation compares ICML against management-centric memory agents and generation-centric dialogue agents.The comparison includes systems such as Mem0, A-Mem, MemoryOS, MemoryBank, LD-Agent, and THEANINE.
  • Management-centric methods: Management-centric baselines organize, update, or retrieve memories using mechanisms such as extraction, structured notes, embeddings, and multi-level storage.Examples include Mem0’s extraction and update phases, A-Mem’s atomic notes, and MemoryOS’s storage hierarchy.
  • Generation-centric methods: Generation-centric baselines target personalized long-term dialogue through hierarchical summaries, modular persona processing, or retention of outdated information.MemoryBank, LD-Agent, and THEANINE represent these approaches.

B.3 Memory-Related Methods

The paper situates ICML among memory-related methods and describes its training, evaluation, and collaborative Planner–Trigger implementation.

  • Memory-related methods: Memory-related baselines use recursive summarization, instruction-based memos, or unified compressive memory to support long-term dialogue.MemoChat cycles through memorization, retrieval, and response; Rsum recursively combines memories with later context; COMEDY unifies memory generation, compression, and response.
  • Training: ICML uses supervised warm-up followed by online reinforcement learning to train its interactive memory framework.The implementation summary identifies synthesized expert data for initialization and PPO-based online optimization.
  • Interactive memory learning: The framework jointly evolves Planner and Trigger agents through on-policy exploration and Cross-Session Truth Rewards.The reward mechanism retrospectively aligns earlier planning decisions with later conversational feedback.
  • Performance scaling: Performance scaling experiments show that increasing Planner and Trigger sizes generally improves metrics, with matched sizes producing better collaboration across MSC and GC.The diagonal pattern indicates that architectural balance is associated with stronger performance in these datasets.

G Training Reward Analysis

Training rewards for Planner and Trigger rise together across CC, MSC, and GC, then stabilize, indicating coordinated policy evolution.

  • Reward trajectories: Planner and Trigger rewards show synchronized upward trajectories across the CC, MSC, and GC datasets.The trajectories indicate that the writing and reading policies co-evolve rather than compete adversarially.
  • Reward convergence: After rapid initial exploration, both agents’ rewards reach a stable plateau without significant oscillation.This convergence is reported as evidence that the collaborative reinforcement-learning framework reaches a steady equilibrium.

H Robustness to Reward Model Choice

ICML remains stable across alternative LLM reward models, while the paper presents reinforcement learning as extending performance beyond a static synthetic-data baseline.

  • Reward-model robustness: Performance remains stable when Gemini2.5, GPT-3.5-turbo, or GPT-4o-mini serves as the reward model.The paper reports negligible differences in key metrics, with Mauve remaining around 80.
  • Interpretation: The results are presented as evidence that ICML’s performance derives from its collaborative design rather than dependence on a particular reward model.The authors consequently characterize the framework as usable with smaller or cheaper language models.
  • Static baseline versus adaptation: Retrospective synthesis provides a warm-up stage, but static synthetic data reaches a performance ceiling as user behavior changes.The reinforcement-learning phase uses real-time interaction feedback to move beyond that static baseline.

J Efficiency Analysis

ICML achieves superior long-term efficiency despite an initial warm-up cost. Its runtime quickly stabilizes, supporting rapid memory updates for real-time online deployment.

  • J Efficiency Analysis: ICML demonstrates superior long-term efficiency compared with THEANINE and MemoryOS.The comparison is presented in Figure 15.
  • J Efficiency Analysis: 52.74s in Session 1 falls to 10.22s by Session 5 after warm-up.The initial overhead includes GPU memory allocation and model initialization.
  • J Efficiency Analysis: ICML’s stabilized runtime enables rapid memory updates without significant latency in real-world interaction scenarios.

K Analysis of Early Exploration and Proxy Rewards

The analysis shows that proxy rewards improve early memory exploration, while ICML remains robust across reward settings, synthetic-data strategies, judges, and zero-shot adaptation comparisons.

  • K Analysis of Early Exploration and Proxy Rewards: The Proxy Reward and Miss-penalty provide immediate Planner feedback when Trigger exploration fails, retaining high-value information during initial learning.This supplements the delayed Truth Reward before useful memories are reliably retrieved.
  • K Analysis of Early Exploration and Proxy Rewards: Miss Rate declines rapidly from Session 1 to Session 3, while later improvements and rising Pearson r indicate refined policies and stronger reward alignment.The evaluation tracks Miss Rate, False Positive Rate, and Proxy Reward–Truth Reward correlation over five sessions.
  • L Significance and Agreement Analysis for Human Evaluation: Human evaluations show substantial evaluator agreement, with Fleiss’ Kappa scores above 0.6 and statistically significant win rates of 66.0% and 70.0%.The win rates have p < 0.01 and narrow confidence intervals.
  • K Analysis of Early Exploration and Proxy Rewards: λ = 0.5 consistently achieves the best performance across three datasets and two backbones, while λ = 0.3 or 0.7 causes slight drops.The tested values are λ ∈ {0.3, 0.5, 0.7}.
  • K Analysis of Early Exploration and Proxy Rewards: A 1B-parameter reward-model judge causes only minimal degradation, and ICML does not collapse under weaker reward signals.The comparison uses Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct as judges.
  • K Analysis of Early Exploration and Proxy Rewards: Backward synthetic-data generation consistently outperforms Forward generation across all metrics.Backward generation provides a higher-quality, logically consistent initialization signal for subsequent reinforcement learning.
  • P Comparison with RMM: ICML consistently outperforms RMM’s RL-only variant across all datasets and backbones in zero-shot adaptation.RMM’s performance drops significantly without offline training data.
Loading 2609.17088v1…