MultiSessionCollab: Learning User Preferences with Memory to Improve Long-Term Collaboration

1 Introduction

• MULTISESSIONCOLLAB evaluates whether conversational agents learn user preferences and use them to improve interactions during multi-session collaboration.Its user simulators have distinct personas and preferences, and users progress most effectively when agent responses align with those preferences.Source: §1 · p. 1
• The proposed memory learns user preferences from interactions, reflects on what will help in future sessions, and retrieves relevant information to guide responses.This shifts the focus from information recall to preference-aligned collaboration over time.Source: §1 · pp. 1–2
• An RL framework uses user-simulator behavior to train more comprehensive reflections and more effective memory updates.GRPO uses an LLM judge to reward reflections that capture preferences revealed during interaction.Source: §1 · p. 2
• Memory improves collaboration across five problem-solving tasks, with higher task success, more efficient interactions, and reduced user effort.It outperforms existing memory architectures across sessions and is competitive with an oracle that receives ground-truth preferences; gains are steepest early and then stabilize.Source: §1 · p. 2
• The 19-participant user study found that memory improves collaboration across sessions, with participants describing agents as more personalized and proactive.Participants also noted weaker personalization in mixed-domain settings and when preferences were not explicitly restated within a session.Source: §1 · p. 2

Multi-Session Conversation Evaluation

• Earlier multi-session evaluations measure consistency, memory updates, or recall, whereas this work evaluates whether memory improves collaborative problem solving.The latter includes identifying useful information and leveraging it to improve user interactions.Source: §2 · p. 3

Memory; RL for Memory

• Memory research commonly targets question-answering, while this work focuses on improving user interactions during multi-session collaboration.For memory training, prior RL approaches use question-answering correctness as a reward; this work derives rewards from user behavior in interactions.Source: §2 · p. 3

Human-AI Collaboration

• Clarifying questions can help agents learn preferences in single-session interactions, but repeated elicitation can burden users across sessions.This work instead enables agents to remember preference information across sessions.Source: §2 · p. 3

3 MULTISESSIONCOLLAB

• MULTISESSIONCOLLAB evaluates whether agents learn user preferences and leverage them to improve collaboration over time.The benchmark uses diverse user simulators and can be instantiated with any problem-solving task.Source: §3 · p. 3

3.1 Collaborative Problem-Solving Session

• Each session runs until the user is satisfied or reaches the limit of 10 user-agent turns, with the user alone seeing the problem statement.Users maintain a draft answer that is never visible to the agent; they update it only when help is both useful and preference-aligned, isolating preference adherence.Source: §3.1 · pp. 3–4

3.2 User Simulator Design

• Each user profile combines a randomly selected persona with three randomly assigned interaction preferences.Preferences cover communication style, learning approach, or proactivity and are grounded in psychology, cognitive science, and human-computer interaction research.Source: §3.2 · p. 4

3.3 Evaluation Metrics

• Collaboration quality is measured by task success, user effort, and conversation length.Task success is final-draft accuracy, user effort counts preference enforcements, and conversation length counts turns; higher success and lower effort and length indicate more effective collaboration.Source: §3.3 · p. 4

3.4 Multi-Session Setting

• Agents begin without knowledge of a user’s preferences and are expected to learn them over sequential sessions involving different problems.Metrics are averaged across sessions for each user and then across users.Source: §3.4 · p. 4

4.1 Long-Term Collaborative Agents

• Session-level reflection identifies useful preference information—including its context and satisfying actions—and updates the agent’s persistent memory.The memory is supplied at the start of each session, with relevant parts retrieved during each turn.Source: §4.1 · pp. 4–5

4.2 Reinforcement Learning in MULTISESSIONCOLLAB

• GRPO trains agents to generate more comprehensive session-level reflections using learning signals derived from user-simulator behavior.The policy generates reflection rollouts; a judge evaluates them against preference-enforcing user utterances, and the policy is updated via GRPO.Source: §4.2 · p. 5, Figure 2 · p. 5
• A successful reflection must capture all preferences revealed during a collaborative session for use in future sessions.The conversation is represented as user-agent turns, and ε denotes the subset of user utterances where preferences are enforced.Source: §4.2 · p. 5
• The LLM judge rewards coverage of preferences in ε without hallucinations and adds a format reward for outputs containing a reasoning trace and reflection.The total reward combines Rcoverage(r, ε) and Rformat(r).Source: §4.2 · p. 5
• GRPO uses the reward to train a policy to produce more comprehensive, non-hallucinated reflections and more meaningful memory updates for future preference-aligned interactions.The method uses Group Relative Policy Optimization with an LLM judge.Source: §4.2 · p. 5

Evaluation

• Evaluation uses 100 user profiles and five benchmarks, with 20 randomly sampled problems per user and at most 10 conversational turns per session.The benchmarks are MATH-500, MATH-Hard, LogiQA, MMLU, and MedQA, totaling 10,000 collaborative sessions per agent.Source: §5 · p. 5

Environment Validation

• Benchmark validation finds lower performance in multi-turn problem-solving than direct problem-solving, with a further drop when user preferences are introduced.The authors relate the multi-turn drop to prior findings on difficulties asking clarifying questions and maintaining coherence, and the preference drop to users correcting preference violations rather than focusing on the task.Source: §5 · p. 5

Training

• Reflection training uses 10,000 sessions from 100 different user profiles and non-overlapping problems, then applies SFT cold-start initialization and GRPO to Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct.Training sessions are generated using Llama-3.3-70B-Instruct as both user simulator and agent.Source: §5 · p. 6

Models

• Evaluation covers Llama-3.1-8B-Instruct, Qwen-2.5-7B-Instruct, gpt-oss-20b, and Llama-3.3-70B-Instruct, with memory and no-memory conditions and GRPO results for two smaller models.All models are evaluated as standard conversational agents and with memory.Source: §5 · p. 6

Main Results

• Memory improves collaboration quality across the benchmark, with higher task success, more efficient interactions, and less user effort.Table 1 reports task success, user effort, and conversation length across five tasks; the prose attributes the changes to learning preferences and adhering to them without repeated user enforcement.Source: §6 · p. 6, Table 1 · p. 7

Reflection Training

• After GRPO training, Qwen-2.5-7B-Instruct shifts from a 1.03% decrease in task success with memory to a 3.43% improvement in task success.The authors report notable improvements across all metrics for the trained Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct models.Source: §6 · p. 6

Comparison with Existing Memory

• The memory architecture outperforms Mem0 across all metrics averaged over tasks, and the authors attribute the difference to its user-centric interaction information rather than factual information for question answering.Table 2 compares task success, user effort, and conversation length averaged across tasks.Source: §6 · p. 6, Table 2 · p. 7

Additional Analysis

• Providing the full multi-session conversation history as context is not a viable alternative to memory.Source: §6 · p. 6

Performance Across Sessions

• Across sessions, task-success deltas trend upward and user-effort deltas downward, with the steepest user-effort improvements in the first five sessions before stabilizing toward the last 10.Some task-success deltas are close to zero; Figure 3 plots memory-versus-no-memory deltas across sessions for Llama-3.3-70B-Instruct.Source: §6 · p. 7, Figure 3 · p. 8

Value of Learning Through Interaction

• The memory architecture is competitive with an oracle given direct access to preferences and can sometimes outperform it, with reflections capturing preference contexts and strategies that are refined across sessions.The oracle comparison was intended to establish an upper bound, although the memory agent begins without user preference information.Source: §6 · pp. 7–8

Setup

• The study involved 19 participants completing three sequential sessions with both standard and memory-equipped agents, adopting preferences they enforced throughout.Participants rated preference adherence, retention, confidence, and satisfaction after each session on a 5-point Likert scale.Source: §7 · p. 8

Results

• Across single-domain and mixed-domain studies, memory agents improved over sessions in conversation length, preference adherence and retention, confidence, and satisfaction.By session three, median turns were 6 for coding and 4 for mixed-domain memory conversations, versus 10 and 8 without memory, respectively.Source: §7 · p. 8
• Participants found personalization less effective in mixed-domain settings, and learned preferences less effective than preferences explicitly restated within a session.The study included coding-only and mixed-domain settings with writing, math problem-solving, and coding.Source: §7 · p. 8

• MULTISESSIONCOLLAB evaluates agents’ ability to learn user preferences and use them to improve collaboration across multiple sessions.Source: §8 · p. 8
• The proposed agents use memory designed to learn preferences and improve interaction quality over time.Source: §8 · p. 8
• Simulator behavior in MULTISESSIONCOLLAB supplies learning signals that encourage more comprehensive reflections and more meaningful memory updates.Source: §8 · p. 8
• Experiments and user studies report that memory improves collaboration quality, task success, interaction efficiency, and user effort.Source: §8 · p. 8

9 Limitations

• The Llama-3.3-70B-Instruct simulator struggled to reliably enforce preferences in complex tasks such as coding, so coding was excluded from the benchmark.Evaluation metrics depend on simulator behavior, and the authors note that human simulation may not capture realistic behavior reliably.Source: §9 · p. 9
• Future simulators could capture implicit behavioral cues and preferences that vary across domains or evolve over time.The authors acknowledge these limitations remain despite a human user study validating findings with real users on realistic tasks.Source: §9 · p. 9

A User Interaction Preferences

• The benchmark’s manually curated interaction preferences draw on psychology, cognitive science, and human-computer interaction research.The taxonomy includes preferences for response length, structure, examples, and multiple approaches.Source: A User Interaction Preferences · pp. 13, 15

B Agent Prompts

• The agent prompts direct session-level reflection toward concise, actionable, non-hallucinated preference notes for future interactions.The system also includes prompts for applying preference notes and extracting those relevant to a current request.Source: B Agent Prompts · p. 16
• The user simulator checks preference satisfaction and enforcement, draft updates, and termination, producing signals used to measure user effort and train reflection and memory updates.The prompt makes these checks explicit to support consistent profile adherence and interpretable user behavior signals.Source: B Agent Prompts · p. 17
• The reflection evaluator assesses coverage, actionability, accuracy, and clarity, including whether notes are grounded, organized, and non-redundant.It scores reflections on a 0–3 scale, from major preference gaps or hallucinations to comprehensive, actionable, hallucination-free notes.Source: B Agent Prompts · p. 18

C Environment Validation

• The ablations evaluate Llama-3.3-70B-Instruct across settings that incrementally add user interaction, preferences, oracle preference descriptions, and memory.S1 is direct problem-solving; S2 adds multi-turn interaction without preferences; S3 adds preferences; S4 provides them at session start; S5 adds memory for learning and retaining them across sessions.Source: C Environment Validation · pp. 18–19
• Moving from direct problem-solving in S1 to multi-turn interaction in S2 produces a notable performance drop, consistent with challenges handling underspecified problems.Source: C Environment Validation · p. 19
• Introducing preferences in S3 further decreases performance; failures to adhere make users repeat their needs and add friction to task progression and collaboration.The passage notes that the drop may reflect difficulty either providing information while following preferences or digesting that information.Source: C Environment Validation · p. 19
• Providing preference descriptions in S4 improves performance, but the gap from S2 indicates that knowing preferences alone is insufficient for effective adherence.Source: C Environment Validation · p. 19

D Training Hyperparameters

• Training reflection models raised risks of catastrophic forgetting, degraded response generation, and overly long outputs favored by GRPO optimization and LLM judges.The authors say careful hyperparameter selection was needed to mitigate these issues and preserve general capabilities.Source: D Training Hyperparameters · p. 19
• The training setup uses LLaMa-Factory for SFT and VERL for GRPO, with unlisted hyperparameters set to defaults.Table 5 summarizes the hyperparameters used for both training stages.Source: D Training Hyperparameters · p. 19

E Memory vs. Full Conversation History

• Conversation history averages ∼21,700 tokens, compared with ∼216 tokens for accumulated memory, making memory about 100x shorter on average.Source: E Memory vs. Full Conversation History · p. 19
• A single experiment using all previous sessions took roughly 17 hours on 8xH100s, and both history variants are significantly more expensive than memory.Source: E Memory vs. Full Conversation History · p. 20
• Memory outperforms both the last-three-session and all-previous-session history variants across all metrics; providing history as context lowers task success.Agents given full history often produce incoherent, meaningless responses that do not address user requests.Source: E Memory vs. Full Conversation History · pp. 19–20

G Real-World User Study

• Nineteen university volunteers completed four three-session studies crossing memory conditions with coding-only or mixed-domain tasks.Problems varied in structure, and participants could introduce preferences beyond those assigned.Source: G Real-World User Study · pp. 20–21
• Agents with memory improved collaboration across sessions in both coding and mixed-domain studies and across preference types.The study reports consistent trends of improvement relative to agents without memory.Source: G Real-World User Study · p. 23
• By session three, median conversation length was 6 turns for coding and 4 for mixed-domain tasks with memory, versus 10 and 8 without memory.In session one, the median was 8 turns regardless of memory.Source: G Real-World User Study · p. 23
• Preference-adherence medians rose from 3 in session one to 5 in session three, while preference-memory medians rose from 1 to 5 across settings.Source: G Real-World User Study · p. 23
• Participants described memory-equipped agents as more personalized and effective, including when queries were vague, and said agents proactively followed preferences.Reflections also became more detailed and captured user-specific preferences such as readability and presentation style.Source: G Real-World User Study · p. 24
• Participants found personalization less effective in mixed-domain settings and less effective than explicitly restating preferences within a session; memory did not yet capture a complete preference representation.Source: G Real-World User Study · p. 24