PaperlayerView exact revision on arXiv

Source-linked AI summary

ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support

Tiantian Chen, Jiaqi Lu, Ying Shen, Lin Zhang

arXiv:2602.01885v1cs.CLcs.AI

TL;DR

Long-term emotional-support agents must integrate dispersed, implicit, and evolving user information, which existing fact-centric benchmarks do not adequately evaluate. The paper introduces ES-MemEval and EvoEmo to assess and support this setting, finding that explicit memory improves reliability and personalization while RAG improves factual consistency but struggles with temporal change.

  • Problem

    Existing long-term dialogue benchmarks focus on static, explicit fact retrieval rather than dispersed, implicit, and evolving information required for emotional-support interactions.

  • Method

    The paper introduces ES-MemEval, which evaluates five memory capabilities across QA, summarization, and dialogue generation, and EvoEmo, a multi-session dataset of evolving user states.

  • Results

    Experiments show that explicit long-term memory reduces hallucinations and supports personalization, while RAG improves factual consistency but struggles with temporal dynamics and evolving user states.

  • Takeaways & Limitations

    Personalized long-term emotional support requires integrating memory with retrieval while accounting for temporal dynamics and evolving user states.

  • Takeaways & Limitations

    EvoEmo is synthetic and may diverge from real-world conversational dynamics despite real-data-derived profiles and human verification.

Abstract

from arXiv · show

Large Language Models (LLMs) have shown strong potential as conversational agents. Yet, their effectiveness remains limited by deficiencies in robust long-term memory, particularly in complex, long-term web-based services such as online emotional support. However, existing long-term dialogue benchmarks primarily focus on static and explicit fact retrieval, failing to evaluate agents in critical scenarios where user information is dispersed, implicit, and continuously evolving. To address this gap, we introduce ES-MemEval, a comprehensive benchmark that systematically evaluates five core memory capabilities: information extraction, temporal reasoning, conflict detection, abstention, and user modeling, in long-term emotional support settings, covering question answering, summarization, and dialogue generation tasks. To support the benchmark, we also propose EvoEmo, a multi-session dataset for personalized long-term emotional support that captures fragmented, implicit user disclosures and evolving user states. Extensive experiments on open-source long-context, commercial, and retrieval-augmented (RAG) LLMs show that explicit long-term memory is essential for reducing hallucinations and enabling effective personalization. At the same time, RAG improves factual consistency but struggles with temporal dynamics and evolving user states. These findings highlight both the potential and limitations of current paradigms and motivate more robust integration of memory and retrieval for long-term personalized dialogue systems.

1 Introduction

Existing benchmarks underrepresent long-term emotional-support memory because they emphasize explicit, stable fact retrieval rather than fragmented, implicit, evolving user information. ES-MemEval and EvoEmo address this gap through three tasks, five capabilities, and multi-session data, with experiments showing distinct strengths and limitations of current LLM paradigms.

  • Motivation: Long-term emotional support requires tracking evolving user states and integrating implicit, fragmented disclosures across sessions.Robust memory supports personalized, coherent responses and can mitigate hallucinations in sensitive services.
  • Motivation: Existing benchmarks primarily test static, explicit fact retrieval, overlooking abstraction and reasoning over evolving user information.They therefore cover only a limited facet of long-term memory needed for emotional-support dialogue.
  • ES-MemEval: ES-MemEval evaluates question answering, summarization, and dialogue generation across information extraction, temporal reasoning, conflict detection, abstention, and user modeling.The tasks respectively assess retrieval, cross-session abstraction, and personalized use of long-term memory.
  • EvoEmo: EvoEmo is a multi-session dataset designed to capture evolving user states and fragmented, implicit disclosures for long-term emotional support.It combines real emotional-support sessions with generated sessions based on structured user profiles and event timelines.
  • Experiments: Experiments across open-source, commercial, and RAG LLMs show that explicit histories reduce hallucinations, while RAG improves factual consistency but struggles with temporal dynamics.The results also link personalization to long-term memory and indicate that session-level retrieval better captures evolving information but can add redundancy.

2 Related Work

Prior long-term dialogue benchmarks increasingly test retrieval through QA, but remain concentrated on explicit facts and narrow formats. Emotional-support resources likewise mostly cover short or single sessions, motivating evaluation of implicit, fragmented, evolving trajectories.

  • Long-Term Dialogue Benchmarks: Recent long-term dialogue benchmarks shifted toward QA-style evaluations to test retrieval and reasoning more directly.Examples include multi-day dialogues and extended child–assistant conversations evaluated for cross-session or proactive recall.
  • Long-Term Dialogue Benchmarks: Many benchmarks remain fact-centric and explicit, limiting assessment of implicit expressions, fragmented information, and evolving user states.Their narrow QA or retrieval formats also constrain cross-session reasoning and personalized response generation.
  • Research Gap: ES-MemEval addresses these gaps by combining QA, summarization, and dialogue generation for long-term emotional-support dialogues.The benchmark evaluates memory utilization and personalized adaptation across broader interaction formats.
  • Emotional Support Dialogue: Existing emotional-support datasets largely focus on limited-turn or single-session dialogues rather than long-term user trajectories.Long-term support spanning days or weeks creates greater demands for memory utilization and personalized adaptation.

3 EvoEmo Dataset

EvoEmo is generated through a three-stage pipeline that constructs user profiles, expands event timelines, and produces multi-session chats. The process is designed to represent longitudinal state changes, implicit disclosures, and cross-session consistency.

  • Generation Pipeline: EvoEmo’s pipeline consists of user profile construction, event timeline expansion, and chat data generation.These stages aim to simulate realistic long-term emotional-support conversations.
  • User Profile Construction: The dataset uses 18 virtual users with diverse profiles manually curated from seed emotional-support dialogues.Profiles include demographic information, social relationships, core beliefs, and initial histories reflecting implicit and fragmented disclosures.
  • Event Timeline Expansion: Event timelines add timestamps and descriptions to create longitudinal and causal structure beyond the short-term seed sessions.GPT-4o generated initial events from the sessions, and human annotators refined them before iterative expansion.
  • Chat Data Generation: GPT-4o generates multi-turn sessions conditioned on the current event, user profile, and relevant prior-session summaries.Six annotators subsequently reviewed sessions for consistency within and across users’ trajectories.
  • Dataset Outcome: The resulting dataset provides a structured, curated testbed for long-term emotional support and longitudinal user modeling.Its construction explicitly targets evolving states and implicit, fragmented user disclosures.

4 ES-MemEval Benchmark

ES-MemEval operationalizes long-term memory evaluation through QA, summarization, and dialogue-generation sets covering complementary capabilities and interaction formats. Its protocols combine automatic, semantic, event-based, observational, and LLM-based assessments.

  • Task Design: ES-MemEval comprises QA, summarization, and dialogue-generation tasks for evaluating long-term memory capabilities.The three formats assess retrieval and integration, cross-session state dynamics, and personalized emotional-support responses.
  • Core Capabilities: The benchmark defines information extraction, temporal reasoning, conflict detection, abstention, and user modeling as its five core capabilities.These cover identifying facts, tracking event relations, handling contradictions, withholding unsupported answers, and updating user traits or states.
  • Evaluation Sets: The evaluation sets are constructed separately for QA, summarization, and dialogue generation and are reported in Table 2.QA samples include question types, references, and supporting evidence, while summarization and dialogue cases require cross-session abstraction or personalized interaction.
  • Evaluation Protocols: QA uses F1-Score, BERTScore, and LLM-as-Judge, while summarization uses ROUGE, LLM-as-Judge, and event-based metrics.These protocols combine lexical overlap with semantic consistency, faithfulness, and event-level assessment.
  • Evaluation Protocols: Dialogue evaluation combines observation-based metrics with LLM ratings of long-term memory, personalization, and emotional support.GPT-4o simulates users and rates overall dialogue quality on a 5-point scale.

5 Experimental Setup

The experiments evaluate open-source long-context models, commercial models, and retrieval-augmented variants on ES-MemEval.

  • The study compares open-source long-context models, commercial models, and retrieval-augmented configurations.Open-source models support 128K-token contexts, while the commercial baselines include gpt-3.5-turbo and gpt-4o with shorter context windows.

6 Experimental Results

Across QA, retrieval, context-length, summarization, and dialogue-generation analyses, retrieval generally improves factual and summary quality, but benefits remain uneven across memory capabilities and context settings.

  • QA performance: RAG raises Mistral-24B QA F1 from 15.5 to 18.8, BERTScore from 47.4 to 50.4, and LLM-as-Judge from 1.01 to 1.27.The gains are especially pronounced for smaller models, while user modeling and temporal reasoning remain weak.
  • QA performance: RAG improves factual recall but does not uniformly benefit all capabilities, with persistent difficulty in long-term user modeling.Abstention improves for open-source models but declines for commercial models, including GPT-4o.
  • Retrieval configurations: Session-level retrieval reaches an LLM-as-Judge score of 1.27 at k=4, exceeding round-level 1.20 and turn-level 1.15.The comparison evaluates Mistral-24B under different retrieval granularities.
  • Context length: Long-context performance deteriorates as input length increases, especially for Mistral-8B, while Mistral-24B remains more robust at 20K tokens.Without RAG, Mistral-8B peaks at 2K tokens and Mistral-24B at 8K tokens.
  • Summarization: RAG raises Mistral-24B summarization ROUGE-L from 10.9 to 21.0, event-level F1 from 26.8 to 48.1, and LLM Score from 1.45 to 2.79.Mistral-24B + RAG surpasses GPT-3.5-turbo + RAG in event-level evaluation.
  • Dialogue generation: Explicit histories improve Recall and Weighted Score across models, while RAG raises Mistral-24B’s Weighted Score from 0.33 to 0.41.No-Mem scores can reflect hallucinated user experiences rather than genuine memory use.
  • Dialogue generation: Long-term history improves memory, personalization, and emotional-support scores, while RAG can provide comparable support with less context.Personalization strongly correlates with long-term memory, whereas emotional-support scores are less sensitive to memory access.

7 Discussion and Future Directions

The discussion identifies explicit memory as essential for reliable personalization, while positioning RAG as useful but limited by temporal reasoning and retrieval-design challenges.

  • Without explicit histories, models may hallucinate user experiences, undermining reliability and personalization.
  • RAG improves factual consistency and alignment with user observations, yet nuanced temporal dynamics remain challenging.The authors motivate retrieval-aware calibration.
  • Personalization depends strongly on long-term memory, whereas emotional support can partly rely on general strategies with limited memory.
  • Session-level retrieval better captures sparse and evolving user signals, although redundancy remains a concern.
  • Smaller models degrade with extended contexts, motivating hybrid memory–retrieval designs.
  • RAG narrows the performance gap between open-source and commercial models by improving personalization and memory alignment.

8 Conclusion

The paper introduces EvoEmo and ES-MemEval to benchmark long-term memory for personalized emotional support across multiple tasks and memory capabilities.

  • EvoEmo contains 18 user trajectories with evolving states across multiple sessions.
  • ES-MemEval evaluates information extraction, temporal reasoning, conflict detection, abstention, and user modeling through QA, summarization, and dialogue generation.
  • Experiments across open-source, commercial, and retrieval-augmented models provide empirical insights into long-term memory, personalization, memory granularity, and retrieval strategies.
  • The benchmark supports development of reliable, user-centered dialogue systems in complex long-term settings.

A Limitations

The paper identifies limitations in EvoEmo’s realism, scale, representativeness, retrieval coverage, and deployment scope. These constraints bound how broadly the benchmark’s findings should be generalized.

  • Dataset realism: EvoEmo is synthetic, so its conversations may diverge from real-world conversational dynamics despite human review and real-data-derived profiles.The dataset was generated with GPT assistance and refined through human review to address ethical and collection-cost concerns.
  • Dataset scale: The dataset remains relatively small compared with large-scale general dialogue corpora, motivating future expansion in users and sessions.Its scale is comparable to or greater than some domain benchmarks, including LOCOMO and MemoryBank.
  • Representativeness: EvoEmo’s eight dialogue-topic categories are imbalanced and do not account for cross-cultural diversity.Self-growth is underrepresented, reflecting skew in the ESConv source dataset.
  • Retrieval scope: The experiments do not explore alternative retrieval algorithms or finer-grained memory units because the work prioritizes benchmark establishment over retrieval optimization.Future work is positioned to investigate methods such as BM25, DRAGON, user observations, summaries, and compressed contexts.
  • Deployment boundary: ES-MemEval is intended for research rather than real-world counseling or clinical deployment, and sensitive scenarios may elicit hallucinated or inconsistent histories.The authors recommend professional oversight for such uses.

C.2 More Dataset Statistics

EvoEmo covers diverse emotional-support topics and long user histories, while ES-MemEval balances within-session reasoning with cross-session user modeling. Its task distributions emphasize temporal reasoning and user modeling in summarization.

  • Topic distribution: EvoEmo covers eight dialogue-topic categories, led by emotion and mood, career and study, social and relationship, and love and intimacy.Family issues, self-growth, treatment and help-seeking, and behavior issues are also represented at varying frequencies.
  • Statistical overview: Figures 4–6 summarize topic distributions, task types, evidence-session counts, and user-level dialogue-history spans.Together, these statistics characterize both the content diversity and longitudinal structure of the benchmark.
  • Task distribution: The QA benchmark covers information extraction, temporal reasoning, conflict detection, user modeling, and abstention in a relatively balanced distribution.The summarization task places greater emphasis on temporal reasoning and user modeling.
  • Evidence requirements: QA primarily requires multiple utterances within one session, whereas summarization often aggregates evidence across multiple sessions.This balances within-session reasoning with cross-session integration and user-trajectory modeling.
  • Longitudinal coverage: EvoEmo user histories span an average of 448 days, with approximately 22 sessions per user.The reported range is 304–553 days, indicating interactions extending over several months to years.

D.1 Evaluation Metrics

ES-MemEval evaluates QA, summarization, and dialogue generation with task-specific LLM-as-Judge protocols and observation-based measures. Reliability checks show strong human agreement for QA and summarization, while dialogue-generation interpretation requires MAD and exact agreement because of ceiling effects.

  • LLM-as-Judge protocol: Each of the three tasks receives a dedicated LLM-as-Judge prompt tailored to its evaluation requirements.Condensed summarization and dialogue-generation prompts are shown in Figure 7, while the full prompts are intended for release.
  • Dialogue generation: Dialogue generation is evaluated by whether responses leverage relevant user observations, using observation recall and a finer-grained incorporation measure.Observations describe objective user states, experiences, or contextual facts relevant to the current input.
  • Question answering: The QA judge scores answers from 0 to 2 for correctness and contextual accuracy against gold answers.A score of 1 denotes partial correctness, while 0 denotes an incorrect or irrelevant answer.
  • Reliability: For dialogue generation, MAD and exact agreement are more informative than Kappa and Spearman because human ratings are heavily concentrated at the maximum.Reported MAD values are 0.40 for Mistral-24B+full and 0.17 for Mistral-24B+RAG, with exact agreement rates of 70% and 86.7%.
  • Reliability: QA and summarization show strong agreement with human judgments, with Kappa and Spearman values above 0.6 and MAD below 0.5.The reliability analysis compares human evaluations with LLM-as-Judge scores across sampled examples.
Loading 2602.01885v1…