ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support

1 Introduction

• Long-term emotional support requires tracking evolving states and integrating implicit, fragmented disclosures [21] [41], but existing benchmarks inadequately test these contexts [24] and mostly focus on static, explicit fact retrieval [42].This leaves untested the integration, abstraction, and application of user information across emotionally complex interactions; robust long-term memory is presented as essential for personalization, coherence, and mitigating hallucinations that threaten user trust [12].Source: §1 · pp. 1–2
• ES-MemEval evaluates long-term memory across question answering, summarization, and dialogue generation, covering information extraction, temporal reasoning, conflict detection, abstention, and user modeling.The tasks respectively assess retrieval and comprehension across extended interactions, synthesis of user-state dynamics, and use of long-term memory for personalized emotional support.Source: §1 · p. 2, Figure 2 · p. 2
• EvoEmo is a multi-session emotional-support dataset for 18 virtual users, averaging 510 turns (≈13.3k tokens) across up to 33 sessions per user.It combines real emotional-support sessions with sessions generated from detailed user profiles and temporally and causally structured event timelines, capturing fragmented disclosures and longitudinally evolving states.Source: §1 · pp. 2–3
• Experiments find that explicit histories reduce hallucinated user experiences, while RAG [19] improves factual consistency but struggles with temporal dynamics and evolving states.Personalization correlates strongly with memory, whereas emotional support is less memory-sensitive; session-level retrieval best captures evolution but can add redundancy, and extra-long inputs degrade smaller long-context models. RAG also narrows open-source/commercial gaps in personalization and memory alignment.Source: §1 · p. 3

2.1 Long-Term Dialogue Benchmarks

• Prior long-term dialogue benchmarks increasingly use QA to test retrieval and reasoning, spanning multi-day, extended, large-scale, skill-based, and child–assistant settings, [25],, [42], [11].Reported examples include 15 virtual users and 194 QA samples, 10 extended dialogues [25], 3,409 dialogues and 8,593 questions, 500 skill-testing questions [42], and 160 simulated child–assistant dialogues [11].Source: §2.1 · p. 3
• Fact-centric QA benchmarks miss implicit, fragmented disclosures and evolving states; ES-MemEval adds QA, summarization, and dialogue generation to evaluate long-term emotional-support memory and personalization [42].Existing narrow QA or retrieval formats also constrain evaluation of cross-session reasoning and personalized response generation.Source: §2.1 · p. 3

2.2 Emotional Support Conversation

• Emotional-support research includes short empathetic scenario dialogues, strategy-annotated ESConv, platform counseling and PsyDial datasets, and generation methods for guidance, reflection, and self-disclosure [35], [24], [20],, [8].ESConv contains 1,035 multi-turn dialogues, and Rashkin et al. collected approximately 25k short empathetic dialogues [24], [35].Source: §2.2 · p. 3
• Most emotional-support efforts focus on limited-turn or single-session dialogues, while EvoEmo captures evolving user states and ES-MemEval evaluates long-term memory for personalized adaptation.Emotional support can span days or weeks, increasing demands on long-term memory and personalization.Source: §2.2 · p. 3

3 EvoEmo Dataset

• EvoEmo’s three-stage pipeline builds a long-term emotional-support dataset to capture dynamic user-state evolution.The stages are user profile construction, event timeline expansion, and chat data generation.Source: §3 · p. 3, Figure 3 · p. 4

3.1 User Profile Construction

• 18 virtual users received manually curated profiles with diverse traits, demographics, relationships, and beliefs, based on ESConv seed dialogues [24].The seed dialogues provide implicit, fragmented disclosures and evolving states as the initial dialogue history.Source: §3.1 · pp. 3–4

3.2 Event Timeline Expansion

• GPT-4o generated and iteratively expanded each user’s event timeline, with human review for temporal and causal consistency, averaging 24.8 events per user.The timelines model state evolution and capture causal and experiential variation beyond prior static user profiles.Source: §3.2 · p. 4

3.3 Chat Data Generation

• GPT-4o generated multi-turn sessions conditioned on profiles, current events, and relevant prior-session summaries; six annotators reviewed consistency within and across sessions.Sessions include emotion, topic, summary, and turn-level user-observation annotations, forming a curated testbed for longitudinal user modeling.Source: §3.3 · p. 4

4.1 Task Formulation

• ES-MemEval tests information extraction, temporal reasoning, conflict detection, abstention, and user modeling to track evolving user states and support personalization.These capabilities cover identifying facts across sessions, inferring event order and causal dependencies, resolving contradictions, withholding unsupported answers, and updating user traits, preferences, and states.Source: §4.1.1 · p. 4

4.1.2 Evaluation Task Formats.

• Question answering spans all five capabilities; summarization targets temporal reasoning and user modeling, while dialogue generation tests context understanding and integrated emotional-support responses.The three formats respectively assess cross-session retrieval and integration, user-state dynamics, and realistic personalized interactions.Source: §4.1.2 · p. 4

4.2 Evaluation Sets

• The QA set contains 1,209 GPT-4o-generated samples based on users’ sessions and timelines, each with a question type, reference answer, and supporting evidence passage.One of six annotators reviewed and corrected each sample for type mismatches, answer errors or omissions, and missing evidence.Source: §4.2.1 · p. 5

4.2.2 Summarization.

• The summarization set contains 125 cross-session cases built from thematic session groups, with questions and candidate answers reviewed and refined by two annotators.Each case requires extracting and integrating user states, temporal event sequences, and behavioral logic across sessions.Source: §4.2.2 · p. 5

4.2.3 Dialogue Generation.

• The dialogue-generation set contains 34 manually reviewed topics grounded in users’ timelines and summaries, designed for extended conversations about evolving experiences.Each topic specifies an overview, details, the user’s physical and psychological state, and relevant prior sessions.Source: §4.2.3 · p. 5

4.3 Dataset Statistics and Analysis

• EvoEmo averages 27.2 sessions and 13.3K tokens per conversation; ES-MemEval includes 1,209 balanced QA questions, 125 summarization cases, and 34 dialogue scenarios.The summarization cases are split between temporal reasoning (59.2%) and user modeling (40.8%).Source: Table 2 · p. 5, §4.3 · p. 5

4.4 Evaluation Protocols

• QA responses are scored with F1-Score, BERTScore, and LLM-as-Judge, while Recall@k and nDCG@k measure retrieval accuracy.GPT-4o assigns LLM-as-Judge scores of 0, 1, or 2 for semantic consistency with the reference answer.Source: §4.4.1 · p. 5

4.4.1 Question Answering.

• Summaries are evaluated with lexical, semantic, and event-based metrics, while dialogue protocols assess user-state reflection and overall long-term support quality.Event metrics compare extracted events using Recall, Precision, and F1; dialogue evaluation reports observation-based recall and weighted accuracy, plus GPT-4o ratings on a 5-point scale across memory, personalization, and emotional support.Source: §4.4.2 · p. 5, §4.4.3 · pp. 5–6

• Three 128K-context open-source models are evaluated: Ministral-8B-Instruct-2410, Phi-3-Medium-128k-Instruct [1], and Mistral-Small-3.1-24B-Instruct-2503.Source: §5 · p. 6
• Commercial baselines are gpt-3.5-turbo [6] with a 4K context window and gpt-4o [29] with a 16K context window.Source: §5 · p. 6
• Retrieval-augmented variants of all five models use bge-m3 [7] to retrieve the top-4 relevant full-session contexts from a FAISS index [15].All experiments ran on an A100 GPU with 80GB of memory.Source: §5 · p. 6

6.2 Analysis of Summarization

• Mistral-24B + RAG raised ROUGE-L from 10.9 to 21.0, event-level F1 from 26.8 to 48.1, and LLM Score from 1.45 to 2.79.It surpassed GPT-3.5-turbo + RAG in event-level evaluation; GPT-3.5-turbo + RAG and GPT-4o + RAG each scored 2.93 on LLM Score, and GPT-4o + RAG achieved the highest event-level F1 at 49.4. RAG improved summary coherence and information richness.Source: §6.2 · pp. 6–7

6.1 Analysis of QA Performance

• Mistral-24B with RAG raised overall F1 from 15.5 to 18.8, BERTScore from 47.4 to 50.4, and LLM-as-Judge Score from 1.01 to 1.27.RAG’s gains were uneven: user-modeling and temporal-reasoning F1 scores seldom exceeded 20.0, and GPT-4o’s LLM Score fell from 1.67 to 1.30. RAG improved factual recall but did not uniformly help every capability.Source: §6.1.1 · p. 6, Table 3 · p. 7

6.1.1 Comparison Across Models.

• Session-level retrieval achieved the highest LLM-as-Judge score for Mistral-24B: 1.27 at k=4, versus 1.20 for round-level and 1.15 for turn-level retrieval.Recall@k exceeded 75% across granularities, while peak NDCG@k ranged from 59% to 63%, indicating strong coverage but moderate ranking quality.Source: §6.1.2 · p. 6, Table 4 · p. 7

6.1.2 Impact of Retrieval Configurations.

• Without RAG, Mistral-8B performed best at 2K tokens and Mistral-24B at 8K, while the larger model remained more robust at 20K.User dialogue histories span 11K–19K tokens; shorter-than-20K contexts require truncation, and performance generally deteriorated as input context length increased.Source: §6.1.3 · p. 6, Table 5 · p. 7

6.3 Analysis of Dialogue Generation

• Against No-Mem, Full-Hist or RAG improved Recall and Weighted Score across models; Mistral-24B rose from 0.20 to 0.33, while RAG reached 0.41 versus Full-Hist’s 0.33 in Weighted Score.No-Mem scores could reflect hallucinated user experiences rather than memory use; retrieval helped models use user-relevant information for more personalized and factually consistent dialogue.Source: §6.3.1 · p. 7

6.3.1 Observation-based Metrics.

• GPT-4o ratings showed that long-term history improved long-term memory, personalization, and emotional-support scores; Full-Hist and RAG performed comparably, with RAG sometimes slightly better.No-Mem models could score relatively high on long-term memory by hallucinating plausible experiences. Personalization correlated strongly with long-term memory, whereas emotional-support scores were less sensitive to memory access.Source: §6.3.2 · p. 7

7 Discussion and Future Directions

• Explicit long-term memory is essential because models without user histories may hallucinate experiences, undermining reliability and personalization.The experiments identify long-term memory as a core requirement for personalized emotional-support dialogues.Source: §7 · pp. 7–8
• RAG improves factual consistency and alignment with user observations, but modeling nuanced temporal dynamics remains challenging.The paper motivates retrieval-aware calibration to address this limitation.Source: §7 · p. 8
• Personalization depends strongly on long-term memory, whereas emotional support can partly draw on general strategies with limited memory.The findings distinguish memory-dependent personalization from emotional support that can rely in part on general strategies.Source: §7 · p. 8
• Session-level retrieval better captures sparse, evolving user signals, though it can introduce redundancy.The paper identifies adaptive memory granularity as a direction for addressing these retrieval tradeoffs.Source: §7 · p. 8
• Smaller models degrade with extended contexts, highlighting the need to integrate memory and retrieval.The paper proposes hybrid memory–retrieval designs for sustained personalized dialogue.Source: §7 · p. 8
• RAG narrows the performance gap between open-source and commercial models by improving personalization and memory alignment.The findings also motivate retrieval-aware calibration alongside hybrid memory–retrieval designs.Source: §7 · p. 8

8 Conclusion

• ES-MemEval evaluates five memory capabilities across question answering, summarization, and dialogue generation in personalized emotional support.The capabilities are information extraction, temporal reasoning, conflict detection, abstention, and user modeling.Source: §8 · p. 8
• EvoEmo comprises 18 user trajectories that capture evolving user states across multiple sessions.Source: §8 · p. 8
• Experiments on open-source long-context, commercial, and retrieval-augmented models provide insights into long-term memory for personalization and the effects of memory granularity and retrieval strategies.The contributions establish an empirically grounded benchmark to support reliable, user-centered dialogue systems in complex, long-term settings.Source: §8 · p. 8

A Limitations

• EvoEmo is synthetic and may diverge from real-world conversational dynamics despite real-data-derived profiles and timelines and manual session-consistency checks.The GPT-assisted dialogues were refined through human review to mitigate ethical concerns and the cost of collecting real long-term support conversations, an approach also used in prior work [13] [16] [18] [25].Source: A Limitations · p. 10
• The dataset remains relatively small, so future work will add users and sessions to improve its representativeness and complexity.Its scale is comparable to or greater than domain benchmarks LOCOMO (272 sessions) and MemoryBank (150 sessions averaging 7.6 turns).Source: A Limitations · p. 10
• ES-MemEval covers eight dialogue-topic categories, but their distribution is imbalanced, self-growth is underrepresented, and cross-cultural diversity is not accounted for.The skew reflects the source dataset ESConv [24]; adding sources is proposed to improve scenario coverage and cultural representativeness.Source: A Limitations · p. 10
• The experiments use common benchmark configurations and leave alternative retrieval algorithms and finer-grained memory units unexplored because the work prioritizes benchmarking over model or retrieval optimization.Future work could examine BM25, DRAGON [23], user observations [25], dialogue summaries [21] [37], and compressed contexts.Source: A Limitations · p. 10

B Ethics Statement

• ES-MemEval simulates sensitive scenarios where models, especially without memory, may hallucinate user experiences or fabricate inconsistent histories, so it is restricted to research rather than counseling or clinical deployment.The synthetic pipeline avoids real counseling or mental-health conversations, and its dialogues were reviewed by trained annotators for coherence, plausibility, and safety.Source: B Ethics Statement · p. 10

C.1 Annotator Details

• Ten English-proficient graduate-student volunteers handled timeline, QA, chat, summarization, and dialogue-generation reviews and were paid 8 USD per hour.Two refined and verified event timelines, six reviewed QA samples and chat data, and two reviewed summarization and dialogue-generation samples.Source: §C.1 · p. 10

C.2 More Dataset Statistics

• EvoEmo dialogues mainly concern emotion and mood and career and study, followed by social and relationship and love and intimacy.Family issues, self-growth, treatment and help-seeking, and behavior issues are also represented.Source: §C.2 · pp. 10–11, Figure 4 · p. 11
• QA covers five relatively balanced question types, whereas summarization emphasizes temporal reasoning and user modeling.The five QA types are information extraction, temporal reasoning, conflict detection, user modeling, and abstention.Source: §C.2 · p. 11, Figure 4 · p. 11
• QA answers primarily require multiple utterances within one session, while summaries aggregate information across sessions to model user trajectories.Together, these evidence requirements balance intra-session reasoning and cross-session integration.Source: §C.2 · p. 11, Figure 5 · p. 11
• User dialogue histories span an average of 448 days (median 458, ranging from 304 to 553 days), capturing interactions over months to years.Figure 6 illustrates these user-level spans from first to last recorded interaction.Source: §C.2 · p. 11, Figure 6 · p. 11
• EvoEmo contains approximately 22 sessions per user.These sessions accompany dialogue histories lasting an average of 448 days.Source: §C.2 · p. 11

D.1 Evaluation Metrics

• Each of the three tasks—QA, summarization, and dialogue generation—has a dedicated LLM-as-judge prompt, with compressed summarization and dialogue prompts shown in Figure 7.The full prompts are to be released in the code repository for transparency, reproducibility, and future benchmarking.Source: §D.1 · p. 11, Figure 7 · p. 12
• For dialogue generation, the observation-based protocol evaluates system use of a candidate set of objective user-state, experience, and contextual descriptions extracted from scenario-related conversations.The protocol is designed to measure whether responses use user information.Source: §D.1 · p. 11
• Mistral-24B scores each observation’s relevance to the current input as 0, 0.5, or 1, then judges whether responses use observations scored 0.5 or 1.Use may be explicit or implicit.Source: §D.1 · p. 12
• Observation Recall measures the proportion of fully relevant observations reflected in responses, while Weighted Accuracy aggregates incorporation of fully and partially relevant observations.Weighted Accuracy provides a finer-grained measure of how well systems incorporate available user information.Source: §D.1 · p. 12

D.2 Reliability Analysis of LLM-as-Judge Evaluations

• QA and summarization show strong human–judge agreement (Kappa > 0.6, Spearman > 0.6, MAD < 0.5), supporting the protocol’s reliability on these tasks.The evaluation compared human ratings of Mistral-24B+Full and Mistral-24B+RAG.Source: Table 9 · p. 12, §D.2 · p. 12
• For dialogue, MAD is 0.40 for Mistral-24B+full and 0.17 for Mistral-24B+RAG, while exact agreement is 70% and 86.7%, respectively.These measures indicate meaningful distinctions and overall alignment with human judgments despite lower Kappa and Spearman values.Source: §D.1 · p. 12, §D.2 · p. 12
• Dialogue agreement is lower for RAG than Full: Kappa is 0.19 versus 0.57, and Spearman is 0.20 versus 0.61, partly due to ceiling effects in human ratings.Human ratings skewed toward the maximum because general model quality was highly positive.Source: §D.2 · p. 12