PaperlayerView exact revision on arXiv

Source-linked AI summary

MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation

Junqing He, Liang Zhu, Rui Wang, Xi Wang, Reza Haffari, Jiaxing Zhang

arXiv:2409.15240v2cs.CLcs.AI

TL;DR

Existing memory-augmented dialogue evaluations emphasize passive retrieval and language quality, leaving proactive recall and human-centered response assessment underrepresented. MADial-Bench evaluates passive and proactive recall in separate memory-search and recognition tasks and adds memory-aware and emotional-support criteria. Its experiments report unsatisfactory recall, relationships among memory injection, emotional-support proficiency, and intimacy, and gaps from human performance.

  • Problem

    Common MADS evaluations focus on query-oriented factualness or language quality, and cover passive recall while omitting proactive memory search and emotional-support dimensions.

    Source: §1 · p. 1

  • Method

    MADial-Bench uses cognitive-science-based passive and proactive recall data in separate memory retrieval and recognition tasks, with criteria including memory injection, ES proficiency, and intimacy.

    Source: §1 · p. 2, §3.3.2 · p. 4, §4.3 · p. 6

  • Results

    Experiments find unsatisfactory memory recall, relationships among memory injection, ES proficiency, and intimacy, and substantial gaps from human performance in memory injection and overall performance.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The reported results identify memory recall, recognition, and dialogue generation as areas where the tested systems lag behind human performance.

    Source: §1 · p. 2

    The study covers retrieval-based dialogue systems and bilingual data, while human evaluation is difficult to conduct consistently across many results.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Long-term memory is important for chatbots and dialogue systems (DS) to create consistent and human-like conversations, evidenced by numerous developed memory-augmented DS (MADS). To evaluate the effectiveness of such MADS, existing commonly used evaluation metrics, like retrieval accuracy and perplexity (PPL), mainly focus on query-oriented factualness and language quality assessment. However, these metrics often lack practical value. Moreover, the evaluation dimensions are insufficient for human-like assessment in DS. Regarding memory-recalling paradigms, current evaluation schemes only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors, e.g., emotions and surroundings, which can be essential in emotional support scenarios. To bridge the gap, we construct a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) covering various memory-recalling paradigms based on cognitive science and psychology theories. The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. We introduce new scoring criteria to the evaluation, including memory injection, emotion support (ES) proficiency, and intimacy, to comprehensively assess generated responses. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement. Extensive testing further reveals correlations between memory injection, ES proficiency, and intimacy.

1 Introduction

MADial-Bench addresses gaps in memory-augmented dialogue evaluation by covering passive and proactive recall and assessing memory use alongside emotional-support qualities. Its design draws on cognitive science and psychology theories.

1 Introduction

  • Existing MADS evaluation is limited to question-answering and language-modeling assessment rather than realistic multi-turn dialogue, and relies on passive recall triggered by user mentions.The passage also notes that parameter-based or learned side-network memory is not scalable as the number of users grows.Source: §1 · p. 1
  • Current MADS use memory as conversational common ground but overlook proactive recall for emotional support, although emotional states can trigger recall.The paper situates this gap in long-term memory’s role in emotion regulation.Source: §1 · pp. 1–2
  • MADial-Bench introduces a benchmark for multi-recall, multi-stage evaluation of memory-augmented dialogue systems, grounded in cognitive science and psychology.It represents memories through states such as emotions and scenes, and separates memory search from recognition.Source: §1 · p. 2
  • The benchmark evaluates passive recall from user-mentioned events and proactive recall from emotions or scenes for interpersonal emotional regulation.Human refinement by people with high emotional intelligence and emotional-support guidelines shape its dialogues and assessment.Source: §1 · p. 2
  • MADial-Bench assesses retrieval, recognition, memory injection, emotional-support proficiency, and intimacy, extending response evaluation beyond language style and naturalness.The paper reports that careful prompting can yield higher naturalness and style coherence, while models lag behind humans in memory recognition and dialogue generation.Source: §1 · p. 2

2 Related Works

Prior evaluations of memory-augmented dialogue systems focus on passive recall or language-modeling tasks, while emotional-support evaluations omit long-term memory and intimacy. MADial-Bench adds memory-aware response aspects and detailed scoring guidance.

2.1 Evaluation of Memory-Augmented Dialogue Systems

  • Memory-augmented dialogue evaluations use retrieval accuracy, response correctness, and coherence, but their test sets cover passive recall and omit proactive memory search for emotional regulation.Long-context language-modeling evaluations use novels or academic papers and metrics such as perplexity and classification accuracy, unlike realistic conversation.Source: §2.1 · pp. 2–3

2.2 Evaluation of Emotional Support DS

  • Existing emotional-support dialogue metrics include empathy, relevance, fluency, and—in advice-seeking settings—identification, comforting, and suggestion, but do not inspect long-term memory augmentation or intimacy.MADial-Bench merges the advice-seeking steps into ES Proficiency and adds Memory-injection Ability and Intimacy.Source: §2.2 · p. 3
  • MADial-Bench uses Likert scoring with explicit criteria; Naturalness has three sub-aspects, each scored from 0 to 1, for a maximum of 3.The passage says Table 1 lists differences between MADial and previous works.Source: §2.2 · p. 3

3 MADial-Bench

MADial-Bench is a simulated memory-augmented dialogue benchmark built around passive factual recall and proactive emotional recall. It separates memory retrieval from memory recognition and response generation, with intimacy evaluation included.

3 MADial-Bench

  • 171 historical memories and 160 dialogues across 1,474 turns form the benchmark, generated with GPT-4 and refined through multiple manual rounds.The simulation features two child users and a virtual assistant with event-summary memories.Source: §3 · p. 3

3.1 Overview

  • MADial-Bench tests proactive recall of emotional memories and passive recall of objective facts, using dialogues of 8–12 turns and memories with time, emotion, scene, and event.Proactive categories cover happiness, sadness, anxiety, and disappointment; passive categories cover activities, objects, and social relationships.Source: §3.1 · p. 3

3.2 Data Contruction

  • The authors selected four emotions from 18 psychological categories and defined five user-scene types; expert-developed emotional-support guidelines inform dialogue construction and scoring.The five scenes are Preferences, Activity, Disease, Emotions, and Others.Source: §3.2 · p. 4
  • GPT-4 generated dialogues and memories, an embedding model ranked memory collections to select golden memories, and human review corrected cases and refined reference responses.When the original dialogue memory was not among the selected relevant memories, the dialogue and introduced memory were modified.Source: §3.2 · p. 4

3.3 Task Definition

  • The benchmark divides memory-augmented dialogue into memory recall, analogous to retrieval, and memory recognition with response generation, analogous to generation.These tasks correspond to the two phases of the two-stage memory theory.Source: §3.3 · p. 4

3.3.1 Memory Recalling

  • For each dialogue, M1 retrieves a suitability-ranked golden memory set from the memory bank; each set contains one to seven memories.The set is denoted Si and the dialogue Di.Source: §3.3.1 · p. 4

3.3.2 Memory Recognition and Response Generation

  • M2 selects an appropriate memory and generates a response under three settings: a single golden memory, that memory plus four negatives, or retrieval results from the best embedding model.The settings respectively probe injection, recognition plus injection, and a more realistic retrieval-based condition.Source: §3.3.2 · pp. 4–5

3.3.3 Intimacy and Memory Usage

  • Annotators compare responses with and without memory in Setting 3 for intimacy, and choose the most intimate response among candidates in Task 2.Five annotators make the with-versus-without-memory comparison, with ties allowed.Source: §3.3.3 · p. 5

4 Experiments

The experiments evaluate memory retrieval and memory-injected response generation using embedding and language models. They combine automatic overlap metrics with human judgments of response quality and memory use.

4 Experiments

  • The experiments evaluate embedding models for memory retrieval and LLMs for memory-injected response generation using human and automatic measures.The paper frames this section around recognition and response generation.Source: §4 · p. 5

4.1.1 Embedding Models

  • The retrieval evaluation tests English and Chinese embedding models, excluding models above 7B parameters because of slow inference.The tested models are described as top-ranked in retrieval tasks on the ETEB Leaderboard at the time of the study.Source: §4.1.1 · p. 5

4.1.2 Metrics

  • Retrieval metrics include MAP, MRR, nDCG, Recall, and Precision, with their mean calculated across scopes.MRR emphasizes the top-ranked memory, MAP treats relevant summaries uniformly, and nDCG accounts for relevance and position.Source: §4.1.2 · p. 5

4.2.1 Models

  • The evaluated LLMs span English and Chinese model series and sizes, and all are tested with zero-shot prompting.The listed models include Qwen2-7B-Instruct, GPT-4turbo, GLM-4-0520, Llama3-8B-Instruct, Llama3-70B-Instruct, and GPT-4o.Source: §4.2.1 · p. 5

4.3 Metrics

  • Response evaluation combines BLEU, Rouge-L, and BertScore with human ratings of naturalness, style coherence, memory injection, ES proficiency, emotion improvement, and intimacy.Human evaluation uses aspect-aware scoring and a pick-the-best comparison; ES proficiency considers at most three dialogue turns.Source: §4.3 · pp. 5–6

5 Results and Discussion

Across memory search, recognition, response generation, and intimacy evaluation, results show persistent model–human gaps alongside benefits associated with memory injection. Automatic metrics do not reliably match human judgments.

5.1 Memory Search

  • OpenAI embeddings lead both language datasets, but retrieval remains weak: they do not exceed 60% @1 and struggle at 62% @10 in the final average.BGM-M3 is the strongest open-source model on English, while Acge leads open-source models on Chinese.Source: §5.1 · p. 6, Table 4 · p. 7

5.2 Memory Recognition and Response Generation

  • No LLM surpasses human average performance; GPT4-Turbo leads tasks 1–2, while GLM-4 leads setting 3.Human-written references score higher in most aspects, with emotion improvement from GPT4-Turbo as an exception.Source: §5.2 · p. 6, Table 5 · p. 8
  • Memory injection remains a major gap: humans score higher, while models achieve about half the scores in setting 3 except GLM-4; prompting improves naturalness and style coherence.Given golden memory, models can exceed 2.5, but scores drop when unrelated memories are blended in, except for Ziya-Character.Source: §5.2 · p. 6, Table 5 · p. 8
  • With a 3.0 memory-injection score, models reach a 3.0 ES score in 61% of cases; GPT4-Turbo, Qwen2-72B, and GLM-4 exceed humans on Emotional Improvement.The figure shows the probability of a 3.0 ES score growing with memory-injection score.Source: §5.2 · pp. 6–7, Figure 3 · p. 7

5.3 Intimacy and Memory Usage

  • Memory improves intimacy in at least 69.4% of comparisons; GPT4-Turbo’s gain is 61.9%, and 73% of human-selected best responses contain memory.The authors relate intimacy performance to memory-injection ability and ES proficiency.Source: §5.3 · p. 7, Table 6 · p. 8, Table 7 · p. 8

5.4 Automatic Evaluation

  • Automatic rankings diverge from human judgments: GLM-4 leads semantic similarity, Qwen2-72B leads Rouge-L, and Ziya-Character leads Dist-1.The authors report that GPT-4 judging was unreliable because it forgot criteria and gave faulty reasoning.Source: §5.4 · pp. 7–8

6 Conclusion

MADial-Bench is a bilingual benchmark for evaluating passive and proactive memory recall in memory-augmented dialogue systems. Its experiments identify links among memory injection, emotional-support proficiency, and intimacy, while finding a gap from human performance.

6 Conclusion

  • MADial-Bench evaluates memory-augmented dialogue systems across passive and proactive recall, memory recall, recognition, and injection.The benchmark is grounded in cognitive science and includes human-centred evaluation.Source: §6 · p. 8
  • Automatic measurements on the Chinese testbed are inconsistent with human evaluation.The referenced table presents automatic evaluation results; the supplied passages do not provide its cell values.Source: Table 8 · p. 8
  • The experiments find that memory injection ability is relevant to emotional-support proficiency, and models with higher injection ability can be more intimate.The conclusion also reports a substantial gap between the most powerful LLMs and human references in memory injection and overall performance.Source: §6 · p. 8

Limitation

The paper limits its evaluation to retrieval-based memory-augmented dialogue systems and bilingual data. It also outlines the scene definition and basic memory-recalling criteria used in the benchmark.

Limitation

  • The study considers only retrieval-based memory-augmented chatbots and dialogue systems, and focuses experiments on bilingual data.The authors hope the conclusions and insights can be extended to other languages.Source: Limitation · p. 9

A Definition of Scenes

  • The definition of scenes in memory is listed in Table 9.The supplied passage does not give the scene definition itself.Source: A Definition of Scenes · p. 11

B Memory recalling criteria

  • Passive recall is based mainly on context similarity, while proactive recall uses the criteria listed in Table 10.The authors describe these proactive situations as basic examples, not an exhaustive set of real-world cases.Source: B Memory recalling criteria · p. 11

C.1 Scoring criteria of each aspect

The scoring criteria assess response quality, memory use, emotional support, emotional improvement, and intimacy. The accompanying guidelines specify responses for emotional states and user-mentioned historical events.

C.1 Scoring criteria of each aspect

  • Naturalness scores grammar and coreference, contextual relevance, and a short, casual oral style; style coherence scores concision, non-AI-like language, and a non-preachy peer tone.Each aspect is scored out of three points.Source: §C.1 · p. 11
  • Memory-injection ability scores relevant memory use, factual correctness without fabrication, and avoiding repeated memory introductions.For a multi-turn candidate, introducing memory in any one turn is sufficient for the relevant point.Source: §C.1 · p. 11
  • Emotional-support proficiency scores following one or all situation-specific guideline points, with multi-turn responses allowed to distribute those points across turns.Emotional improvement separately scores avoiding hurt, empathy, and improving the user's emotions.Source: §C.1 · pp. 11–12
  • Intimacy is a side-by-side, best-candidate judgment of closeness that requires a positive, helpful response familiar with the user and sharing conversational common ground.It considers the other assessed abilities.Source: §C.1 · p. 12

C.2 ES guidelines

  • When users mention historical events, guidelines adapt recall to activities, objects, or people; responses reference relevant past details and suggest present actions or offer support as appropriate.For people, the guidance distinguishes positive from negative feelings and prescribes different response steps.Source: §C.2 · p. 12
  • When users do not mention a historical event, the guidelines direct the system to assess whether they are happy, sad, anxious, or disappointed and respond accordingly.Source: §C.2 · p. 12
  • For happiness, the guidelines call for raising a relevant historical event and asking whether to repeat the activity; for sadness, they call for comfort, assessing controllability, and then guidance or diversion.Disappointment and anxiety call for identifying the cause, offering sympathy, and helping find a solution.Source: §C.2 · p. 12

D Automatic Evaluation

The automatic evaluation reports results for Chinese and English versions, using language-specific BERTScore models and standard text-overlap metrics. The English table caption identifies setting 3 as more difficult, with lower performance across models.

D Automatic Evaluation

  • BERTScore uses bert-base-uncased for English and bert-base-chinese for Chinese, while BERTScore, Rouge-L, and BLEU are computed with Python libraries.Chinese text is segmented using jieba.Source: D Automatic Evaluation · pp. 12–13
  • Table 11 presents automatic evaluation results for the Chinese version.Source: Table 11 · p. 13
  • Setting 3 yields lower performance across all models, and the smallest LLM has the lowest Rouge-L and BLEU.Source: Table 12 · p. 13

E Correlation of human evaluation

Human annotator correlations are strong in the first two settings and moderate in setting 3.

E Correlation of human evaluation

  • The evaluation calculates Spearman correlations between annotators for each setting.It reports average correlation scores across aspects and models in Table 13.Source: E Correlation of human evaluation · p. 13
  • Annotator correlations are strong in the first two settings and moderate in setting 3, according to psychological assessment standards.Table 13 reports average Spearman correlations across aspects and models for memory recognition and response generation.Source: Table 13 · p. 13
  • Table 13 covers human annotation correlations for memory recognition and response generation.Source: Table 13 · p. 13

F Example of LLM results

The appendix illustrates model responses under different memory inputs, gives the setting 3 inference prompts and hyperparameters, and walks through a two-stage dialogue example.

F Example of LLM results

  • Setting 1 supplies golden memory, while Setting 3 supplies the five top retrieved memories, which may omit the golden memory; examples show replies invoking different shared events.Source: F Example of LLM results · pp. 14, 15, 17

G Prompts

  • The Setting 3 English prompt asks the model to select one suitable event from five memories or respond directly if none fits.The Chinese prompt also requests a natural, colloquial response that provides emotional support.Source: G Prompts · p. 18

H Hyperparameters

I Example of two-stage theory in Dialogue.

  • The two-stage dialogue example retrieves memories, selects the grandparents’ necklace memory, and prompts Lisa to wear it or call her grandparents.Lisa responds that she wants to wear the necklace and call them.Source: I Example of two-stage theory in… · p. 19

Related papers

Loading 2409.15240v2…