Source-linked AI summary
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation
Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, Joelle Pineau
TL;DR
The paper asks whether automatic metrics can evaluate unsupervised dialogue responses without supervised labels. It compares word-overlap and embedding-based metrics with human judgements and finds weak correlation on Twitter and none on Ubuntu, motivating better metrics.
Problem
Unsupervised dialogue systems lack supervised labels for evaluation, while adopted metrics assume a single target response adequately represents valid dialogue responses.
Method
The paper compares word-overlap and word-embedding metrics with human judgements across response-generation models and dialogue datasets.
Results
The metrics correlate only weakly with human judgements on Twitter and not at all on Ubuntu, despite embedding metrics distinguishing several model types.
Takeaways & Limitations
The findings support shifting away from commonly used metrics toward evaluation measures that correlate more strongly with human judgement.
Takeaways & Limitations
The analysis assumes only one ground-truth response per context, although multiple plausible responses may exist.
Abstract
from arXiv · showhide
We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available. Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response. We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain. We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.
1 Introduction
The paper examines how to evaluate unsupervised dialogue responses when supervised labels are unavailable. It tests commonly adopted automatic metrics and finds weak alignment with human judgements, motivating better alternatives.
- Motivation: Unsupervised dialogue models avoid large-scale supervised labels, but their automatic evaluation remains an open question.Such metrics could accelerate deployment of unsupervised response-generation systems.
- Motivation: Metrics borrowed from machine translation and summarization assume valid responses share substantial word overlap with a single ground-truth response.Dialogue permits diverse valid responses, including reasonable responses with no shared words.
- Approach: The study compares word-overlap metrics, including BLEU, METEOR, and ROUGE, with word-embedding metrics against human judgements.It evaluates several response-generation models across dialogue domains.
- Findings: All examined metrics show weak or no correlation with human judgements: small positive correlation on Twitter and none on Ubuntu.Embedding metrics can distinguish baseline from state-of-the-art models while still failing to track human ratings.
- Implications: The results call for moving away from existing metrics toward a metric that correlates more strongly with human judgement.The paper combines statistical, qualitative, and sensitivity analyses to expose metric shortcomings.
2 Related Work
Prior dialogue research adopted evaluation practices from related language-generation tasks, but dialogue poses a harder evaluation problem because each context admits many correct responses.
- Scope: This paper excludes perplexity, retrieval-based recall, and supervised evaluation methods because its focus is model-independent comparison against ground-truth responses.Perplexity is not computed per response and cannot evaluate retrieval models in the paper’s setting.
- Related Work: Dialogue evaluation studies commonly use BLEU and related automatic metrics adopted from machine translation and other language-generation tasks.Earlier work also compared dialogue models using BLEU alongside human evaluations.
- Related Work: Dialogue is harder to evaluate automatically than translation or surface realization because the set of correct responses is larger.The paper characterizes dialogue response generation as having higher response diversity, or entropy.
3 Evaluation Metrics
The paper evaluates two families of metrics against a single ground-truth response: surface word overlap and semantic similarity from word embeddings.
- Word Overlap-based Metrics: Word-overlap metrics compare proposed and ground-truth responses using BLEU, METEOR, or ROUGE.These metrics were originally developed for machine translation or summarization and had not been thoroughly investigated for dialogue.
- Word Overlap-based Metrics: BLEU combines modified n-gram precision with a brevity penalty, commonly using up to 4-grams.The metric is usually computed at corpus level and can require smoothing when 4-gram overlap is absent.
- Word Overlap-based Metrics: METEOR aligns candidate and target tokens through exact matches, synonyms, stems, and paraphrases, then computes a precision–recall harmonic mean.Its alignment procedure addresses several weaknesses attributed to BLEU.
- Word Overlap-based Metrics: ROUGE-L measures an F-score based on the longest common subsequence, allowing matched words to be noncontiguous.The words must remain in the same order.
- Embedding-based Metrics: Embedding-based metrics represent words as vectors and compare responses through semantic similarity rather than direct word overlap.The paper studies greedy matching, embedding averages, and vector extrema.
- Embedding-based Metrics: Greedy matching averages cosine similarities from token-level matches, while embedding average and vector extrema construct sentence-level representations.Greedy matching is asymmetric and therefore requires averaging scores in both directions.
4 Dialogue Response Generation Models
The paper evaluates retrieval and generative dialogue models with vector-based metrics across technical Ubuntu and casual Twitter domains. These comparisons show apparent model differences, but embedding metrics correlate weakly or not at all with human judgements.
- Dialogue Response Generation Models: The evaluation covers retrieval and generative models, including TF-IDF, Dual Encoder, LSTM, and HRED systems.Retrieval models include TF-IDF and Dual Encoder architectures, while generative models include an LSTM baseline and HRED.
- Retrieval Models: TF-IDF retrieves responses using cosine similarity between context-response representations or context-context representations.The C-TFIDF variant compares the input context with corpus contexts, whereas R-TFIDF compares it with corpus responses.
- Retrieval Models: The Dual Encoder estimates response correctness from context and response vectors using a learned weighted dot product.It uses two RNNs to compute representations and trains with negative sampling to minimize cross-entropy error.
- Evaluation Domains: The study compares models on Ubuntu's technical problem-solving conversations and Twitter's non-technical casual chit-chat.These datasets provide contrasting dialogue domains for testing whether metrics behave consistently across settings.
- Evaluation Results: Embedding metrics rank Dual Encoder above both TF-IDF baselines and HRED above the basic LSTM in both domains, but correlate weakly on Twitter and not at all on Ubuntu.The apparent model distinctions therefore do not establish alignment with human judgements.
5 Human Correlation Analysis
The study compares automatic dialogue-response metrics with human judgements using surveys across Twitter and Ubuntu, finding weak metric–human relationships and diagnosing qualitative failure modes.
- Human evaluation: 25 volunteers rated response quality from 1 to 5.5 for contexts and proposed responses from each dataset.The survey included 20 unique contexts per dataset and five responses per context, including model-generated and human-written responses.
- Correlation analysis: Pearson correlation estimates linear association, whereas Spearman correlation estimates any monotonic association between metric scores and human judgements.
- Quantitative results: BLEU-4 showed very weak if any correlation with human judgement in both domains, while BLEU-3 and BLEU-4 scores were near zero for most response pairs.Only four examples had BLEU-4 scores above 10^-9; smoothing made higher-order BLEU behave like scaled, noisy BLEU-2.
- Quantitative results: Small positive correlations appeared for some metrics on Twitter, but no metric significantly correlated with human judgements on Ubuntu.Figure 1 presents BLEU-2, embedding-average, and inter-rater correlations; the metric plots appear close to random noise.
- Metric sensitivities: BLEU and METEOR were particularly sensitive to large response-length differences, unlike Embedding Average and human judgement.
- Qualitative analysis: Qualitative disagreements showed metrics scoring reasonable responses lowly when wording differed, because embedding metrics could not separate salient words from other words.The analysis suggests weighting word saliency could improve embedding-based metrics.
- Qualitative analysis: Metrics also scored some responses highly when humans did not, likely because frequent tokens and nearby embeddings obscured context-specific semantic differences.Both Figure 2 examples had no overlapping n-grams above unigrams, causing BLEU-2, BLEU-3, and BLEU-4 to assign near-zero scores.
6 Discussion
The discussion identifies scope conditions and open problems for evaluating unsupervised dialogue systems, including constrained tasks, multiple responses, sentence-level metrics, and context-aware evaluation.
- Constrained tasks: BLEU may correlate better with human judgements in constrained tasks, but this possibility still requires empirical investigation.The authors contrast unconstrained dialogue with mapping dialogue acts to natural-language sentences, which has lower diversity.
- Incorporating multiple responses: The analysis assumes one ground-truth response per context, while no standard method exists for incorporating multiple plausible responses.Future work should examine whether retrieving additional responses changes correlations with word-overlap metrics.
- Searching for suitable metrics: The paper provides evidence against existing metrics but does not yet provide good alternatives for unsupervised evaluation.The authors view distributed sentence representations as promising because basic word-vector averages inadequately model sentence-level compositionality.
- Context-aware evaluation: Existing metrics compare proposed responses directly with ground truth without considering conversational context.The authors discuss learned discriminative or human-like scoring models, while noting that learning them may be no easier than generating responses.
Distribution of kappa scores
Table 6 reports the distribution of pairwise κ scores between retained human annotators.
- Distribution of kappa scores: Table 6 summarizes pairwise κ scores for each pair of human annotators after excluding annotators discarded for low scores.The supplied caption identifies the table’s population and exclusion criterion but does not report individual values.
Full scatter plots
The full scatter plots compare metric scores or human-judgement agreement across Twitter and Ubuntu. They show very little correlation for the evaluated metrics, with BLEU-3 and BLEU-4 often near zero.
- Full scatter plots: Very little correlation appears for the evaluated metrics, while BLEU-3 and BLEU-4 scores are often close to zero.The scatterplots cover the metrics shown in Figures 3–7.
- Human agreement: Figure 3 compares correlations between two randomly chosen groups of human volunteers on Twitter and Ubuntu.The Twitter corpus is shown on the left and the Ubuntu Dialogue Corpus on the right.
- Word-overlap metrics: Figure 4 plots ROUGE and METEOR against human judgement for Twitter and Ubuntu.ROUGE is panel (a), and METEOR is panel (b).
- Embedding metrics: Figure 5 plots vector extrema, greedy matching, and vector averaging against human judgement across the two corpora.The panels correspond to vector extrema (a), greedy matching (b), and vector averaging (c).
- BLEU metrics: Figure 6 plots BLEU-1 through BLEU-4 against human judgement across Twitter and Ubuntu.The four panels correspond respectively to BLEU-1, BLEU-2, BLEU-3, and BLEU-4.