PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Assessment of Dialog Evaluation Metrics

Yi-Ting Yeh, Maxine Eskenazi, Shikib Mehri

arXiv:2106.03706v4cs.CLcs.AI

TL;DR

Dialog evaluation lacks a consistent comparison of recently proposed metrics across datasets, despite the need for automatic measures alongside costly human evaluation. This paper assesses dialog metrics across datasets and evaluation settings. Its reported findings show that the strongest metrics vary with evaluation level, dataset, and response-generation systems.

  • Problem

    Recently proposed dialog evaluation metrics had been assessed on different datasets without a consistent comparison, while human evaluation is costly and automatic metrics are needed during model development.

    Source: §6 · p. 18

  • Method

    The paper compares dialog evaluation metrics across datasets and examines their correlations, metric similarities, and combinations.

    Source: §5.3 · p. 16, §5.5 · p. 17, §6 · p. 18

  • Results

    USR, GRADE, DEB, and USL-H perform best on turn-level evaluation, while FlowScore, PredictiveEngage, DynaEval, and FED perform best on dialog-level evaluation for FED and DSTC9.

    Source: §6 · p. 18

  • Takeaways & Limitations

    Metric performance varies with training and test datasets and response-generation system type, so the best metric depends on the data being evaluated.

    Source: §6 · p. 18

    Metrics generally perform better on data they were trained on, and adapting them to new domains remains a future-work direction.

    Source: §6 · p. 19

Abstract

from arXiv · show

Automatic evaluation metrics are a crucial component of dialog systems research. Standard language evaluation metrics are known to be ineffective for evaluating dialog. As such, recent research has proposed a number of novel, dialog-specific metrics that correlate better with human judgements. Due to the fast pace of research, many of these metrics have been assessed on different datasets and there has as yet been no time for a systematic comparison between them. To this end, this paper provides a comprehensive assessment of recently proposed dialog evaluation metrics on a number of datasets. In this paper, 23 different automatic evaluation metrics are evaluated on 10 different datasets. Furthermore, the metrics are assessed in different settings, to better qualify their respective strengths and weaknesses. Metrics are assessed (1) on both the turn level and the dialog level, (2) for different dialog lengths, (3) for different dialog qualities (e.g., coherence, engaging), (4) for different types of response generation models (i.e., generative, retrieval, simple models and state-of-the-art models), (5) taking into account the similarity of different metrics and (6) exploring combinations of different metrics. This comprehensive assessment offers several takeaways pertaining to dialog evaluation metrics in general. It also suggests how to best assess evaluation metrics and indicates promising directions for future work.

1 Introduction

Standard language metrics are ineffective for dialog, while human evaluation is costly and generally reserved for final evaluation. The paper assesses recently proposed dialog metrics across datasets and settings to compare their performance and reveal their strengths and weaknesses.

1 Introduction

  • The paper addresses the lack of consistent comparisons by quantifying the relative performance of dialog metrics and identifying their strengths and weaknesses.The paper notes that the metrics had been evaluated on different datasets.Source: §1 · p. 1
  • Human evaluation is expensive and time-consuming, yet automatic metrics are needed during development to guide model design and hyperparameter choices.For these metrics to serve as meaningful proxies, they must better correlate with human judgments.Source: §1 · p. 1
  • Dialog metrics face a one-to-many response problem: many valid replies may differ from the ground-truth response, making word-overlap evaluation unsuitable.Some newer metrics instead assess a response in the context of dialog history.Source: §1 · pp. 1–2
  • The assessment compares recently proposed metrics across datasets and settings, examining turn- and dialog-level evaluation, dialog lengths and qualities, model types, metric similarities, and combinations.The datasets have human annotations measuring response quality.Source: §1 · p. 2
  • The assessment code is open-sourced to facilitate evaluation of future metrics on multiple datasets.Source: §1 · p. 2

2 Task Definition

The task is to score a generated response using its dialog context and, optionally, a human-written reference. Metric performance is assessed by correlating its scores with human quality judgments.

2 Task Definition

  • A dialog evaluation metric maps dialog context, model response, and human reference to a score; a reference-free metric uses only context and response.Source: §2 · p. 2
  • Human annotators score response quality conditioned on the dialog context, yielding the judgments used to evaluate metrics.Source: §2 · p. 2
  • Metric performance is measured by correlating its scores with the corresponding human quality annotations.Source: §2 · p. 2

3 Overview of Automatic Metrics

The paper surveys the assessed dialog metrics, their training and design choices, and the datasets behind them. It also explains reference-free evaluation and the reasons some metrics were excluded.

3 Overview of Automatic Metrics

  • The assessment distinguishes rule-based metrics, which use heuristics, from model-based metrics, which are often trained with self-supervised objectives.Source: §3.1 · p. 2, §3 · p. 2

3.1 Design of the Metrics in this Assessment

  • Assessed metrics span overlap-based and learned approaches, including BLEU, ADEM, RUBER, and dialog-specific metrics such as GRADE, USR, FED, and FlowScore.BERTScore, QuestEval, and BLEURT are also included as general-purpose NLG metrics.Source: §3.1 · pp. 2, 4, 5

3.2 Use of Pretrained Language Models

  • Pretrained language models vary in training data and objectives, and their choice may influence metric performance and generalizability.The paper identifies the impact of different pretrained models as a topic for future work.Source: §3.2 · p. 5

3.3 Training Data

  • Training-data domain, quality, and conversation setting can shape model-based metrics; the surveyed training corpora include daily-life, persona-based, knowledge-grounded, and online conversations.These datasets were generally used with self-supervised training objectives.Source: §3.3 · p. 6

3.4 Referenced vs Reference-Free

  • Reference-free metrics address dialog’s one-to-many response problem by modeling the context and generated response rather than comparing against a reference.Using multiple references could mitigate the problem, but collecting enough references to cover possible responses is infeasible.Source: §3.4 · p. 6

3.5 Metrics not Assessed in this Paper

  • Metrics were omitted when they were not dialog-specific, lacked released resources for reproduction, duplicated assessed approaches, or required annotations unavailable in the experimental setting.Some non-dialog-specific metrics were retained as baselines or to represent that category.Source: §3.1 · p. 5, §3.5 · p. 6

4 Testing Datasets

The evaluation datasets contain human quality annotations for generated responses, and their collection settings and response sources vary. Annotation coverage also differs across datasets and response qualities.

4 Testing Datasets

  • Testing datasets pair dialog context and generated response with a human quality score, sometimes including a ground-truth response; they are commonly built by generating system responses and collecting human ratings.Only reference-free metrics were assessed when a dataset lacked references.Source: §4 · pp. 6–7

4.1 Data Collection

  • Dataset characteristics vary across conversation domains, context lengths, collection settings, and response-generation systems, affecting the challenge posed to metrics.DailyDialog contexts and references are shorter than TopicalChat’s, while DSTC9 and FED include interactive human-system conversations.Source: §4.1 · p. 7

4.2 Quality Annotation

  • Most datasets annotate overall response quality, while HolisticEval-DailyDialog labels coherence and USR and FED also provide fine-grained quality annotations.Source: §4.2 · p. 8

5 Experiments

The experiments compare dialog evaluation metrics across datasets and settings, finding that metric performance varies with dataset, response quality, and the quality being assessed.

5 Experiments

  • The assessment reproduces released models where possible, trains some metrics without released models, and excludes some evaluations because of metric and data constraints.FlowScore requires more than three utterances; HolisticEval response diversity could not be calculated without the original response-generation models, and its fluency data were excluded because context and response were not clearly distinguished.Source: §5 · p. 8, §5.1 · p. 8

5.1 Results on Datasets with Human Reference

  • Rule-based metrics perform well on USR-TopicalChat, USR-PersonaChat, and DSTC6 but fall short on GRADE, where BERTScore and BLEURT perform better.Pretrained models help BERTScore and BLEURT capture semantic content in the more challenging GRADE responses.Source: §5.1 · p. 8
  • DEB achieves the highest performance on GRADE-DailyDialog and GRADE-EmpatheticDialogue, while PONE performs better on GRADE data than BERT-RUBER.The authors attribute these results to DEB's constructed adversarial-response dataset and PONE's data augmentation and weighted negative sampling.Source: §5.1 · pp. 8, 13
  • Metric performance can be domain-specific: MAUDE, USR, GRADE, and Deep AM-FM show stronger results on datasets related to their training or tuning data.The authors recommend testing across varied quality-annotated datasets to assess generality and avoid over-optimizing for particular datasets.Source: §5.1 · pp. 13, 15
  • USR performs well on GRADE, while USL-H outperforms USR on USR-PersonaChat; USL-H also incorporates a model that validates grammatical correctness.USR combines models approximating fluency, relevance, and knowledge grounding; the authors associate USL-H's results with VUP identifying poor responses.Source: §5.1 · p. 13
  • Averaging fine-grained scores from FED, HolisticEval, and PredictiveEngage correlates poorly with overall quality, although these metrics correlate with human scores on specific qualities.The authors caution against concluding that the metrics are poorly designed based only on their overall-score correlations.Source: §5.1 · p. 15

5.2 Results on Datasets without Human Reference

Reference-free metrics vary across datasets and evaluation levels: some excel at turn-level judgments, while others better assess whole-dialog quality. Performance also depends on dialog length and the response-generation systems being evaluated.

FED

  • On FED turn-level annotations, DynaEval, USL-H, and DEB perform best, with DynaEval considerably ahead of other metrics.DynaEval models dependencies between utterance pairs with a graph-based dialog representation.Source: §5.2 · p. 15

Dialog length

  • FED contexts average 86.5 words and 11.8 utterances, compared with 23.4 words for GRADE-ConvAI2; metrics not trained on long contexts may struggle at test time.The paper suggests longer contexts may help explain why metrics that perform well on USR and GRADE do less well on FED.Source: §5.2 · p. 15

FED

  • On FED dialog-level annotations, FED and DynaEval perform best; DynaEval has stronger correlation than the other metrics.PredictiveEngage also exceeds other metrics apart from FED and DynaEval; HolisticEval’s dialog-level correlation is much lower than its turn-level correlation.Source: §5.2 · p. 15

DSTC9

  • PredictiveEngage, DynaEval, FED, USL-H, and FlowScore have the highest dialog-level correlations on DSTC9.DSTC9 contains interactive human-system dialogs from state-of-the-art systems and has longer contexts, making it challenging for evaluation.Source: §5.2 · p. 15

Results across datasets

  • DynaEval, FED, and PredictiveEngage correlate strongly with dialog-level judgments on FED and DSTC9, but underperform on USR and GRADE.The authors suggest that dialog-level focus and, for FED and PredictiveEngage, simpler architectures may contribute to this pattern.Source: §5.2 · p. 15

Cross-dataset performance

  • FlowScore performs strongest on DSTC9 but underperforms on other quality-annotated datasets, including FED, suggesting strength on long, complex dialogs.Only reference-free metrics can be evaluated on the quality-annotated datasets without reference responses.Source: §5.2 · pp. 15–16

5.3 Results on System-Level Correlation

System-level correlation indicates how well a metric ranks response-generation systems. Turn-level and system-level performance often align, but some metrics differ across these evaluation levels.

Overall pattern

  • Metrics with higher turn-level correlation generally have higher system-level correlation, but some turn-level underperformers do better when scores are averaged across a system’s examples.On DSTC6, GRADE has system-level correlation comparable to the best-performing Deep AM-FM despite lower turn-level correlation.Source: §5.3 · p. 16

DSTC6

  • On DSTC6, BERTScore has the highest turn-level correlation but performs poorly on system-level correlation.The authors suggest word-overlap metrics may struggle to assess entire systems accurately.Source: §5.3 · p. 16

Many-system datasets

  • Deep AM-FM attains high system-level correlation on DSTC6, while FlowScore does so on DSTC9; both datasets contain many systems.DSTC6 has 11 systems and DSTC9 has 20, making high system-level correlation difficult to attain.Source: §5.3 · p. 16

5.4 Performance on Various Dialog Qualities

Metric performance differs across dialog qualities and datasets. Coherence labels may not be comparable across datasets, and the findings favor metrics that combine multiple quality signals for an overall score.

Evaluation setup

  • Because dialog quality is multi-faceted, the paper compares metric correlations with different qualities, using corresponding fine-grained scores when available.FED includes eight fine-grained turn-level qualities and ten fine-grained dialog-level qualities.Source: §5.4 · p. 16

Quality-specific performance

  • HolisticEval performs best on HolisticEval-DailyDialog, while USL-H performs best on the specific qualities in FED.HolisticEval performs poorly at measuring coherence on FED; possible explanations include its language-modeling objective and differing dataset interpretations of coherence.Source: §5.4 · p. 16

Coherence annotations

  • HolisticEval-DailyDialog does not provide precise annotation instructions, so its coherence definition cannot be compared with FED’s.GRADE and USR correlate highly with coherence on HolisticEval-DailyDialog but perform relatively poorly on FED’s dialog-level coherent annotation.Source: §5.4 · p. 16

Multiple qualities

  • USR and USL-H do well on coherence and engaging, respectively, while HolisticEval and PredictiveEngage do well on those qualities but underperform on USR and GRADE.The authors conclude that modeling only fine-grained qualities is insufficient and favor aggregating multiple qualities into an overall score, as in USR and USL-H.Source: §5.4 · pp. 16–17

5.5 Metric Output Similarities

Metric outputs cluster in ways associated with shared scoring approaches, training data, and pretrained models. Deep AM-FM’s similarity to word-overlap metrics is notable despite its dialog-evaluation purpose.

Metric clusters

  • Deep AM-FM correlates strongly with BERTScore, BLEURT, and rule-based metrics, possibly because its reference-response comparison favors generated responses with high word overlap.The authors note this is surprising because Deep AM-FM was designed for dialog evaluation, unlike the general-purpose NLG metrics.Source: §5.5 · p. 17
  • BERT-RUBER, PONE, GRADE, and USL-H behave similarly; shared DailyDialog training and BERT use may help explain the pattern.BERT-RUBER, PONE, and GRADE share a triplet rank loss, while USR has slightly higher correlation with this group.Source: §5.5 · p. 17

5.6 Effect of Context Length

Most metrics lose performance as dialog contexts lengthen, although several remain stable and HolisticEval, DialogRPT, and FED improve. The authors relate better long-context performance to GPT-2-based scoring and identify different sources of robustness in USL-H, DEB, and PredictiveEngage.

5.6 Effect of Context Length

  • HolisticEval, DialogRPT, and FED improve as context lengthens, while USL-H, DEB, and PredictiveEngage change little.Most other metrics decline with longer contexts; the authors suggest some metrics may struggle because they were trained on shorter dialogs.Source: §5.6 · p. 17
  • GPT-2-based HolisticEval, DialogRPT, and FED improve on longer dialogs, whereas most other metrics rely on BERT-based models.The authors associate this pattern with BERT’s local-coherence training and context-length limits.Source: §5.6 · p. 17
  • USL-H, DEB, and PredictiveEngage remain comparatively stable across context lengths, with different proposed explanations for each metric.The authors point to PredictiveEngage’s MLP pooling, USL-H’s VUP model, and DEB’s Reddit-pretrained BERT.Source: §5.6 · p. 17

5.7 Performance on Different Response Generation Models

Across the assessed datasets, averaging all metric scores yields surprisingly good results, while other combinations help some datasets and hurt others. Metric performance also varies by response-generation model: USR and GRADE especially score retrieval responses well, while USR performs less well on generative models.

5.7 Performance on Different Response Generation Models

  • USR and GRADE especially score retrieval responses well; on generative responses, USR performs significantly less well while GRADE achieves the highest correlation.Most metrics perform similarly across retrieval and generative models, but these metrics show notable differences.Source: §5.7 · p. 17

5.8 Combining Metrics

  • The explored metric combinations use simple averaging, and the authors leave more sophisticated combination mechanisms for future work.The combinations were selected from some of the best-performing metrics rather than exhaustively enumerated.Source: §5.8 · p. 17
  • Combining PredictiveEngage, RUBER, and PONE significantly improves results on USR-TopicalChat and USR-Personachat, but lowers correlation on other datasets.The combination was proposed in (Ghazarian et al., 2020).Source: §5.8 · pp. 17–18
  • Combining USR with GRADE, USL-H, PONE, and PredictiveEngage helps on USR and GRADE-ConvAI2, while combinations involving GRADE often hurt except with DEB.Combining DEB with USR and USL-H also hurts; GRADE and DEB outputs are highly correlated, which the authors suggest may help indicate which metrics to combine.Source: §5.8 · p. 18
  • A simple average of all assessed metric scores performs surprisingly well across the referenced quality-annotated datasets.The authors identify this result as evidence of potential in combining metrics, while calling for smarter combination strategies.Source: §5.8 · p. 18

6 Conclusions

This paper systematically assesses dialog evaluation metrics across datasets and settings, finding that performance depends on evaluation level, dataset, response-generation model, and context length. The results favor strong but setting-dependent metrics and suggest promise in combining metrics, alongside further study of pretrained models and domain adaptation.

6 Conclusions

  • USR, GRADE, DEB, and USL-H perform best for turn-level response-generation evaluation, with consistently higher performance on most datasets.Source: §6 · p. 18
  • FlowScore, PredictiveEngage, DynaEval, and FED perform best for dialog-level evaluation on FED and DSTC9; system-level correlations are also strong across metrics.Deep AM-FM has high correlation on DSTC6 with 11 systems, and FlowScore on DSTC9 with 20 systems.Source: §6 · p. 18
  • Metric performance varies with dataset and response-generation model, so the best metric depends on the data and system type being evaluated.Metrics generally perform best on datasets on which they were originally evaluated; results also differ between datasets with simple and more sophisticated models.Source: §6 · p. 18
  • Many metrics struggle with longer contexts, especially BERT-based metrics, whereas metrics using GPT-2-based models perform better.Source: §6 · p. 18
  • Evaluating metrics on fine-grained dialog qualities reveals information about their underlying behavior.Source: §6 · p. 18
  • Combining different models and metrics yields strong results, making more sophisticated metric-combination strategies a promising direction.The paper suggests selecting a metric suited to a particular response as one possible mechanism.Source: §6 · pp. 18–19
  • Because metrics generally perform best on their training data, the authors recommend exploring self-supervised in-domain fine-tuning to adapt them to new domains.Many metrics are trained on specific dialog datasets.Source: §6 · pp. 18–19
  • The assessment code is open-sourced to support efficient evaluation of future dialog metrics across quality-annotated datasets.Source: §6 · p. 19
Loading 2106.03706v4…