Source-linked AI summary
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Shikib Mehri, Maxine Eskenazi
TL;DR
Meaningful automatic metrics for open-domain dialog remain lacking because standard language-generation metrics are ineffective. USR addresses this with an unsupervised, reference-free metric whose interpretable sub-metrics strongly correlate with human judgment on Topical-Chat and PersonaChat.
Problem
Standard language-generation metrics are ineffective for dialog evaluation, while human evaluation is costly and typically reserved for final models.
Method
USR trains unsupervised models to measure desirable dialog qualities using configurable, interpretable sub-metrics without reference responses.
Results
USR strongly correlates with human judgment on Topical-Chat and PersonaChat at both turn and system levels.
Takeaways & Limitations
USR provides interpretable dialog-quality measures and can be adapted across tasks and datasets.
Takeaways & Limitations
USR should be used alongside human evaluation and not to claim superior performance over another method.
Abstract
from arXiv · showhide
The lack of meaningful automatic evaluation metrics for dialog has impeded open-domain dialog research. Standard language generation metrics have been shown to be ineffective for evaluating dialog models. To this end, this paper presents USR, an UnSupervised and Reference-free evaluation metric for dialog. USR is a reference-free metric that trains unsupervised models to measure several desirable qualities of dialog. USR is shown to strongly correlate with human judgment on both Topical-Chat (turn-level: 0.42, system-level: 1.0) and PersonaChat (turn-level: 0.48 and system-level: 1.0). USR additionally produces interpretable measures for several desirable properties of dialog.
1 Introduction
USR addresses the limitations of existing dialog metrics with an unsupervised, reference-free metric composed of interpretable, configurable sub-metrics. It correlates strongly with human judgments across Topical-Chat and PersonaChat and supports future benchmarking through released human annotations.
- Motivation and contribution: Standard metrics struggle with dialog because valid responses can differ from a single reference and dialog quality is multi-faceted and task-dependent.The paper identifies one-to-many responses, uninterpretable single scores, and differing desired qualities as key shortcomings.
- Motivation and contribution: USR measures desirable dialog qualities with unsupervised models instead of relying on a ground-truth reference response.Its sub-metrics are combined configurably to produce an overall quality score.
- Evaluation: 0.42 and 0.48 turn-level Spearman correlations were obtained on Topical-Chat and PersonaChat, respectively.System-level Spearman correlation was 1.0 on both datasets.
- Evaluation: USR's system-level Spearman correlation with human judgment was 1.0 on both Topical-Chat and PersonaChat.The evaluation used human quality annotations for models trained on both corpora and included a variety of model types.
- Evaluation: The paper releases human quality annotations to facilitate future benchmarking of dialog evaluation metrics.The contribution is presented alongside the proposed unsupervised, reference-free metric.
2 Related Work
Related work includes reference-based, multi-reference, supervised, and reference-free approaches to dialog evaluation. USR differs by requiring no explicit supervision while evaluating correlation with human judgment.
- Existing evaluation metrics: Existing automatic metrics correlate poorly with human judgments of dialog, and F-1 can be gamed by outputting frequent words regardless of context.The cited example illustrates a mismatch between metric optimization and contextual dialog quality.
- Reference-based approaches: Multi-reference metrics address dialog's one-to-many nature, but thoroughly covering the space of valid responses is infeasible.Multiple references can be obtained through retrieval models or additional data collection.
- Supervised evaluators: ADEM and the Alexa Prize evaluation framework use explicit quality annotations or user ratings to assess dialog quality.The passage contrasts these supervised approaches with USR's lack of explicit supervision.
- USR's distinction: USR requires no explicit supervision and is described as more readily generalizable to new datasets and tasks than supervised evaluators.This distinction is presented as a contrast with approaches trained on explicit quality annotations.
- Reference-free approaches: A prior reference-free discriminator evaluated human versus generated responses but did not evaluate correlation with human judgment.The paper notes that relying solely on a discriminator may be insufficient for measuring dialog quality.
3 Human Quality Annotation
The study collects human quality annotations across Topical-Chat and PersonaChat using multiple response types and detailed instructions, then analyzes agreement and quality patterns. Results show moderate-to-strong agreement, annotator-specific quality priorities, and a strong distinction between newly written human responses and original ground-truth responses.
- Annotation setup: Human annotations covered 60 contexts per dataset, with six responses per Topical-Chat context and five per PersonaChat context.Responses included system outputs, a newly written human response, and an original ground-truth response; each response received six quality scores.
- Annotation setup: Detailed instructions and a preliminary annotation round were used to reduce subjectivity and refine the rating procedure.The instructions were adjusted after preliminary annotations and discussion, including changing Maintains Context from a two-point to a three-point scale.
- Agreement: Most inter-annotator correlations exceeded 0.4, while Overall Quality agreement reached 0.71 for Topical-Chat and 0.66 for PersonaChat.Understandable agreement was lower on PersonaChat partly because annotators differed in their background knowledge.
- Quality patterns: Across both datasets and all six qualities, newly written human responses strongly outperformed other response types, including original ground-truth responses.The newly written responses were produced with quality annotation in mind, whereas original ground-truth responses emphasized overall-dialog quality.
- Quality patterns: Annotators weighted qualities differently, but generally expected good responses to be natural, maintain context, and be interesting.For example, one annotator emphasized naturalness while another emphasized grounding in knowledge.
- Quality patterns: A regression predicting Overall Quality from the specific quality ratings achieved a 0.9654 Spearman correlation with the original overall scores.This result indicates that the selected qualities contained enough information to reflect the overall response quality.
4 Automatic Metrics
Existing dialog metrics rely heavily on reference overlap, while USR uses unsupervised language and retrieval models to assess interpretable dialog qualities without a reference response. Its configurable sub-metrics include masked language modelling and dialog retrieval measures.
- 4.1 Baseline Metrics: Vector Extrema computes cosine similarity between sentence embeddings formed by taking each embedding dimension’s maximum word value, and has outperformed other metrics on dialog tasks.The maximum operation is motivated by de-emphasizing common words that lie closer to the embedding origin.
- 4.1 Baseline Metrics: BERTScore greedily matches reference and generated words with pre-trained BERT embeddings, but remains ineffective for dialog’s one-to-many output space.Its reference-based word-similarity design may suit tasks with limited possible outputs better than open-domain dialog.
- 4.2 USR: USR is designed as an unsupervised, reference-free metric because dialog’s one-to-many nature means there is no single correct response.It uses pre-trained RoBERTa models and several sub-metrics to measure qualities such as naturalness, interestingness, and knowledge use.
- 4.2 USR: The MLM sub-metric masks each response word in a context-response sequence and uses fine-tuned RoBERTa likelihoods to assess understandability and naturalness.Each response word is masked in turn, and the metric aggregates its masked log likelihoods.
- 4.2 USR: Dialog retrieval sub-metrics are trained without additional quality labels to distinguish true responses from randomly sampled ones, using context and optionally facts.They are motivated particularly for evaluating properties such as maintaining context and using knowledge.
- 4.2 USR: USR combines its sub-metrics through configurable regression weights, allowing overall quality scores to prioritize different properties or datasets.The paper reports that individuals used different weights when producing overall scores, suggesting personalization by adjusting the regression model.
5 Results
USR strongly correlates with human judgments across Topical-Chat and PersonaChat, outperforming standard automatic metrics at turn and system levels. Its performance reflects combining reference-free sub-metrics that capture multiple dialog qualities.
- USR strongly outperforms word-overlap and embedding-based metrics across dialog qualities on Topical-Chat.
- USR strongly outperforms word-overlap and embedding-based metrics across all dialog qualities on PersonaChat.PersonaChat’s dependence on dialog context and persona helps explain stronger correlations for Maintains Context and Uses Knowledge.
- USR performs strongly on Overall Quality ratings because MLM and DR quantify response qualities without references and are combined into a correlated overall metric.
- DR (x = c) outperforms USR on PersonaChat Overall Quality, likely because it explicitly leverages both dialog context and persona.
- 1.0 Spearman correlation: USR significantly outperforms all other metrics at system level on both datasets.Pearson correlations are 0.92 for Topical-Chat and 0.82 for PersonaChat.
- USR’s effectiveness holds across two datasets and is significantly different from the F-1 score used for Topical-Chat evaluation.
6 Discussion
USR correlates significantly with human judgment across Topical-Chat and PersonaChat and can be configured for different evaluation needs. The authors recommend using it alongside human evaluation and identify scope limitations.
- USR achieves statistically significant correlations with human judgment on both Topical-Chat and PersonaChat.
- USR combines dialog-quality sub-metrics through an adjustable regression, allowing qualities to be removed or re-weighted for other tasks, datasets, or users.
- USR should support model selection and hyperparameter tuning alongside human evaluation, not claims of superior performance over another method.
- USR may not work with non-generative models, which were not addressed in this study.
7 Conclusions
The paper concludes that USR is a configurable, unsupervised, reference-free dialog metric composed of interpretable quality-specific sub-metrics. It strongly correlates with human judgment on Topical-Chat and PersonaChat.
- USR is a reference-free metric composed of multiple sub-metrics that evaluate specific dialog qualities.
- USR’s definition of good dialog is configurable, allowing adaptation to different tasks and datasets.
- USR strongly correlates with human judgment on Topical-Chat and PersonaChat.The conclusion reports turn-level correlations of 0.42 for Topical-Chat and 0.48 for PersonaChat, with system-level correlations of 1.0 for both.
A Annotation Instructions
The appendix provides the annotation instructions and examples shown verbatim to human annotators.
- Tables 6, 7, and 8 reproduce the annotation instructions and examples shown to annotators.
B Metric Evaluation
Table 3 reports turn-level correlations for each specific dialog quality, showing only the best-correlated metrics because of space limitations. Full results appear in Tables 9–21.
- Table 3 reports turn-level correlations for each specific quality using only the best-correlated metrics.The complete results are provided in Tables 9–21.
C Code and Data Release
The paper releases metric code and human quality annotations, enabling benchmarking of additional metrics. Supplementary tables provide annotation instructions and correlations across multiple qualities and datasets, with significance indicated by italicization.
- Code and data release: Metric code is available at github.com/shikib/usr.
- Code and data release: Human quality annotations are released to support benchmarking of additional metrics.
- Annotation instructions: Tables 7–9 provide the three parts of the annotation instructions.
- Correlation tables: USR improves system-level Understandable and Natural correlations over USR-MLM by leveraging predictions from other sub-metrics.
- Correlation tables: USR strongly outperforms other metrics on turn-level Maintains Context correlations in both Topical-Chat and PersonaChat.Several referenced metrics perform strongly on system-level correlations.