Source-linked AI summary
Unsupervised Evaluation of Interactive Dialog with DialoGPT
Shikib Mehri, Maxine Eskenazi
TL;DR
Dialog evaluation needs meaningful measures because standard metrics correlate poorly with human judgments and reference responses do not capture interactive, one-to-many conversations. This paper introduces FED, using DialoGPT and quality-specific follow-up likelihoods to evaluate eighteen qualities without supervision or references. FED achieves moderate to strong correlation with human judgment at both turn and dialog levels, while its generalization beyond open-domain chit-chat remains unresolved.
Problem
Standard dialog metrics correlate poorly with human judgment, and reference-based evaluation is unsuitable for interactive conversations with multiple valid responses.
Method
FED uses DialoGPT to score likelihoods of follow-up utterances associated with eighteen dialog qualities, without training data or reference responses.
Results
Moderate to strong correlation with human judgment is obtained for both turn-level and dialog-level evaluation.
Takeaways & Limitations
Pretrained language models can be leveraged to extract fine-grained dialog quality information without supervision.
Takeaways & Limitations
The FED dataset contains only open-domain chit-chat, so whether FED generalizes to goal-oriented dialog requires future work.
Abstract
from arXiv · showhide
It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research. Standard language generation metrics have been shown to be ineffective for dialog. This paper introduces the FED metric (fine-grained evaluation of dialog), an automatic evaluation metric which uses DialoGPT, without any fine-tuning or supervision. It also introduces the FED dataset which is constructed by annotating a set of human-system and human-human conversations with eighteen fine-grained dialog qualities. The FED metric (1) does not rely on a ground-truth response, (2) does not require training data and (3) measures fine-grained dialog qualities at both the turn and whole dialog levels. FED attains moderate to strong correlation with human judgement at both levels.
1 Introduction
Dialog evaluation needs meaningful, interpretable measures because standard reference-based metrics poorly capture interactive, multi-faceted conversations. The paper introduces FED, which uses DialoGPT and follow-up likelihoods to evaluate eighteen dialog qualities without references or training data.
- Motivation: Standard automatic metrics are ineffective for dialog, while human evaluation is costly and usually reserved for final assessment.Development may therefore rely on poorly correlated automatic metrics.
- Motivation: Interactive dialog evaluation requires reference-free measures because valid responses can differ from a single ground-truth response.The paper identifies dialog’s one-to-many nature and interactive setting as central challenges.
- Method: FED uses DialoGPT’s likelihoods for quality-specific follow-up utterances to assess dialog without training data or reference responses.The method treats DialoGPT as an implicit model of dialog quality.
- Results: Moderate to strong correlation with human judgment is obtained for both turn-level and dialog-level evaluation.The result is achieved without training data or a ground-truth response.
- Contributions: The FED dataset contains annotations for eighteen fine-grained dialog qualities at both turn and dialog levels.It supports benchmarking automatic metrics against human judgment and analyzing which qualities matter to annotators.
2 Related Work
Related work shows that reference-based metrics correlate poorly with human judgments and that dialog quality is multi-faceted. FED differs from prior approaches by combining reference-free evaluation with unsupervised assessment of eighteen qualities.
- Reference-based evaluation: Word-overlap metrics perform poorly because dialog’s one-to-many nature permits valid responses that differ from the ground truth.Several prior approaches address this issue by using multiple reference responses.
- Prior automatic metrics: Some existing automatic evaluators are trained on explicit quality annotations, unlike FED, which requires no supervision.ADEM and the Alexa Prize evaluation framework are examples of supervised approaches.
- Prior automatic metrics: USR is unsupervised and reference-free but assesses five qualities with hand-designed models and tasks, whereas FED covers eighteen qualities.The comparison positions FED as more general in the number of qualities it encapsulates.
- Multi-faceted evaluation: Human evaluation is often reduced to overall quality or appropriateness despite dialog quality being multi-faceted.Prior frameworks have measured properties such as interestingness, fluency, relevance, empathy, and specificity.
- Pretrained dialog models: DialoGPT was pretrained on Reddit and is used as the pretrained model underlying FED.The model had previously achieved human-level performance on response generation.
3 Data Collection
The FED dataset annotates interactive human-system and human-human conversations for fine-grained dialog quality at turn and dialog levels. Its annotations are designed for evaluating automatic metrics against human judgments rather than for training models.
- Conversation collection: The dataset contains Human-Meena, Human-Mitsuku, and Human-Human conversations collected for correlation-based evaluation of automatic metrics.Human-human dialogs used one participant in the system role.
- Annotation procedure: AMT workers annotated 40 Human-Meena, 44 Human-Mitsuku, and 40 Human-Human conversations, with five workers per conversation.Selected system responses were annotated sequentially at the turn level before whole-dialog annotation.
- Annotation scheme: Eighteen fine-grained dialog qualities are measured: eight at the turn level and ten at the dialog level.Turn-level judgments assess system responses in context, while dialog-level judgments cover quality across an entire conversation.
- Annotation scheme: Turn-level annotation uses eight fine-grained measures plus overall quality, with No, Somewhat, Yes, and N/A options.Overall impression is recorded on a five-point Likert scale.
- Annotation scheme: Dialog-level annotation covers ten fine-grained qualities plus overall impression across the entire conversation.The Somewhat option is omitted for the binary inconsistency judgment.
- Dataset scale: The dataset includes 3348 turn-level and 1364 dialog-level data points, totaling 4712.It is intended solely for evaluating metrics because the annotated conversations are insufficient for both training and testing.
4 Data Analysis
The FED dataset supports fine-grained analysis of annotator agreement, system performance, and the qualities contributing to overall dialog impressions at turn and dialog levels.
- Inter-Annotator Agreement: Inter-annotator agreement was high across dialog qualities, although understandable and consistent had lower correlations in the 0.5–0.6 range.These two qualities lacked a Somewhat response option, which probably contributed to the lower agreement.
- System Performance: The FED dataset expands interactive system comparison from two qualities to eighteen fine-grained dialog qualities.This provides more information about the strengths and weaknesses of Mitsuku, Meena, and human dialogs.
- System Performance: For all turn-level qualities, Meena outperforms both Mitsuku and Human, with its strongest advantages in interesting, engaging, and specific.The fine-grained system scores are reported in Table 4.
- System Performance: Humans significantly outperform both systems on dialog-level qualities, while Meena exceeds Mitsuku by 1 point in overall score.Meena is fairly coherent, understanding, and flexible, but struggles with diversity, topic depth, and likeability.
- System Performance: Turn-level qualities are insufficient alone because a sub-optimal response can sometimes produce a better long-term dialog.Dialog-level evaluation captures the multi-turn nature of interaction and shows stronger human performance.
- Fine-Grained Quality Analysis: Interesting, relevant, and fluent are the most important turn-level qualities, while coherent, likeable, and understanding are most important at dialog level.These importance estimates come from regression weights predicting annotators’ overall impressions.
5 Methods
FED evaluates dialog quality without reference responses, training data, or supervision by using DialoGPT and likelihoods of positive and negative follow-up utterances across eighteen qualities.
- FED Metric: FED measures eighteen fine-grained dialog qualities without reference responses or training data.The metric is designed for automatic evaluation without supervision.
- Motivation: The method hypothesizes that DialoGPT implicitly captures dialog quality through large-scale pretraining on conversational data.The paper connects this hypothesis to prior findings that pretrained language models capture other forms of knowledge.
- Follow-Up Utterance for Evaluation: FED uses follow-up utterances as feedback about a system response in an interactive conversation.A user’s subsequent response can provide implicit or explicit information about the quality of the preceding system utterance.
- Evaluating with DialoGPT: For each quality, FED compares DialoGPT likelihoods for hand-written positive and negative follow-up utterances conditioned on the dialog context and system response.The follow-up sets contain several utterances per quality and were minimally tuned on ten conversations.
- Evaluating with DialoGPT: Dialog-level quality scores are computed by removing the system response from the scoring equation.Turn-level scoring conditions on context plus response, whereas dialog-level scoring uses the context alone.
- Evaluating with DialoGPT: Positive and negative follow-up likelihoods can be unevenly informative because negative utterances are generally more meaningful than positive acknowledgments.The number of positive utterances ranges from 0 to 4, while negative utterances range from 1 to 4.
6 Results
FED correlates with human judgement at both turn and dialog levels without training data or reference responses, but performance varies across dialog qualities and settings.
- Experimental Setup: Table 6 reports correlations for four pre-trained DialoGPT variations, with turn-level metrics generally using the last system response and selected metrics using the entire conversation.The evaluated models vary by size and initialization, including 345M and 762M models fine-tuned from GPT-2 or trained from scratch.
- Correlation with Human Judgement: 0.209 is the best overall turn-level correlation, while 0.443 is the best overall dialog-level correlation with human judgement.These results are reported across four DialoGPT model variations using Spearman correlation.
- Correlation with Human Judgement: FED performs competitively with prior dialog metrics despite using neither training data nor ground-truth responses.Prior reported correlations range from 0.10–0.27 for multi-reference evaluation and 0.28–0.42 for ADEM, but direct comparison is unavailable.
- Discussion: FED works better for qualities such as interesting and engaging than for understandable and consistent because DialoGPT was trained on Reddit.Reddit exhibits more variation in the former qualities, while inconsistencies are less likely to be reflected in responses from its multi-participant discussions.
- Discussion: DialoGPT shows some compositionality by composing uncommon follow-up utterances, but it struggles with less frequently observed concepts such as consistent and understandable.The paper gives “You really don’t know much?” as an example of a composed follow-up utterance.
- Discussion: FED may be improved through additional conversational or human-annotated fine-tuning, but it already measures many qualities without additional fine-tuning.The metric remains vulnerable to gaming and is intended for validation and model tuning rather than replacing human evaluation.
- Discussion: FED was assessed on Meena, Mitsuku, and Human conversations, with strong correlation across all three systems.The authors suggest this performance may hold for other open-domain systems, while noting that the assessment covered only these agents.
- Discussion: The FED dataset contains only open-domain chit-chat, so its generalization to goal-oriented dialog remains unresolved.The paper notes that self-supervised fine-tuning on a new domain may be necessary because DialoGPT did not observe goal-oriented training data.
7 Conclusion
The paper introduces FED as an unsupervised, reference-free metric for fine-grained dialog evaluation and releases a dataset for benchmarking automatic metrics. It positions pre-trained models as a basis for extending evaluation to additional qualities and dialog domains.
- Conclusion: FED combines an eighteen-quality annotated dataset with a metric that uses DialoGPT likelihoods for follow-up utterances.The dataset covers interactive conversations annotated at both turn and dialog levels; the metric does not compare against a ground-truth response.
- Conclusion: FED attains moderate to strong correlation with human judgement without training data, supervision, or reference responses.The metric is unsupervised and measures fine-grained dialog qualities.
- Future Work: The FED dataset can benchmark automatic metrics across eighteen fine-grained dialog qualities.Future work can also investigate further uses of pre-trained models for dialog evaluation.
- Future Work: Future work includes extending FED beyond open-domain chit-chat to goal-oriented dialog.The conclusion identifies this as one of several directions enabled by the dataset and metric.
http://arxiv.org/ps/2006.12719v1
The supplied figure references identify two PNG files, responses.png and weights.png.
- Figures: responses.png is identified as an available PNG figure.No axes, encoding, or comparison are described in the supplied passage.
- Figures: weights.png is identified as an available PNG figure.No axes, encoding, or comparison are described in the supplied passage.