Source-linked AI summary
Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Ryan Lowe, Michael Noseworthy, Iulian V. Serban, Nicolas Angelard-Gontier, Yoshua Bengio, Joelle Pineau
TL;DR
Dialogue response evaluation is costly and existing word-overlap metrics correlate poorly with human judgements. The paper formulates evaluation as learning, trains ADEM on human scores, and reports significant utterance- and system-level correlations, including a Pearson correlation of 0.954 at system level. ADEM can also evaluate models whose responses were unseen during training, while its appropriateness focus may preserve bias toward generic responses.
Problem
Human dialogue evaluation is expensive and difficult to scale, while word-overlap metrics are biased and correlate poorly with human response-quality judgements.
Method
ADEM learns to predict human dialogue-response scores using a crowdsourced response-score dataset and a hierarchical recurrent neural network.
Results
ADEM correlates well with human judgements at response and system levels, achieving a significant system-level Pearson correlation of 0.954 and often generalizing to unseen models.
Takeaways & Limitations
ADEM provides a strong first step toward effective automatic dialogue response evaluation for rapid dialogue-system development.
Takeaways & Limitations
Because ADEM targets human appropriateness scores, it may favor generic responses and may not fully capture the desired end-goal of chatbot systems.
Abstract
from arXiv · showhide
Automatically evaluating the quality of dialogue responses for unstructured domains is a challenging problem. Unfortunately, existing automatic evaluation metrics are biased and correlate very poorly with human judgements of response quality. Yet having an accurate automatic evaluation procedure is crucial for dialogue research, as it allows rapid prototyping and testing of new models with fewer expensive human evaluations. In response to this challenge, we formulate automatic dialogue evaluation as a learning problem. We present an evaluation model (ADEM) that learns to predict human-like scores to input responses, using a new dataset of human response scores. We show that the ADEM model's predictions correlate significantly, and at a level much higher than word-overlap metrics such as BLEU, with human judgements at both the utterance and system-level. We also show that ADEM can generalize to evaluating dialogue models unseen during training, an important step for automatic dialogue evaluation.
1 Introduction
Automatic dialogue evaluation is difficult because human testing is costly and existing metrics correlate poorly with response quality. The paper introduces ADEM, which learns human-like scores and can evaluate unseen dialogue models.
- Human evaluation is expensive, difficult to scale, and can introduce bias through evaluator instructions and domain-specific expertise requirements.
- BLEU and related word-overlap metrics are biased and correlate poorly with human judgements of dialogue response quality.
- The paper collects appropriateness scores and trains ADEM with a hierarchical RNN to predict human response evaluations.
- ADEM correlates significantly with human judgement at both utterance and system levels and can often evaluate models unseen during training.
- ADEM is presented as a strong first step toward effective automatic dialogue response evaluation.
2 Data Collection
The authors build a human-scored dataset of diverse Twitter dialogue responses using crowdsourced evaluations. Candidate responses come from multiple models and are scored for quality and appropriateness-related variation.
- The dataset contains human judgements of Twitter responses conditioned on dialogue contexts and spanning a broad range of response qualities.
- Candidate responses are drawn from several different dialogue models to provide relevant, irrelevant, coherent, and non-coherent examples.
- The Twitter Corpus is chosen because advanced models are pretrained on it and its topics are broad enough to support other chit-chat domains.
3 Technical Background
The technical background contrasts dialogue’s high response diversity and context dependence with word-overlap metrics’ limited matching-based evaluation. It also introduces RNNs and LSTMs as sequential modeling foundations.
- 3.1 Recurrent Neural Networks: RNNs maintain a hidden state updated over sequential inputs, while LSTMs add gates that control hidden-state updates.
- 3.2 Word-Overlap Metrics: Word-overlap metrics such as BLEU, METEOR, and ROUGE were developed for other tasks and correlate poorly with human judgements in dialogue evaluation.
- 3.2 Word-Overlap Metrics: BLEU computes n-gram precision between reference and proposed responses and applies a brevity penalty; BLEU-N uses n-grams up to N.
- 3.2 Word-Overlap Metrics: Word-overlap metrics miss semantic similarity when appropriate responses share few or no words with the reference.
- 3.2 Word-Overlap Metrics: Because dialogue responses are highly diverse, comparing a candidate with a single reference is unlikely to capture all appropriate responses.
- 3.2 Word-Overlap Metrics: Word-overlap scores ignore conversational context, even though response adequacy depends on the context and can change when the context changes.
4 An Automatic Dialogue Evaluation
ADEM is a learned dialogue-evaluation model designed to move beyond word-overlap statistics by using hierarchical representations of the context, reference response, and model response. It combines these representations to predict human scores, with VHRED pre-training supporting learning from few labeled examples and enabling fast evaluation.
- ADEM is designed to capture semantic similarity beyond word-overlap statistics and use both context and reference response when scoring a model response.
- ADEM encodes the context, reference response, and model response, then computes a score from their representations in a linearly transformed space.The learned matrices project the model response into context and reference-response spaces, favoring responses similar to both.
- A hierarchical RNN encoder represents dialogue context by encoding words at the utterance level and utterance representations at the context level.The context-level encoder’s final hidden state provides the vector representation of the input context.
- ADEM uses the reference response to compare a model response with a known good response rather than inferring quality from context alone.The model’s simplicity also supports accurate predictions and fast evaluation for rapid dialogue-system prototyping.
- VHRED pre-training initializes the encoder from a neural dialogue model so ADEM can make predictions from few labeled examples.After VHRED training, the context-level encoder’s last hidden state is used as the input-text representation.
5 Experiments
Experiments evaluate ADEM against word-overlap and embedding baselines at utterance and system levels, including generalization to unseen dialogue models. ADEM correlates better with human judgements, though qualitative analysis reveals conservative predictions.
- Experimental Procedure: ADEM is trained with response-length subsampling to avoid using length as a proxy for human scores.Examples are binned by response length and score, then oversampled across equal-score bins.
- Utterance-level correlations: Utterance-level results show ADEM correlates far better with human judgements than word-overlap baselines.The comparison includes BLEU, ROUGE, embedding baselines, and ADEM.
- Utterance-level correlations: ADEM with VHRED embeddings outperforms the version using tweet2vec embeddings.Tweet2vec obtains reasonable but inferior performance compared with VHRED embeddings.
- System-level correlations: ADEM achieves a system-level Pearson correlation of 0.954 and produces almost the same model ranking as human scores.Word-overlap metrics fail to capture the ranking among TFIDF, Dual Encoder, HRED, and human responses.
- Generalization to previously unseen models: ADEM generalizes to all tested unseen models except the Dual Encoder, including a generative HRED model.Its performance on the full test set is comparable to an ADEM model trained on 25% less randomly selected data.
- Qualitative Analysis: Qualitative analysis finds that ADEM often identifies poor responses and suitable responses accurately, but can under-score highly rated responses.The conservative behavior is attributed to squared error loss, which penalizes extreme prediction errors.
6 Related Work
Related work spans learned evaluation for machine translation, chat-oriented dialogue, reinforcement-learning rewards, and task-oriented dialogue. ADEM differs by evaluating response quality in dialogue contexts rather than relying only on overlap, handcrafted features, or task completion.
- Evaluation methods: Machine-translation evaluation has used regression and Tree-LSTMs, whereas dialogue evaluation must additionally condition scores on conversational context.The cited approaches address translation-system evaluation rather than the dialogue-specific setting.
- Chat-oriented dialogue: Chat-oriented dialogue metrics have been trained from role-playing sessions, Wizard-of-Oz data, problematic-situation labels, or classifiers distinguishing human and generated utterances.These approaches target policy evaluation, coherence, problematic situations, or source classification.
- Reinforcement-learning rewards: Reinforcement-learning dialogue rewards use handcrafted features such as ease of answering, information flow, appropriateness, and conversational depth.The paper notes that such features capture only a small set of relevant aspects.
- Integration with ADEM: Response-level metrics from prior work could be incorporated into ADEM as learned feature terms.The paper describes adding a dot product between response features and learned parameters.
- Task-oriented dialogue: Task-oriented evaluation frameworks such as PARADISE and MeMo incorporate task-completion signals, unlike evaluation of unstructured conversational responses.PARADISE learns an automatic evaluation function through linear regression but requires measurable task completion.
7 Discussion
The discussion frames ADEM as a step toward broader automatic evaluation while identifying domain-transfer and generic-response limitations. The model evaluates appropriateness, not the full goal of engaging and meaningful interaction.
- Dataset scope: The Twitter Corpus supports training because it contains broad non-task-oriented conversations and has been used for state-of-the-art dialogue models.The authors suggest extension to Reddit when suitable pre-trained models become available.
- Limitations: Domain-transfer ability remains untested and is left for future work.The paper specifically identifies transfer to related dialogue domains as an open question.
- Limitations: ADEM favors responses rated highly for appropriateness, which may reward generic responses that fit many contexts.The authors propose future modifications to reduce this bias, including censoring length information or combining adversarial evaluation.
- Future directions: Future evaluation should target engaging and meaningful human interaction, which is closer to the chatbot end-goal than scoring a single response.The paper characterizes fully automatic evaluation of this capability as extremely challenging.
Appendix A: Further Notes on Crowdsourcing Data Collection
The crowdsourcing experiments compared several response-quality questions and found that wording strongly affected agreement. Because auxiliary questions added limited reliable information, the study retained overall quality scores for the dataset.
- Primary data collection: Approximately 2,000 human responses were collected by asking AMT workers to generate reasonable Twitter-dialogue continuations.Workers answered 20-question surveys and were instructed to avoid simple one-word responses.
- Primary data collection: Overall quality was rated on a 1–5 scale for responses drawn from human generations and approximately 1,000 responses from each evaluated model.Workers evaluated four responses from 50 contexts, with attention checks and overlapping contexts for agreement measurement.
- Initial survey design: The survey asked about appropriateness, topicality, specificity, and background information, while omitting fluency because most responses were human-written.The background question was asked only once per context.
- Initial survey findings: Overall and topicality scores had fairly high inter-annotator agreement but were strongly correlated, whereas specificity and background had very low agreement.Specificity reached κ = 0.12 and background κ = 0.05.
- Revised survey: Rephrasing specificity and background questions improved informativeness agreement to κ = 0.31, but context agreement varied sharply across participants.Context agreement exceeded κ > 0.9 for 45 participants and fell below κ < 0.1 for 113 participants.
- Final data-collection decision: The study therefore collected only overall quality scores because the auxiliary questions provided unclear additional information for dialogue evaluation.Separating the overall question and excluding misunderstood contexts improved agreement only slightly.
Appendix B: Metric Description
This appendix describes word-overlap metrics used for comparison, including BLEU, METEOR, and ROUGE-L. It also visualizes survey-score distributions and highlights how metric components account for overlap, recall, and sentence length.
- BLEU: BLEU computes corpus-level n-gram precision between ground-truth and proposed responses, then applies a brevity penalty to reduce preference for short candidates.BLEU-N combines weighted n-gram precisions with a geometric mean; the commonly used version uses N = 4.
- METEOR: METEOR explicitly aligns candidate and target responses using exact matches, synonyms, stems, and paraphrases, then combines precision and recall harmonically.Its penalty term reflects the chunkiness of resolved matches.
- Survey-score visualization: Figure 6 plots normalized response scores vertically against responses horizontally for four survey questions, with clustering indicating agreement.Overall and topicality cluster more clearly than specificity and background information.
- ROUGE-L: ROUGE-L computes an F-measure from the Longest Common Subsequence, allowing matched words to remain ordered without requiring contiguity.The LCS-based formulation differs from n-gram matching because intervening words are permitted.
Appendix C: Latent Variable Hierarchical Recurrent Encoder-Decoder (VHRED)
VHRED supplies dialogue-context representations for the evaluation model by combining hierarchical recurrent encoding with a stochastic latent variable at each dialogue turn. The latent variable models ambiguity and uncertainty while supporting diverse, coherent responses.
- Model architecture: VHRED extends HRED with a high-dimensional stochastic latent variable at every dialogue turn.Its utterance-level and context-level RNNs encode the dialogue context before response generation.
- Training objective: VHRED trains by maximizing a lower bound on the log-likelihood of generating the next response.The objective includes a KL-divergence term between approximate posterior and prior distributions.
- Latent representation: The multivariate Gaussian latent variable represents dialogue ambiguity and uncertainty through distributional mean and variance parameters.This inductive bias helps encode ambiguous contexts and generate more diverse responses.
- Pre-training motivation: VHRED context vectors are similar when their contexts induce similar distributions over subsequent responses, aligning pre-training with evaluation representations.The representation is intended to support semantic relatedness and paraphrase detection.
- Evaluation motivation: Word-overlap metrics correlate poorly with dialogue-response human judgements, motivating learned representations rather than relying only on lexical overlap.The supplied table caption states that none of the reported correlations are significant.
Appendix D: Experiments & results
The experiments selected ADEM settings using external validation and validation-based hyperparameter tuning. The reported best configurations differed between the standard and tweet2vec embedding variants.
- Model selection: ADEM used external validation for early stopping and selected PCA dimension, L2 penalty, learning rate, and batch size from validation results.This procedure was applied when obtaining the best parameter setting.
- Hyperparameters: The best standard ADEM configuration used γ = 0.075, learning rate a = 0.01, and batch size b = 32.The supplied passage does not report an associated performance score.
- Hyperparameters: ADEM with tweet2vec embeddings used PCA dimension n = 150, γ = 0.01, learning rate a = 0.01, and batch size b = 16.These settings were selected through a similar hyperparameter search.
Additional Results
Additional experiments show that preprocessing artifacts can inflate word-overlap correlations, while ADEM learns to predict human scores and remains practical, data-efficient, and less sensitive to response length. Failure analysis identifies both missed good responses and high scores for contextually unrelated short responses.
- Preprocessing effects: Removing speaker tokens caused almost all significant word-overlap correlations to disappear, especially for BLEU variants.The tokens enabled spurious bigram matches between model and reference responses.
- Evaluation speed: Approximately 2.5 minutes on a Titan X GPU were required to evaluate the test set, compared with approximately 45 minutes on CPU.The GPU timing includes encoding contexts, model responses, and references, plus PCA projection, but the model was not speed-optimized.
- Learning curves: At the beginning of training, ADEM's correlation with human scores was approximately 0.The result supports learning the evaluation parameters rather than relying on VHRED embeddings alone.
- Data efficiency: ADEM reached a Spearman correlation of 0.4 using half the available training data, or 1000 labelled examples.It still correlated significantly with humans using 100 labelled examples, or 5% of the original training data.
- Word-overlap comparison: 147 of 237 human-approved responses were ranked poorly by both BLEU-2 and ROUGE, while ADEM correctly scored 60 of those 147 above 4.These cases demonstrate ADEM's ability to recognize appropriate responses lacking reference-word overlap.
- Response-length effects: ADEM did not exhibit word-overlap metrics' bias toward responses whose lengths matched the reference response.The comparison concerns BLEU-1, BLEU-2, and METEOR, which had previously shown this similar-length bias.