Source-linked AI summary
Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation
Shikhar Sharma, Layla El Asri, Hannes Schulz, Jeremie Zumer
TL;DR
Automated metrics correlate weakly with human judgments in non-task-oriented dialogue, raising the question of whether narrower task-oriented responses produce better alignment. The paper empirically studies this question across datasets, models, and metrics, finding stronger correlation in task-oriented dialogue and still stronger correlation with multiple references. It also finds that some evaluated datasets are solvable with simple models, motivating more challenging corpora.
Problem
Prior dialogue studies found weak metric–human correlation in non-task-oriented dialogue, while task-oriented dialogue lacked empirical validation of whether those findings applied there.
Method
The paper studies correlations between human judgments and unsupervised metrics on two task-oriented datasets, evaluates model variants, and compares human and automated evaluation.
Results
Automated metrics correlate more strongly with human judgments in task-oriented dialogue, correlate even more with multiple references, and METEOR correlates most among word-overlap metrics.
Takeaways & Limitations
METEOR is suggested instead of BLEU for task-oriented NLG, while high automated scores indicate that DSTC2 and Restaurants are not very challenging.
Takeaways & Limitations
The DSTC2 and Restaurants datasets are sufficiently easy that simple models achieve high automated-metric performance.
Abstract
from arXiv · showhide
Automated metrics such as BLEU are widely used in the machine translation literature. They have also been used recently in the dialogue community for evaluating dialogue response generation. However, previous work in dialogue response generation has shown that these metrics do not correlate strongly with human judgment in the non task-oriented dialogue setting. Task-oriented dialogue responses are expressed on narrower domains and exhibit lower diversity. It is thus reasonable to think that these automated metrics would correlate well with human judgment in the task-oriented setting where the generation task consists of translating dialogue acts into a sentence. We conduct an empirical study to confirm whether this is the case. Our findings indicate that these automated metrics have stronger correlation with human judgments in the task-oriented setting compared to what has been observed in the non task-oriented setting. We also observe that these metrics correlate even better for datasets which provide multiple ground truth reference sentences. In addition, we show that some of the currently available corpora for task-oriented language generation can be solved with simple models and advocate for more challenging datasets.
1 Introduction
The paper examines whether automated metrics align with human judgments in task-oriented dialogue, where narrower domains and lower response diversity may make comparison to references more appropriate.
- Background: Rule-based and template-based systems remain common in production, but their template maintenance becomes cumbersome as complexity increases.Earlier corpus-based, statistical, and deep-learning approaches provide alternatives.
- Motivation: Automated language evaluation is difficult because word-overlap metrics can score appropriate diverse dialogue responses poorly.Human evaluation is reliable but expensive and not always available.
- Motivation: Task-oriented dialogue generates responses in narrower domains with less diversity than non-task-oriented dialogue.Generation is often framed as translating an abstract dialogue-act representation into a sentence.
- Contributions: The study measures correlations between human judgments and several unsupervised metrics on two popular task-oriented dialogue datasets.It also introduces variants of existing models and evaluates them using these metrics.
- Findings: Automated metrics correlate more strongly with human judgments in task-oriented dialogue than in non-task-oriented dialogue.Correlation is even stronger when datasets provide multiple reference sentences.
2 Related Work
Prior evaluation research found weak metric–human agreement in non-task-oriented dialogue, while task-oriented dialogue lacked empirical validation of whether those conclusions transfer.
- Automated evaluation: Previous studies found only weak correlation between automated metrics and human judgments in non-task-oriented dialogue.The evaluated metrics included word-overlap and embedding-based metrics.
- Learned metrics: ADEM predicts human-like NLG scores and correlated better with human judgments than other automated metrics.Its drawback is the requirement for expensive human ratings.
- Alternative approaches: Reinforcement-learning approaches train end-to-end dialogue systems through simulated dialogues and policy-gradient rewards.The reward function evaluates specific properties of generated responses.
- Alternative approaches: Adversarial approaches use discriminators or independent evaluation metrics, but their viability and standardization require further analysis.They are also model-dependent, and more research is needed for task-oriented dialogue.
- Research gap: Task-oriented dialogue had not yet been empirically validated against the non-task-oriented conclusions, despite using BLEU and human evaluation.Most prior work focused on non-task-oriented dialogue because automated metrics performed poorly there.
3 Metrics
The metric evaluation pipeline compares predictions with references and aggregates scores across the corpus, selecting the maximum similarity when multiple references exist.
- Evaluation procedure: For multiple references, the pipeline computes prediction–reference similarity separately and selects the maximum value.The resulting scores are then averaged across the entire corpus.
3.1 Word-overlap based metrics
The paper evaluates word-overlap metrics based on n-gram matching, aligned unigram precision and recall, and longest-common-subsequence overlap.
- BLEU: BLEU compares candidate and reference n-grams using corpus-level modified precision.The paper computes BLEU-1 through BLEU-4 and also evaluates sentence-level correlation with human judgments.
- BLEU: BLEU-N uses a brevity penalty and weighted log modified precisions through maximum n-gram order N.The paper reports corpus-level BLEU and computes sentence-level BLEU for correlation analysis.
- METEOR: METEOR aligns candidate and reference unigrams using exact, stem, synonym, and paraphrase matches.It then combines unigram precision and recall in an Fmean score with a fragmentation penalty.
- METEOR: METEOR weights recall nine times as highly as precision when computing Fmean.The score also includes a penalty term.
- ROUGE: ROUGE-L is an F-measure based on the longest common subsequence between candidate and reference utterances.ROUGE was originally introduced for summarization.
3.2 Embedding based metrics
Embedding-based metrics compare generated and reference sentences through sentence representations or direct similarity, rather than relying only on word overlap. The section covers averaged embeddings, vector extrema, and greedy matching.
- Embedding-based metrics compute cosine similarity between embeddings of predicted and reference sentences instead of relying on word overlaps.
- Skip-Thought uses an unsupervised recurrent encoder to embed sentences and a decoder trained to predict preceding and following sentences.The paper uses the authors’ pre-trained Skip-Thought encoder, trained on BookCorpus.
- Embedding average: Average embeddings represent a sentence by averaging the embeddings of its composing words.The word vectors are denoted e_w for words w in candidate sentence C.
- Vector extrema: Vector extrema represent a sentence by selecting the most extreme word-embedding value for each dimension.For each dimension d, the method chooses the maximum or minimum value with the greater absolute magnitude.
- Greedy matching: Greedy matching directly computes similarity between a candidate and reference by averaging bidirectional word-level cosine similarities.Each candidate word is greedily matched to a reference word, then the roles are reversed and the two scores are averaged.
4 Response Generation Models
The study evaluates response-generation models that translate dialogue acts and slot information into utterances. Models range from random retrieval and LSTM decoding to sc-LSTM and hierarchical lexicalized architectures.
- All models take dialogue acts with possible slot types and values as input and translate them into an utterance.For example, inform(food = Chinese) maps to a sentence about looking for a Chinese restaurant.
- The random model selects a training reference matching the dialogue act and slots while ignoring slot values.Ignoring values mishandles special values such as “yes”, “no”, and “don’t care”, lowering performance slightly.
- LSTM: The LSTM model encodes dialogue acts and slot types as a binary Dialogue Act vector and feeds it to the decoder at every time-step.It produces delexicalized sentences with placeholders, whose slot values are copied directly from the input.
- The delex-sc-LSTM replaces the LSTM decoder units with sc-LSTM units while retaining an input vector without slot-value information.This setup tests whether the sc-LSTM reading gate improves performance on small task-oriented dialogue datasets.
- hierarchical-lex-delex-sc-LSTM: The hld-scLSTM uses a hierarchical LSTM encoder that processes each dialogue act’s delexicalized act-slot pair and lexicalized slot value.Its lexicalized value encoder is an LSTM over word embeddings, unlike the mean-based encoding in the earlier ld-sc-LSTM model.
- hierarchical-lex-delex-sc-LSTM: The hld-scLSTM decoder receives the encoder output, Dialogue Act vector, and previous-word embedding at each time-step.The Dialogue Act vector is also supplied to the sc-LSTM cell so its reading gate can regulate decoding.
5 Experiments
Experiments compare automated metrics with human ratings on task-oriented dialogue datasets and evaluate several generation models. The results show positive metric–human associations, stronger evidence with multiple references, and that simple models perform well on these corpora.
- Evaluation: High automated scores and comparable LSTM and d-scLSTM results indicate that these datasets can be handled by simple models, while hld-scLSTM consistently leads word-overlap metrics.The d-scLSTM’s larger parameter count may cause overfitting on the relatively small datasets; hld-scLSTM benefits from access to lexicalized slot values.
- Human rating collection: Human evaluation used 20 dialogue acts per dataset, five responses per context, 18 evaluators, and a 1–5 Likert scale.The responses included the gold response and four model-generated responses, presented in random order.
- Human rating collection: Seven evaluators were removed for Cohen’s κ < 0.1, leaving 11 users whose pairwise agreement was generally fair.Most user pairs had Cohen’s κ > 0.3.
- Correlation between automated metrics and human scores: Automated metrics generally correlate positively with human scores, except Spearman correlation for BLEU-N, while human scores correlate best with other human scores in most cases.This contrasts with the absence of strong correlation trends previously observed in non-task-oriented dialogue.
- Correlation between automated metrics and human scores: Multiple gold references can improve word-overlap correspondence because generated responses are more likely to overlap substantially with at least one reference.DSTC2 has one gold reference per context, whereas Restaurants provides two.
- Correlation between automated metrics and human scores: METEOR consistently correlates with human evaluation on both datasets, partly because its alignment includes synonymy and paraphrasing.The authors relate this result to the task-oriented NLG setting’s translation-like structure.
6 Discussion
The study finds that METEOR correlates best with human judgments among tested word-overlap metrics, while multiple references improve metric reliability. High scores also suggest DSTC2 and Restaurants are too easy, motivating larger and more complex datasets.
- METEOR correlates most with human judgments among the evaluated word-overlap metrics, making it preferable to BLEU for task-oriented NLG.Human evaluation remains more reliable than the automated metrics overall.
- Multiple reference sentences improve the reliability of word-overlap metrics, whereas all BLEU-N metrics negatively correlate with human evaluation on DSTC2’s single-reference examples.The comparison is between Restaurants, which provides multiple references, and DSTC2, which provides one.
- Beam search improves generated-sentence quality significantly compared with stochastic sampling and raises automated-metric scores relative to prior stochastic-decoding results.The comparison is reported for similar models and references Wen et al. (2015b).
- After adding a slot error rate penalty, the vanilla LSTM performs as well as the delexicalized semantically controlled LSTM on these two datasets.This suggests that the additional sc-LSTM complexity offers no significant advantage for DSTC2 and Restaurants.
- High automated-metric performance indicates that DSTC2 and Restaurants are not very challenging for task-oriented NLG.The authors advocate larger and more complex datasets, including Frames and E2E NLG Challenge.