PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Coherence in Dialogue Systems using Entailment

Nouha Dziri, Ehsan Kamalloo, Kory W. Mathewson, Osmar Zaiane

arXiv:1904.03371v2cs.CLcs.LG

TL;DR

Open-domain dialogue evaluation lacks reliable, scalable metrics because automatic measures correlate weakly with human judgments and human assessment is expensive. The paper frames coherence evaluation as natural language inference, trains entailment models on synthesized conversational data, and finds that the approach can serve as a reasonable surrogate for human judgment. It therefore supports large-scale, less biased evaluation of response quality within the measured coherence scope.

  • Problem

    Open-domain dialogue quality is difficult to evaluate because possible responses are diverse, automatic metrics correlate weakly with human judgments, and human evaluation is costly and not scalable.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The paper treats conversation history as an NLI premise and a generated response as its hypothesis, training ESIM and BERT models on synthesized conversational inference data.

    Source: §1 · p. 1, §3 · p. 2, §4 · p. 3

  • Results

    The proposed entailment-based approach correlates reasonably with human judgment and can automatically evaluate dialogue coherence without human annotation.

    Source: §5.1 · p. 4, §6 · p. 5

  • Takeaways & Limitations

    The metrics provide a scalable, unbiased estimate of response quality for evaluating dialogue systems on large-scale datasets.

    Source: §1 · p. 2, §6 · p. 5

Abstract

from arXiv · show

Evaluating open-domain dialogue systems is difficult due to the diversity of possible correct answers. Automatic metrics such as BLEU correlate weakly with human annotations, resulting in a significant bias across different models and datasets. Some researchers resort to human judgment experimentation for assessing response quality, which is expensive, time consuming, and not scalable. Moreover, judges tend to evaluate a small number of dialogues, meaning that minor differences in evaluation configuration may lead to dissimilar results. In this paper, we present interpretable metrics for evaluating topic coherence by making use of distributed sentence representations. Furthermore, we introduce calculable approximations of human judgment based on conversational coherence by adopting state-of-the-art entailment techniques. Results show that our metrics can be used as a surrogate for human judgment, making it easy to evaluate dialogue systems on large-scale datasets and allowing an unbiased estimate for the quality of the responses.

1 Introduction

The paper addresses the difficulty of evaluating open-ended dialogue quality, where coherence is important but human judgments are costly and existing metrics are inadequate. It proposes entailment-based, automatically calculable measures as scalable approximations of human evaluation.

  • Open-ended dialogue lacks an explicit goal, making response quality and semantic adequacy difficult to measure automatically.Coherence can be assessed through consistency across the conversation, such as avoiding contradictions between successive responses.Source: §1 · p. 1
  • The paper frames dialogue consistency as a natural language inference task, treating conversation history as the premise and a generated response as the hypothesis.This projects automatic dialogue evaluation into entailment-based prediction of conversational coherence.Source: §1 · p. 1
  • The authors synthesize conversational inference data, train entailment models, and use them to evaluate coherence in dialogue conversations.The method is then compared against existing automated metrics without requiring direct human annotation for evaluation.Source: §1 · pp. 1–2
  • The proposed approach is presented as a scalable alternative that provides calculable approximations of human judgments about conversational coherence.The paper evaluates whether inference models can automatically assess dialogue coherence at scale.Source: §1 · p. 2

2 Related Work

Prior dialogue-evaluation methods either correlate weakly with human judgments, depend heavily on annotated data, or rely on human labels. This work also builds on NLI-based consistency evaluation while removing the need for human labels in its proposed procedure.

  • Word-overlap metrics such as BLEU correlate weakly with human evaluation, while ADEM depends on annotated scores and can struggle to generalize beyond its training data.These limitations make such approaches less flexible for scalable evaluation.Source: §2 · p. 2
  • Other frameworks measure topical diversity, coherence, engagement, and conversational depth but rely substantially on human labels.This dependence limits their scalability according to the paper.Source: §2 · p. 2
  • Prior NLI work used human-annotated Dialogue NLI data to classify utterance pairs as entailment, neutral, or contradiction.The present approach likewise uses NLI to detect coherence but proposes an evaluation procedure without human labels.Source: §2 · p. 2

3 Natural Language Inference

The paper formulates dialogue coherence as entailment reasoning between conversation history and a generated response. It applies ESIM and BERT models to classify these premise-hypothesis pairs as entailment, contradiction, or neutral.

3 Natural Language Inference

  • Dialogue evaluation is cast as determining whether a generated response is entailed by, contradicts, or is neutral to the conversation history.The history is the premise and the response is the hypothesis.Source: §3 · p. 2

3.1 Coherence in Dialogue Systems

  • Coherence is needed across multiple turns because responses can contradict earlier statements or follow illogical reasoning throughout the conversation.The paper illustrates this with inconsistent claims about owning cats.Source: §3.1 · p. 2
  • ESIM encodes premise and hypothesis with Bi-LSTMs, uses attention and inference-composition layers, and is further enhanced with ELMo embeddings.Its architecture also incorporates syntactic parse-tree information through Tree-LSTM components.Source: §3.1 · pp. 2–3
  • BERT uses pretrained bidirectional Transformer representations and is fine-tuned for the NLI task.The paper adopts BERT because it can model long-range dependencies and requires only a task-specific output layer.Source: §3.1 · p. 3
  • Both models learn a function that predicts entailment, contradiction, or neutral from premise-hypothesis pairs.These predictions provide the categories used to assess dialogue coherence.Source: §3.1 · p. 3

4 Inference Corpus for Dialogues

The authors construct InferConvAI, a synthesized conversational inference corpus derived primarily from Persona-Chat. It contains labeled entailment, contradiction, and neutral premise-hypothesis pairs for training dialogue-coherence models.

  • InferConvAI is built from Persona-Chat because conversational continuity makes previous utterances useful for constructing entailment examples.Appropriate on-topic responses from the conversations are mapped to the entailment label.Source: §4 · p. 3
  • Contradiction examples combine grammatically impaired sentences formed from conversation words with contradictory instances sampled from MultiNLI.These sources are intended to represent both synthetic and meaningful inconsistencies.Source: §4 · p. 3
  • Table 1 summarizes the distribution of entailment, contradiction, and neutral labels in InferConvAI.Source: Table 1 · p. 3, §4 · p. 3
  • Neutral examples come from random utterances in other conversations or generic responses such as “I don’t know.”The resulting corpus contains 1.1M premise-hypothesis pairs.Source: §4 · p. 3

5 Experiments

The experiments compare entailment-based coherence evaluation with word-level and semantic-similarity metrics against human judgments across Reddit and OpenSubtitles. Entailment models show meaningful correspondence with human coherence ratings, while semantic similarity can outperform word-level metrics and achieve competitive correlation without pre-training.

5 Experiments

  • The experiments use OpenSubtitles and Reddit, sampling 6M training dialogues, 700K development dialogues, and 40K test dialogues per dataset, with human ratings for 150 dialogues from each.Each dialogue contains three turn exchanges, and five native English speakers conducted the evaluation.Source: §5 · pp. 3–4

5.1 NLI in Dialogues

  • BERT and ESIM evaluate generated responses by treating conversation history as the premise and the response as the hypothesis, with BERT outperforming ESIM on inference-model evaluation.The models are trained on InferConvAI and tested on Reddit and OpenSubtitles, whose human ratings come from majority votes on a four-scale judgment.Source: §5.1 · p. 4
  • Entailment-predicted responses tend to receive higher human ratings, whereas neutral predictions often identify acontextual or off-topic responses that annotators dislike.Neutral-detected responses receive lower scores than contradiction-detected responses in the reported analysis.Source: §5.1 · p. 4, Figure 1 · p. 4

5.2.1 Word-level Metrics

  • Word-level Average, Greedy, and Extrema metrics correlate weakly with human judgment because they ignore word order and dependencies.These embedding baselines treat sentences as bags of independent words, motivating metrics that model richer semantic interactions.Source: §5.2.1 · pp. 4–5

5.2.2 Semantic Similarity

  • Semantic-similarity metrics correlate better than word-level metrics, with performance depending on the embedding model and dataset.USE performs better on Reddit, whereas ELMo achieves higher correlation on OpenSubtitles.Source: §5.2.2 · p. 5
  • SS with no pre-training reaches Pearson correlation -0.404 against the most recent utterance on OpenSubtitles, compared with ADEM’s 0.436 correlation.The comparison is presented as evidence that semantic similarity can achieve competitive human-alignment with substantially less training data and computation.Source: §5.2.2 · p. 5

6 Conclusion

The paper concludes that entailment techniques provide a scalable, reasonably human-aligned approach to evaluating dialogue coherence without human annotation. It also identifies engagingness as an important area for improving evaluation of dialogue strategies.

6 Conclusion

  • Entailment-based evaluation correlates reasonably with human judgment and provides an unbiased estimate of response quality without requiring human annotation.The authors present this as a scalable approach for evaluating dialogue coherence.Source: §6 · p. 5
  • Measuring conversational engagingness remains an open improvement area for evaluating different dialogue strategies.The conclusion distinguishes engagingness from the coherence-focused evaluation developed in the paper.Source: §6 · p. 5
Loading 1904.03371v2…