Source-linked AI summary
Dialogue Natural Language Inference
Sean Welleck, Jason Weston, Arthur Szlam, Kyunghyun Cho
TL;DR
Dialogue models can generate inconsistent responses, motivating a formulation of dialogue consistency as natural language inference. The paper creates Dialogue NLI and re-ranks candidate utterances with a model trained on it; re-ranking improves consistency across three evaluation sets. Its scope excludes previous-utterance contradiction pairs from the reported experiments.
Problem
Dialogue models face a long-standing consistency problem because semantically plausible responses can still contradict prior dialogue or a persona.
Method
The paper creates Dialogue NLI and penalizes candidate utterances predicted to contradict a persona sentence when re-ranking dialogue responses.
Results
Re-ranking improves all three reported metrics across three evaluation sets, including Hits@1, Contradict@1, and Entail@1.
Takeaways & Limitations
Dialogue NLI provides a new domain for NLI models and supports incorporating NLI components into downstream dialogue tasks.
Takeaways & Limitations
Reported experiments do not use the additionally released previous-utterance pairs, and filtering previous-utterance contradictions is left for future work.
Abstract
from arXiv · showhide
Consistency is a long standing issue faced by dialogue models. In this paper, we frame the consistency of dialogue agents as natural language inference (NLI) and create a new natural language inference dataset called Dialogue NLI. We propose a method which demonstrates that a model trained on Dialogue NLI can be used to improve the consistency of a dialogue model, and evaluate the method with human evaluation and with automatic metrics on a suite of evaluation sets designed to measure a dialogue model's consistency.
1 Introduction
Dialogue models can produce semantically plausible but inconsistent responses, and existing persona-based approaches do not eliminate this problem. The paper frames consistency as NLI, introduces Dialogue NLI, and uses it to improve dialogue consistency.
- Inconsistent responses such as “i’m a lawyer” followed by “i’m a doctor” are jarring despite being individually plausible.
- Persona-based dialogue aims to make utterances reflect an agent’s personal facts, but these models still face consistency errors.
- NLI maps sentence pairs to entailment categories and has been used for sentence representations and NLP model evaluation.
- Downstream use of NLI models remains underexplored, although such use may improve task performance and produce new datasets or expose model issues.
- The paper creates Dialogue NLI with entailment, neutral, and contradiction labels, then re-ranks dialogue utterances using a Dialogue NLI model.
2 Dialogue Consistency and Natural Language Inference
The paper defines persona consistency errors as contradictions between utterances and a persona, then reduces their detection to sentence-pair NLI. Dialogue generation supplies candidate utterances that an NLI model can classify as entailing, neutral, or contradicting.
- Dialogue Generation: Dialogue generation predicts the next utterance from a conversation prefix, while persona-based dialogue additionally conditions prediction on the agent’s persona.
- Consistency: A persona consistency error occurs when an utterance contradicts part of the agent’s persona, including statements unlikely to be said by the same persona.
- Natural Language Inference: NLI associates sentence pairs with entailment, neutral, or contradiction labels and learns to generalize this classification to new pairs.
- Reducing Dialogue Consistency to NLI: Dialogue consistency is reduced to NLI by assuming contradictions, entailments, and neutral interactions are contained in sentence pairs.
- The paper constructs a dialogue-derived dataset for training the NLI function and proposes incorporating it into next-utterance prediction.
3 Dialogue NLI Dataset
Dialogue NLI is a human-generated dataset of sentence pairs labeled entailment, neutral, or contradiction, constructed from persona-based dialogue using annotated fact triples. It combines rule-based pair generation with a crowdsourced gold-standard test set.
- Dataset composition: Dialogue NLI contains utterance–persona and persona–persona sentence pairs labeled entailment, neutral, or contradiction.The dataset also includes additional utterance–utterance pairs, but the paper’s experiments do not use them.
- Label generation: Sentence pairs are labeled by triple structure: sentences sharing a triple entail each other, while independent or unrelated triples provide neutral examples.Neutral examples include miscellaneous utterances, relation-swapped facts, and persona pairings whose triples differ.
- Label generation: Contradictions are generated through swapped relations, changed entity values, or altered numbers in associated sentences.Examples include like activity versus dislike, short versus tall, and replacing a sentence’s number with a different sampled integer.
- Triple annotation: Each persona sentence is annotated as a fact triple (e1, r, e2) through schema-guided Mechanical Turk labeling of relations, entity categories, values, and optional quantities.The annotation covers 10,832 persona sentences and permits out-of-schema entity values through a text box.
- Validation: The gold-standard test set retains examples for which at least two of three annotators agree with the dataset label.Utterances are associated with persona triples when an entity value appears in the utterance or word similarity reaches τ = 0.9.
- Dataset composition: All sentences come from human-generated Persona-Chat dialogues, creating a natural dialogue-domain NLI dataset distinct from synthetic-template or image-caption benchmarks.Persona-Chat supplies both dialogue utterances and persona sentences for the constructed pairs.
4 Consistent Dialogue Agents via Natural Language Inference
The method re-ranks dialogue candidates by penalizing persona contradictions detected by a Dialogue NLI model, with confidence and hyperparameters controlling the penalty. It targets fewer persona contradictions, while future work could also address contradictions between previous utterances.
- Candidate utterances are re-ranked when a Dialogue NLI model predicts contradiction with a persona sentence.Contradicting candidates receive a score penalty weighted by the model’s contradiction confidence.
- The described method focuses on persona contradictions, while future work could additionally filter contradictions between previous utterances.
- The pipeline applies NLI to each candidate-persona pair, predicting entailment, neutrality, or contradiction with a confidence value.The dialogue model first produces ranked scores for candidate next utterances, which are then adjusted using NLI-based contradiction scores.
- For each candidate, the contradiction score is the highest confidence among its predicted contradictions with persona sentences.Candidates with no predicted persona contradiction receive a contradiction score of zero.
- New candidate scores combine the dialogue model’s score with the contradiction penalty before candidates are sorted.The penalty’s influence is controlled by λ and k; λ = 0 corresponds to no re-ranking, while λ = 1 can move a candidate with contradiction score 1.0 to the k’th position.
5 Experiments
The experiments evaluate NLI models on Dialogue NLI and test whether NLI-based utterance reranking improves persona consistency through automatic metrics and human judgments.
- 5 Experiments: ESIM reaches 88.2% test accuracy on Dialogue NLI, while InferSent reaches 85.68%.
- 5 Experiments: An InferSent model trained on SNLI achieves only 47.03% on Dialogue NLI, indicating strong distribution mismatch between the datasets.
- 5 Experiments: Triple-based inputs achieve 99.69% accuracy, showing that the underlying sentence triples contain sufficient information for near-perfect Dialogue NLI classification.
- 5 Experiments: The automatic evaluation uses Haves, Likes, and Attributes sets containing ground-truth, entailment, contradiction, and random candidates.
- 5 Experiments: NLI reranking improves all three metrics across all evaluation sets, increasing Hits@1 and Entail@1 while reducing Contradict@1.
- 5 Experiments: Human evaluation also improves with NLI reranking, notably for fine-grained consistency (0.27 vs. 0.35) and contradiction score (0.25 vs. 0.16).
6 Conclusion
The paper concludes that NLI can improve a downstream dialogue task through Dialogue NLI, NLI-based reranking, and persona-consistency evaluation sets.
- 6 Conclusion: The paper demonstrates that natural language inference can improve performance on a downstream dialogue task.
- 6 Conclusion: Its contributions include Dialogue NLI, a reranking method, and an evaluation set for measuring persona consistency.
A.1 Schema
The schema organizes the relations and entity categories used to represent persona and dialogue facts in Dialogue NLI.
- A.1 Schema: The relation schema includes attributes such as origin, location, profession, education, preferences, abilities, and activities.
- A.1 Schema: Additional triples with a not have relation are extracted using a dependency-tree pattern.
- A.1 Schema: The entity schema includes categories such as people, places, organizations, professions, media, foods, animals, and vehicles.
A.2 Relation Swaps
Contradiction and neutral examples are generated by swapping relations or entities while preserving controlled semantic relationships.
- A.2 Relation Swaps: Contradiction swaps pair positive relations with opposing relations, including have with not have and like with dislike.
- A.2 Relation Swaps: Neutral swaps replace entities or combine have and not have relations across supertype-subtype pairs such as pet and dog.
A.3 Entity Swaps
Contradiction examples are constructed by swapping entities for specified relation types and by applying antonym or predefined opposing-entity swaps.
- A.3 Entity Swaps: For listed relations, replacing entities is assumed to produce a contradiction.The relations include professions, preferences, locations, attributes, and statuses.
- A.3 Entity Swaps: Other relations use WordNet antonym pairs and predefined opposing swaps such as blonde–brunette and clean–dirty.Additional pairs include large–tiny, carnivore or omnivore–vegan or vegetarian, and depressed–happy or cheerful.
B Experiment Details
The experiments specify separate InferSent and ESIM NLI configurations, a ParlAI dialogue-model setup, and Bayesian models for human-rating and utterance-pair analyses.
- Experiment 1: InferSent uses Adam with learning rate 0.001, while ESIM uses a 1-layer bidirectional LSTM with hidden dimension 1024 and learning rate 0.0001.The remaining hyperparameters follow the stated open-source or InferSent configurations.
- Experiment 2: The dialogue model is trained in ParlAI on personachat:self original using KVMemnnAgent hyperparameters, with ESIM as the NLI model.This setup uses the same ESIM model described for Experiment 1.
- 1-5 star rating: The 1–5 star model represents approach quality and annotator bias as latent normal variables and infers each approach’s posterior mean and variance from observed scores.Observed scores are modeled with means combining underlying model quality and annotator bias.
- Utterance-pair selection: The utterance-pair model represents latent model scores and annotator bias with normal variables and adds turn bias for dialogue-turn effects.Annotators label each utterance pair as consistent and/or contradictory relative to the personas.
- Inference: Posterior inference uses Pyro with the no-u-turn sampler.The sampling procedure is specified for the Bayesian models.