Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features

• Knowledge-grounded dialogue should provide verifiable information from source documents rather than subjective or invented personal information.Source: §1 · p. 1
• Existing datasets mix evidence-supported responses with chit-chat, personal experiences, and observations, so naïve training cannot ensure faithful informative outputs.Source: Figure 1 · p. 1, §1 · p. 2
• The paper avoids costly new data collection by disentangling conversational styles with controllable text-generation techniques.Source: §1 · p. 2
• Control features based on evaluation measures are added as input tokens, while resampling restricts generated outputs to satisfy those measures.Source: §1 · p. 2
• Human evaluations find the controlled models more faithful and objective, while the proposed evaluation measures correlate with human judgments of groundedness.Source: §1 · p. 2

2 Task

• The task generates a natural response from conversation history and labelled evidence while excluding unsupported, subjective, or hallucinated information.Source: §2 · p. 2
• Hallucination is defined as information neither directly stated nor inferrable from external documents, and purely extractive responses are insufficient.Source: §2 · p. 2

2.1 Evaluation measures

• The proposed measures estimate objective voice, whether the response adds information beyond the evidence, and whether it is entailed by the evidence.Source: §2.1 · p. 2
• Objective voice is estimated from first-person singular pronouns, lexical precision measures evidence-word overlap, and entailment uses an MNLI-trained RoBERTa model.Source: §2.1 · p. 3

• Wizard of Wikipedia contains multi-turn dialogues in which a wizard labels evidence spans, with development and test sets divided into seen and unseen topics.Source: §3 · p. 3
• The experiments model wizard responses using gold-labelled evidence and previous dialogue history as inputs.Source: §3 · p. 3
• Many dataset utterances embellish document information with chit-chat, opinions, intuitions, or world knowledge because crowdworkers naturally personalize conversations.Source: §3 · p. 3
• Training only on highly grounded responses would exclude substantial data: 44% of training responses are first-person and only 23% are predicted entailed.The paper instead incorporates conversational-style features so models can learn differences between faithful and other utterances.Source: §3 · p. 3

4.1 Generation Model

• T5 and GPT-2 encode evidence and dialogue history, then generate the next utterance by minimizing cross-entropy.The input contains evidence tokens and speaker-delimited conversation history; GPT-2 additionally uses token-type embeddings for evidence and speakers.Source: §4.1 · p. 4
• The generative models have incompletely understood worst-case behavior, and their training data may contain unfair distributions and factual inaccuracies that models can inherit.Generated outputs may also be only superficially similar to human-written dialogue.Source: §4.1 · p. 4

4.2 Adding controllable generation

• Control codes and decoding-time resampling are introduced to enhance groundedness according to entailment, lexical precision, and objective-voice measures.Control codes are learned during training, while resampling repeatedly samples until responses satisfy the desired evaluation criteria, up to a cutoff d.Source: §4.2 · p. 4, §4.2.1 · pp. 4–5, §4.2.2 · p. 5

4.2.1 Control Code Features

• Control-code features prepend labels for entailment, lexical precision, and objective voice, with decoding set to entailed, high-precision, and no-first-person responses.The objective-voice control excludes first-person pronouns, lexical precision uses high, medium, and low terciles, and entailment uses NLI-based labels.Source: §4.2.1 · p. 5

4.2.2 Controlled resampling

• Controlled resampling directly filters generated responses for high lexical precision, objective voice, and predicted entailment when retraining is unavailable or direct control is needed.Sampling continues until a response satisfies the measures or the cutoff d is reached.Source: §4.2.2 · p. 5

5 Experiments

• Experiments use GPT-2 and T5 with automatic metrics including BLEU, objective voice, evidence-word precision and recall, and predicted entailment.Human evaluation is emphasized because controllable models use the same evaluation measures as training controls.Source: §5 · p. 5, §5.2 · p. 5

5.3 Ablation of Control Code Features

• Using all control-code features generally improves every automatic metric over the base model in the seen-topic development ablation.Individual features usually improve the metric used during their training, while also improving other metrics.Source: §5.3 · pp. 5–6, Table 2 · p. 6

5.4 Automatic Metric Results on Test Set

• Control codes improve automatic metrics over baselines, while adding resampling provides further gains but is less effective alone.Resampling is more impactful for models trained without controllable features than for models combining control codes and resampling.Source: §5.4 · pp. 6–7
• Controllable T5 models generally outperform other models on the evaluation measures for objective voice, evidence overlap, and entailment.The dodecaDialogue model exceeds the best controllable variant on BLEU and recall, possibly because its outputs average 19 tokens versus 16.Source: Table 3 · p. 6, §5.4 · p. 7

5.5 Human Evaluation

• Human raters assess fluency, relevance, faithfulness, and objectivity on model responses, with high agreement across all four qualities.The study samples 100 seen-topic examples per model variant and uses three raters per example on a 1–5 Likert scale.Source: §5.5 · p. 7
• Human evaluation rates controlled T5 models significantly higher in faithfulness and objectivity than uncontrolled models, with only a slight relevance decrease.Combining control codes with resampling retains nearly the same faithfulness and objectivity as control codes alone while improving relevance.Source: §5.5 · p. 7, Table 4 · p. 7
• Proposed evaluation measures correlate with human judgments of faithfulness and objectivity but not with relevance or fluency.Absence of first-person language correlates with objectivity at Pearson r = 0.8 and p < 0.001; lexical precision and entailment also correlate with perceived groundedness.Source: §5.5 · pp. 7–8

5.6 Qualitative Examples

• Controlled responses tend to be more concise and restrict content to information entailed by the evidence, but may become less relevant or overly extractive.A challenging example requires inference beyond directly stated evidence, which both models answer incorrectly.Source: §5.6 · p. 8

Knowledge-Grounded Dialogue

• The work assumes evidence has already been retrieved and focuses on crafting generations that remain faithful to it.Source: §6 · p. 8

Controlling hallucinations in text generation

• Prior hallucination-reduction research largely targets summarization and data-to-text generation, whereas this paper applies the problem to knowledge-grounded dialogue.Source: §6 · p. 8

Controllable Text Generation

• The approach draws on control-code inputs and decoding-time reranking or resampling to constrain generated outputs.Source: §6 · pp. 8–9
• Its novel contribution is using controllable generation to disentangle evidence-based utterances from subjective utterances in dialogue training data.Source: §6 · p. 8

• The paper frames its contribution around three groundedness measures and two methods for integrating them into controllable dialogue systems.Source: §7 · p. 9
• Human evaluations find the controllable dialogue system's responses more objective and faithful to document-based evidence.Source: §7 · p. 9

• The proposed models could constrain dialogue outputs to verifiable information, supporting educational and information-seeking applications that require accurate information.Source: Impact Statement · p. 10
• The approach depends on reliable, unbiased evidence sources because conditioning on unreliable resources could create misuse risks.Source: Impact Statement · p. 10

• Two complete Wizard of Wikipedia training conversations are included as examples.Source: §A.1 · p. 12

A.2 Training Over Faithful Responses only

• Up to 2% higher NLI-based metrics distinguish the fully controlled model from faithful-only training, while lexical precision is fairly similar.The faithful-only model nevertheless outperforms base T5 across all automatic metrics.Source: §A.2 · p. 12
• Training on only 12k examples that avoid first person and satisfy lexical-precision and entailment criteria produces a strong T5 baseline.This filtered model is trained without explicit control codes or resampling.Source: §A.2 · p. 12, Table 6 · p. 12

A.3 Human Evaluation Instructions

• Human raters score fluency, relevance, document support, and objectivity on a 1–5 scale.The support question asks whether every part of a response is supported by the document, while objectivity excludes subjective information.Source: §A.3 · p. 12

A.4 Correlations between human judgements and automatic metrics

• The proposed automatic metrics generally correlate with human perceptions of response faithfulness and objectivity.Pearson correlation coefficients are computed by comparing each labelled example’s human rating with its automatic measurement.Source: §A.4 · p. 12, Table 7 · p. 12

A.5 Example Generation Output

• Examples compare generations produced with different control-code features and across multiple models on the Wizard of Wikipedia development set.The example tables include ablation outputs and additional model outputs.Source: §A.5 · p. 12, Table 9 · p. 14, Table 10 · p. 15