PaperlayerView exact revision on arXiv

Source-linked AI summary

Incremental Transformer with Deliberation Decoder for Document Grounded Conversations

Zekang Li, Cheng Niu, Fandong Meng, Yang Feng, Qian Li, Jie Zhou

arXiv:1907.08854v3cs.CL

TL;DR

Document Grounded Conversations calls for dialogue responses that use document knowledge while integrating it with multi-turn history. The paper proposes an Incremental Transformer and a two-pass Deliberation Decoder; it reports better context coherence and knowledge relevance than competitive baselines. The authors conclude that the model generates responses with greater relevance, correctness, and coherence than state-of-the-art baselines.

  • Problem

    Document Grounded Conversations requires relevant document knowledge to be integrated with multi-turn dialogue history, a capability existing dialogue models do not exploit effectively enough.

    Source: §1 · p. 1

  • Method

    An Incremental Transformer encodes multi-turn utterances with document knowledge, while a two-pass Deliberation Decoder first focuses on context coherence and then consults relevant documents to refine responses.

    Source: §1 · p. 2, 2) Incremental Transformer Encoder · p. 3

  • Results

    The model substantially outperforms competitive baselines on context coherence and knowledge relevance, with empirical results also reporting greater response relevance, correctness, and coherence than state-of-the-art baselines.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that incremental encoding and deliberation yield a knowledge-aware and context-aware conversation representation for generating more relevant, correct, and coherent responses.

    Source: §5 · p. 9

    Simple concatenation of context and document knowledge is limited: Wizard Transformer has lower context coherence than some baselines and often produces related but incorrect knowledge responses.

    Source: §3.6 · p. 8

Abstract

from arXiv · show

Document Grounded Conversations is a task to generate dialogue responses when chatting about the content of a given document. Obviously, document knowledge plays a critical role in Document Grounded Conversations, while existing dialogue models do not exploit this kind of knowledge effectively enough. In this paper, we propose a novel Transformer-based architecture for multi-turn document grounded conversations. In particular, we devise an Incremental Transformer to encode multi-turn utterances along with knowledge in related documents. Motivated by the human cognitive process, we design a two-pass decoder (Deliberation Decoder) to improve context coherence and knowledge correctness. Our empirical study on a real-world Document Grounded Dataset proves that responses generated by our model significantly outperform competitive baselines on both context coherence and knowledge relevance.

1 Introduction

Document Grounded Conversations requires responses that use relevant document knowledge while remaining consistent with multi-turn dialogue. The paper proposes an Incremental Transformer with a two-pass Deliberation Decoder, and reports that it substantially outperforms competitive baselines on context coherence and knowledge relevance.

1 Introduction

  • Document Grounded Conversations generates natural dialogue responses about a specific document by integrating its knowledge with multi-turn dialogue history.Unlike document question answering, the task also requires contextually consistent conversation responses.Source: §1 · pp. 1–2
  • The paper identifies the need for a unified representation of multi-turn utterances and relevant document knowledge, and proposes an Incremental Transformer to build it.The authors describe the two-pass decoder as their first application of that design to document grounded conversations.Source: §1 · p. 2
  • The proposed Transformer architecture incrementally encodes multi-turn history with document knowledge and uses a two-pass decoder to improve context coherence and knowledge correctness.The first pass focuses on contextual coherence; the second consults relevant document knowledge to refine the response.Source: §1 · p. 2
  • Automatic and manual evaluations show that the model substantially outperforms competitive baselines on context coherence and knowledge relevance.The authors also report that document knowledge sometimes guides following conversations.Source: §1 · p. 2

2 Approach

The task is to generate a response using the related document and the preceding conversation. The model description introduces a framework with a Self-Attentive Encoder, Incremental Transformer, and Deliberation Decoder.

2.1 Problem Statement

  • Document Grounded Conversations aims to incorporate relevant document knowledge into multi-turn conversations.For each response, the task uses the related document and the preceding utterances with their related documents.Source: §2.1 · p. 2
  • The task defines the next response using its related document together with the preceding conversation and the documents related to earlier turns.A document may be shared across consecutive utterances.Source: §2.1 · p. 2

2.2 Model Description

  • The model framework comprises a Self-Attentive Encoder, an Incremental Transformer, and a Deliberation Decoder.The Self-Attentive Encoder independently encodes document knowledge and the current utterance.Source: §2.2 · p. 3

2) Incremental Transformer Encoder

The encoder incrementally combines multi-turn utterances with document knowledge, while a two-pass decoder first uses conversation context and then refines responses with document knowledge.

Self-Attentive Encoder

  • The encoder’s motivation is to capture long-range dependencies and identify relevant information in documents that may contain several sentences.Source: Self-Attentive Encoder · p. 3
  • The Self-Attentive Encoder represents document knowledge and utterances; its layers use multi-head self-attention followed by a position-wise feed-forward network.The document and utterance encoders share an architecture but have different parameters.Source: Self-Attentive Encoder · pp. 3–4

Incremental Transformer Encoder

  • The Incremental Transformer Encoder uses attention to combine the current utterance with prior context and document knowledge.It takes previous utterances and the document’s self-attentive representation as input.Source: Incremental Transformer Encoder · p. 4

Deliberation Decoder

  • The Deliberation Decoder first generates a context-focused response, then uses the relevant document to refine it for greater knowledge relevance.The two passes share an architecture but use different inputs.Source: Deliberation Decoder · p. 4

Training

  • The training description contrasts its loss with the original Deliberation Network’s complex joint learning framework, noting it minimizes loss as Xiong et al. (2018) do.Source: Training · p. 5

3 Experiments

The experiments evaluate document-grounded dialogue models on a shared dataset using automatic and human ratings. ITE+DD leads the reported comparisons, while analyses and examples examine the roles of incremental encoding and two-pass decoding.

3.1 Dataset

  • The dataset contains 72,922 training, 3,626 validation, and 11,577 test utterances; associated documents average about 200 words.Utterances may be casual or document-grounded, and consecutive turns by the same speaker are treated as one utterance.Source: §3.1 · p. 5

3.2 Baselines

  • Comparisons include knowledge-free and knowledge-augmented dialogue systems, KAT without context attention, and ITE+CKAD without the deliberation decoder.KAT is configured to test context exploitation; ITE+CKAD tests the deliberation decoder.Source: §3.2 · pp. 5–6

3.5 Experimental Results

  • ITE+DD achieves 0.56 Knowledge Relevance, 0.90 Context Coherence, 0.95 BLEU, and 15.11 perplexity, leading all models on these reported measures.ITE+DD also has the highest manual fluency score, 1.68; the strongest baseline scores are 0.77 BLEU and 70.30 perplexity.Source: Table 1 · p. 6, §3.5 · p. 7

3.3 Experiment Setup

  • Transformer models use three encoder and decoder layers, 512-dimensional hidden and word embeddings, eight attention heads, and the previous three utterances with related documents.Decoding uses beam size 5.Source: §3.3 · p. 6

3.4 Evaluation Metrics

  • Evaluation combines perplexity and BLEU with human 0/1/2 ratings for fluency, knowledge relevance, and context coherence; BLEU is extremely low with one reference per response.The human evaluation sampled 30 conversations containing 606 test utterances and obtained 5,454 utterances from nine models.Source: §3.4 · p. 6

3.6 Analysis and Discussion

  • The second-pass decoder uses document knowledge to reduce ground-truth perplexity and yields higher score-2 percentages for relevance and coherence than ITE+CKAD.The analysis attributes the reduced perplexity to the second pass; it also reports that ITE+CKAD surpasses KAT on BLEU and all three manual measures despite slightly worse perplexity.Source: Table 2 · p. 6, §3.6 · pp. 7–8
  • Wizard Transformer’s related-but-often-incorrect knowledge responses and weaker context coherence illustrate the limitation of representing context and documents by simple concatenation.Its Knowledge Relevance score-1 results occur twice as often as score-2 results.Source: §3.6 · p. 8

3.7 Case Study

  • Case studies report that ITE+DD responses outperform Wizard Transformer on knowledge relevance and context coherence in continuous dialogues.The examples also show second-pass corrections of a knowledge error, more detailed knowledge use, and a knowledge-related question that guides subsequent conversation.Source: Table 3 · p. 7, §3.7 · p. 8, Table 4 · p. 8

4 Related Work

Related work addresses knowledge-grounded dialogue, document-level translation, and story generation. This paper combines document knowledge with incremental multi-turn encoding and designs its two-pass decoder differently from earlier deliberation work.

4 Related Work

  • Unlike open-domain systems that incorporate unstructured knowledge, this work combines document knowledge and context utterances through attention and incrementally encodes knowledge-aware multi-turn conversations.The cited approaches include Ghazvininejad et al. (2018), Parthasarathi and Pineau (2018), and Dinan et al. (2018).Source: §4 · p. 8
  • The Deliberation Decoder assigns context and knowledge to separate passes, and its second pass directly fine-tunes the first-pass result, unlike Xia et al. (2017).The Incremental Transformer also adapts incremental encoding to create a knowledge-aware context representation.Source: §4 · pp. 8–9

5 Conclusion and Future Work

The paper concludes that its Incremental Transformer with Deliberation Decoder improves document-grounded conversations through knowledge-aware, context-aware representations and reports stronger responses than state-of-the-art baselines. Future work is to apply reinforcement learning.

5 Conclusion and Future Work

  • The proposed Incremental Transformer with Deliberation Decoder uses incremental encoding for knowledge- and context-aware representations, and the authors report more relevant, correct, and coherent responses than state-of-the-art baselines.They plan to apply reinforcement learning to further improve performance.Source: §5 · p. 9
Loading 1907.08854v3…