RefNet: A Reference-aware Network for Background Based Conversation

1 Introduction

• BBCs supply background material, such as a movie plot or review, to promote topic-specific conversation.Source: §1 · p. 1
• Existing BBC generation methods can use the wrong background unit or split a complete unit, while extraction-based replies can be unnatural.A semantic unit is a background span expressing complete semantic meaning.Source: §1 · pp. 1–2
• RefNet combines generation with direct selection of background semantic units to address problems in both response fluency and background use.A decoding switcher chooses between reference and generation decoding, while a hybrid decoder selects a semantic unit or generates a token.Source: §1 · p. 2
• RefNet significantly outperforms state-of-the-art methods in automatic and especially human evaluations.The introduction attributes the architecture's contribution to combining generation-based and extraction-based methods while retaining fluency and informativeness.Source: §1 · p. 2

2.1 Generation-based methods

• Generation-based methods can produce fluent, natural replies but have difficulty locating background information and may break complete semantic units.Prior approaches include pre-selection with dynamic bi-directional attention and transforming background information into a knowledge graph.Source: §2.1 · p. 2

2.2 Extraction-based methods

• Extraction-based methods can locate relevant background spans, but rigid extraction does not meet BBCs' need for fluent, conversational responses.They were developed for reading-comprehension tasks with standard factoid questions, unlike typical BBCs.Source: §2.2 · p. 2
• RefNet is proposed to combine generation- and extraction-based methods while minimizing effects on response fluency.The stated challenge is referring to the right background information at the right time and place in a conversation.Source: §2.2 · p. 2

3 Reference-aware Network

• The BBC task takes background text K and conversational context Cτ as inputs and generates response Xτ.Source: §3 · p. 2
• The response is represented as a sequence of units, with the supplied formulation indicating a unit may be a token or a span containing n tokens.Source: §3 · p. 2
• RefNet consists of a background encoder, context encoder, decoding switcher, and hybrid decoder.Source: §3 · p. 2

3.1 Background and context encoders

• The background and context encoders produce representations that a matching layer combines into a context-aware background representation for decoding.A bidirectional GRU encodes the sequences; the matching layer uses attention to relate background states to the context.Source: §5 · p. 3, §3.1 · p. 3

5. Reference Decoding 𝑛|𝑥<𝑡

• RefNet’s four-module architecture routes decoding through a switcher to either select a background semantic unit or generate a token.The encoders represent the background and conversational context; the hybrid decoder uses those representations and the previous token.Source: §5 · p. 3

3.2 Hybrid decoder

• Reference decoding scores a background semantic unit with two-hop pointers, while generation decoding can predict a vocabulary token or copy one from the background.The reference decoder uses probabilities for the unit’s start and end positions; its soft pointers do not influence automatic differentiation during training.Source: §3.2 · pp. 3–4

3.3 Decoding switcher

• At each test-time decoding step, RefNet chooses reference decoding when P(r) ≥ P(g), and otherwise uses generation decoding.P(g) combines the probabilities of predicting a token and copying one from the background.Source: §3.3 · p. 4

3.4 State updating

• When the previous output is a span of n tokens, the state update runs n times with one token as input, so the final state encodes the span’s full semantics.The decoding state is initialized from the last states of the matching background and context representations.Source: §3.4 · p. 4

3.5 Training

• Training maximizes target-response prediction probability using generation, reference, and switcher losses, combined into a final loss.RefNet’s parameters and word embeddings are learned end to end by back-propagation.Source: §3.5 · p. 4

4.1 Implementation details

• The models share 128-dimensional word embeddings, 256-dimensional GRU states, a 25,000-word vocabulary, and a 65-token context limit.Training runs for 30 epochs, with the best validation model selected by BLEU.Source: §4.1 · p. 4

4.2 Dataset

• Holl-E is a movie-chat dataset with boundary annotations for background information used in each response; experiments use its mixed-short backgrounds truncated to 256 words.Responses are explicitly generated by copying and/or modifying background sentences; the original train, validation, and test split is used.Source: §4.2 · p. 5

4.3 Baselines

• Baselines include extraction models BiDAF, R-Net, and QANet, and generation models S2S, HRED, S2SA, GTTP, CaKe, and AKGCM.S2S and HRED do not use background information; the other generation baselines incorporate background through attention, copying, or knowledge selection.Source: §4.3 · p. 5

4.4 Evaluation metrics

• Automatic evaluation uses BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L; human evaluation rates 500 samples for naturalness, informativeness, appropriateness, and humanness.F1 is also reported for extraction-based methods and RefNet, evaluating extracted spans rather than whole responses.Source: §4.4 · p. 5

5.1 Automatic evaluation

• RefNet significantly outperforms all generation-based methods on every metric except BLEU versus AKGCM, and beats CaKe by around 2%-4%.The authors attribute the gains to better use and location of background information.Source: §5.1 · p. 5, Table 1 · p. 6
• RefNet outperforms extraction-based methods in most cases, including QANet, while matching their average response length.The authors suggest generation decoding adds natural conversational words to extracted content.Source: §5.1 · p. 5, Table 1 · p. 6
• The three extraction-based methods perform comparably, but their extractive nature limits their ability to produce natural, fluent responses.The paper notes that a stronger extraction model would lead to the same conclusion for this task.Source: §5.1 · p. 5, Table 1 · p. 6

5.2 Human evaluation

• RefNet achieves the best human-evaluation performance across all metrics and surpasses CaKe in naturalness, humanness, appropriateness, and informativeness.The authors associate its naturalness and humanness with a balance between reference and generation decoding.Source: §5.2 · p. 6, Table 2 · p. 6
• QANet can locate relevant semantic units, but its responses lack contextual explanations that help workers understand them.The authors argue that extracting a background span alone is insufficient for BBCs.Source: §5.2 · p. 6, Table 2 · p. 6

6.1 Reference vs. generation decoding

• Combining reference and generation decoding improves all metrics, although either mode alone performs comparably to the other.The force-generation comparison with GTTP differs in matching and decoder initialization, so it is not a direct comparison of decoding modes alone.Source: §6.1 · pp. 6–7

6.2 Switcher loss

• Switcher loss increases performance on all metrics, especially F1, by providing supervision for when and where to choose each decoding mode.The cited analysis concerns the mixed-short background version.Source: §6.2 · p. 7, Table 5 · p. 7

6.3 Case study

• Case studies show RefNet selecting background spans or generating fluent tokens at appropriate positions, producing more informative responses.In one example, it cites “$279,167,575” between generated framing text; another illustrates accurate localization of longer spans.Source: §6.3 · p. 7
• RefNet sometimes selects short or meaningless semantic units, such as “i” and “it”; considering unit length is proposed as a way to improve reference decoding.Source: §6.3 · p. 7

• RefNet incorporates reference decoding to generate more informative responses while retaining their naturalness and fluency.Source: §7 · p. 7
• RefNet outperforms state-of-the-art methods by a large margin in automatic and human evaluations.Source: §7 · p. 7
• Supervised training requires semantic-unit boundary annotations; future work proposes weakly or unsupervised training and considering unit length or frequency.The proposed training schemes are intended to support application to other datasets and tasks.Source: §7 · p. 7