PaperlayerView exact revision on arXiv

Source-linked AI summary

Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems

Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, Ming Zhang

arXiv:1610.07149v1cs.CL

TL;DR

Open-domain dialog systems face limitations in both retrieval and generation. This paper combines them by feeding a retrieved reply to a generator and reranking retrieved and generated candidates. The ensemble outperforms its underlying retrieval and generation systems, and ablations indicate that both mechanisms contribute.

  • Problem

    Open-domain retrieval cannot produce replies absent from its database and often ranks by surface forms, while generators can produce short, meaningless replies.

    Source: §1 · p. 2

  • Method

    The system feeds a retrieved reply alongside the query to an RNN-based biseq2seq generator, then sends the generated reply back for post-reranking with the retrieved candidate.

    Source: §1 · p. 2

  • Results

    The ensemble outperforms its underlying retrieval and generation systems by a large margin, and ablations show that biseq2seq and post-reranking both play an important role.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The ablation results support the contribution of both biseq2seq and post-reranking to this ensemble.

    Source: §5 · p. 9

    Training RNN generators on very large datasets is time-consuming, and their performance may saturate after several million samples.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Open-domain human-computer conversation has attracted much attention in the field of NLP. Contrary to rule- or template-based domain-specific dialog systems, open-domain conversation usually requires data-driven approaches, which can be roughly divided into two categories: retrieval-based and generation-based systems. Retrieval systems search a user-issued utterance (called a query) in a large database, and return a reply that best matches the query. Generative approaches, typically based on recurrent neural networks (RNNs), can synthesize new replies, but they suffer from the problem of generating short, meaningless utterances. In this paper, we propose a novel ensemble of retrieval-based and generation-based dialog systems in the open domain. In our approach, the retrieved candidate, in addition to the original query, is fed to an RNN-based reply generator, so that the neural model is aware of more information. The generated reply is then fed back as a new candidate for post-reranking. Experimental results show that such ensemble outperforms each single part of it by a large margin.

1 Introduction

Open-domain conversation relies on data-driven retrieval and generation, but each has distinct shortcomings. The paper combines them by conditioning a generator on a retrieved reply and reranking both candidates; the ensemble outperforms either component across subjective and objective metrics.

1 Introduction

  • Retrieval systems return replies from a query-reply database, but cannot produce new utterances and typically rank candidates using surface-form features rather than semantics.The database contains query-reply pairs, so replies must already appear there.Source: §1 · p. 1
  • Generative systems can synthesize replies, but open-domain generators tend to produce short, generic, low-substance utterances when a query provides insufficient information.The paper cites examples such as “I don’t know” and “something.”Source: §1 · pp. 1–2
  • The proposed system feeds a retrieved reply alongside the query to a biseq2seq generator, then reranks the generated and retrieved candidates with the retrieval scorer.It returns whichever candidate ranks higher.Source: §1 · p. 2
  • The ensemble integrates the systems through retrieved-candidate conditioning and post-reranking, with the latter scoring both retrieved and generated replies.The authors describe this as, to their knowledge, the first combination of retrieval and generation for open-domain conversation.Source: §1 · p. 2
  • The ensemble consistently outperforms each single component on several subjective and objective metrics.The introduction also reports that both retrieval and generation contribute importantly to the overall approach.Source: §1 · p. 2

2 The Proposed Model Ensemble

The ensemble retrieves a candidate reply, conditions a biseq2seq generator on both that reply and the query, then reranks the retrieved and generated candidates. Its components are trained separately, with retrieval using crowd-sourced relevance labels and generation using human-human utterance pairs.

2.1 Overview

  • Retrieval first returns a candidate reply, and a reranker selects between it and the generated reply as the final response.The retrieval system searches tens of millions of human-human utterance pairs; it uses bag-of-words retrieval to produce up to 1,000 candidates, then ranks them using query-query and query-reply relevance features.Source: §2.1 · pp. 2–3, §2.2 · p. 3

2.2 Retrieval-Based Dialog System

  • The retrieval system finds up to 1,000 coarse candidates by query-word overlap, then ranks them using a classifier’s relevance scores and query-query and query-reply features.Its database contains tens of millions of human-human utterance pairs; features include word overlap and cosine measures of topic-model coefficients and word embeddings.Source: §2.2 · p. 3

2.3 The biseq2seq Utterance Generator

  • The biseq2seq generator encodes the query and retrieved reply separately, concatenates their vectors, and uses the transformed result to initialize reply generation.The retrieved reply is included as a reference alongside the query; the two encoders and decoder use different parameter sets.Source: §2.3 · pp. 3–4, Figure 2 · p. 4

2.4 Post-Reranking

  • A post-reranker uses manually engineered features to choose between the retrieved and generated replies, filtering replies that are meaningless or less relevant.The retrieval system’s query-reply scorer selects the final candidate.Source: §2.4 · p. 4

2.5 Training

  • The components are trained separately: retrieval uses a 10k-sample relevance dataset, while biseq2seq learns from human-human pairs with retrieved candidates as input.Crowd-sourced labels outperform original negative sampling; generation uses standard cross-entropy over reply words and mini-batched AdaDelta.Source: §2.5 · pp. 4–5

3 Evaluation

The evaluation tests the retrieval–generation ensemble on Chinese human-computer conversation against its component systems. The full ensemble leads on human evaluation and BLEU, while the analyses examine the contribution of its components and show examples of reranking.

3.1 Experimental Setup

  • The evaluation uses a 7-million-pair retrieval database and a separate generation dataset with 1,606,741 query-reply pairs.The generation data are formed as query, retrieved candidate, and target-reply triples; 6,741 triples are held out for testing.Source: §3.1 · p. 5

3.3 Overall Performance

  • +13.6% in averaged human scores over the state-of-the-practice retrieval system, with the ensemble achieving the highest human-evaluation and BLEU performance.The comparison is reported in Table 2; the retrieval baseline also outperforms RNN-based sequence generation in this experiment.Source: Table 2 · p. 6, §3.3 · p. 7

3.2 Competing Methods

  • The comparisons include retrieval, seq2seq, biseq2seq, and post-reranking ensembles with either seq2seq or biseq2seq.The retrieval system is described as a state-of-the-practice component and a strong baseline; baselines were trained and tuned comparably when applicable.Source: §3.2 · p. 6

3.3 Overall Performance

  • The retrieval baseline outperforms RNN-based sequence generation here, unlike the result reported in.The authors suggest database size, features, and retrieval methods may help explain the difference.Source: §3.3 · p. 7

3.4 Analysis and Discussion

  • Biseq2seq significantly outperforms conventional seq2seq on BLEU, and its replies are reported as more meaningful and longer.The authors attribute the richer output to incorporating a retrieved candidate; retrieved-reply entropy is not a fair comparison because retrieval filters short, meaningless replies.Source: §3.4 · p. 7, Table 3 · p. 7
  • The post-reranker selects retrieved replies 54.65% of the time with seq2seq, versus 44.77% with biseq2seq.With biseq2seq, generated replies are selected 55.23% of the time; the authors note both sources contribute a significant portion.Source: §3.4 · pp. 7–8, Figure 3 · p. 8
  • The full Rerank(Retrieval,biseq2seq) model outperforms Rerank(Retrieval,seq2seq), Retrieval, and biseq2seq on all reported metrics except BLEU-4.The ablations support a role for both biseq2seq and post-reranking in the ensemble.Source: §3.4 · p. 8

3.5 Case Study

  • In the case study, biseq2seq outputs reuse content words from retrieved replies, and the post-reranker selects a more appropriate reply.The table also includes conventional seq2seq outputs, which are not considered for reranking.Source: §3.5 · pp. 8–9, Table 4 · p. 8

4 Related Work

Prior open-domain dialog work includes retrieval and generation approaches, while the paper presents combining them as novel.

4 Related Work

  • Earlier domain-specific dialog systems commonly used preconstructed ontologies, state tracking, and slot filling to match user requirements.Examples include train routing, movie information, and tutoring.Source: §4 · p. 9
  • Open-domain retrieval work searched related queries and replies, using handcrafted or neural matching features and, in one approach, knowledge-base content.The knowledge-base approach used related entities when conversation reached a stalemate.Source: §4 · p. 9
  • Prior generation work included phrase-based machine translation and RNN reply generation, alongside methods adding informative words or topics and sampling from a variational encoder.The passage notes that RNNs tend to produce short, meaningless utterances.Source: §4 · p. 9
  • The paper claims to be the first to combine retrieval-based and generation-based dialog systems, and identifies biseq2seq with post-reranking as its new insight.Source: §4 · p. 9

5 Conclusion and Future Work

The conclusion reports that the retrieval-generation ensemble outperforms its underlying systems, with both proposed mechanisms contributing; future work may explore other combinations and data-driven approaches.

5 Conclusion and Future Work

  • The ensemble outperforms its underlying retrieval and generation systems by a large margin.Source: §5 · p. 9
  • Ablation tests show that both biseq2seq and post-reranking play an important role in the ensemble.Source: §5 · p. 9
  • Future work directions include developing new ways to combine retrieval and generative systems and incorporating other data-driven approaches to conversation.Source: §5 · p. 9

Related papers

Loading 1610.07149v1…