PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Approaches to Conversational AI

Jianfeng Gao, Michel Galley, Lihong Li

arXiv:1809.08267v3cs.CL

TL;DR

Building dialogue systems that emulate conversation, answer questions, and fulfill complex tasks has long been an elusive AI goal. This paper surveys neural approaches across conversational AI and relates them to traditional approaches. It concludes that neural methods have significantly improved dialogue-agent performance across a wide range of tasks and domains.

  • Problem

    The longstanding goal of dialogue systems that emulate conversation, answer questions, and fulfill complex tasks remained elusive until recently.

    Source: Introduction · p. 5

  • Method

    The paper reviews state-of-the-art approaches to training dialogue agents using supervised and reinforcement learning.

    Source: Introduction · p. 5

  • Results

    Neural approaches leverage deep learning and reinforcement learning breakthroughs to significantly improve dialogue-agent performance across a wide range of tasks and domains.

    Source: Conclusions and Research Trends · p. 72

  • Takeaways & Limitations

    Commercial dialogue systems provide conversational access to services and information, and most combine symbolic methods with neural models.

    Source: Conclusions and Research Trends · p. 72

    Reward functions can conflict between efficient task completion and user engagement, requiring dialogue systems to balance short- and long-term gains.

    Source: §1.3 · p. 8

Abstract

from arXiv · show

The present paper surveys neural approaches to conversational AI that have been developed in the last few years. We group conversational systems into three categories: (1) question answering agents, (2) task-oriented dialogue agents, and (3) chatbots. For each category, we present a review of state-of-the-art neural approaches, draw the connection between them and traditional approaches, and discuss the progress that has been made and challenges still being faced, using specific systems and models as case studies.

Introduction

The paper surveys neural approaches to question answering, task-oriented dialogue, and social chatbots through a unified view of optimal decision making. It connects neural and traditional methods, reviews supervised and reinforcement learning, and uses case studies to discuss progress and ongoing challenges.

Introduction

  • The survey expands prior work to cover neural approaches across QA, task-oriented dialogue, and social bots, connecting them with traditional methods and training approaches and illustrating progress and challenges through case studies.The authors describe it as what they believe is the first survey of neural approaches to conversational AI; it does not focus on Spoken Language Understanding.Source: Introduction · pp. 5–6

1.1 Who Should Read this Paper?

  • The paper targets NLP and IR audiences while offering an accessible introduction for readers from backgrounds such as machine learning and a resource for students, researchers, and software developers.It aims to provide a unified view and ideas for understanding and creating modern dialogue agents.Source: §1.1 · p. 6

1.2 Dialogue: What Kinds of Problems?

  • Dialogue problems include direct question answering, task completion, and social chat; task-oriented systems use NLU, state tracking, policy, and generation modules, with a trend toward unified end-to-end neural systems.Task-oriented bots commonly use modular systems and external databases, while social chatbots often use unitary response-generation models trained on human conversations.Source: §1.2 · pp. 6–8

1.3 A Unified View: Dialogue as Optimal Decision Making

  • A hierarchical decision-making view treats dialogue as selecting an agent for a subtask and then choosing its lower-level actions; options over MDPs and RL policies provide a framework for this process.Sounding Board, winner of the 2017 Amazon Alexa Prize, and Microsoft XiaoIce use hierarchical dialogue managers with a master process and lower-level skills.Source: §1.3 · p. 8
  • Because real-user RL can be expensive, the paper describes initializing agents with imitation or supervised learning before RL refinement and balancing short- and long-term rewards.XiaoIce is optimized for expected CPS and has more than 230 skills; task-oriented skills may reduce short-term CPS while supporting longer-term user engagement.Source: §1.3 · pp. 8–9

1.4 The Transition of NLP to Neural Approaches

  • Neural methods learn task-specific continuous representations and can be trained end to end, reducing reliance on engineered features and traditional component boundaries.The paper contrasts their paraphrase robustness with weaknesses in execution efficiency and explicit interpretability; symbolic methods are more efficient and interpretable but are difficult to train and sensitive to paraphrase variation.Source: §1.4 · pp. 9–10

Machine Learning Background

This background chapter reviews machine-learning, deep-learning, and reinforcement-learning concepts used in later discussions of conversational AI, framing dialogue as an interactive learning problem and introducing neural methods for representation learning, ranking, and text generation. It also introduces value-function and policy-gradient algorithms and describes exploration as gathering useful transitions while balancing exploration and exploitation.

2.1 Machine Learning Basics

  • Dialogue learning maps to a task of fulfilling user goals, cumulative reward as performance, and dialogues as experience; an agent learns a policy that selects actions from dialogue states to maximize reward.The chapter notes that agents may need to learn through interaction when representative examples of desired behavior are difficult to obtain.Source: §2.1 · pp. 11–12
  • Supervised learning trains a model on input–ground-truth pairs by minimizing a differentiable loss, typically with mini-batch SGD; generalization to unseen inputs remains a central challenge.Neural approaches may help by using deep networks’ representation-learning capacity.Source: §2.1 · pp. 11, 13

2.2.1 Foundations

  • Deep learning replaces hand-engineered text features with representations learned jointly with the classifier, shifting design effort toward DNN architectures for effective representation learning.Classical systems first engineer features and then train a classifier; deep models learn feature generators and classifiers end to end.Source: §2.2.1 · p. 13
  • Neural layers provide different text representations: embeddings map words to dense vectors, CNNs capture local dependencies, and RNNs process word sequences into semantic vectors.Layers can be combined and stacked to capture semantic and contextual information at different abstraction levels.Source: §2.2.1 · pp. 13–14

2.2.2 Two Examples

  • DSSM ranks candidates by cosine similarity between neural representations of an input pair, with the shared semantic space optimized for the task, including through pairwise ranking loss.The framework applies to tasks such as query-document ranking and QA answer selection.Source: §2.2.2 · pp. 15–16
  • Seq2seq generates an output sequence conditioned on an input by encoding it into a context vector and decoding from that vector; gated RNNs can better capture long-term dependencies.Encoder and decoder are trained jointly, and input and output sequences may differ in length.Source: §2.2.2 · p. 16

2.3 Reinforcement Learning

  • Reinforcement learning learns decisions through interaction without supervisory labels, unlike supervised learning, and requires balancing exploration of unknown states with exploiting known high-reward actions.Rewards may be delayed until a dialogue ends, requiring temporal credit assignment; dialogue observations can also reveal only part of the environment state.Source: §2.1 · pp. 12–13, §2.3.1 · p. 17

2.3.1 Foundations

  • The chapter models agent–environment interaction as an MDP with states, actions, transition probabilities, rewards, and a discount factor, and defines policies by their expected long-term reward.It introduces Q-learning and policy-gradient methods as two popular RL algorithm classes without detailing them in this part.Source: §2.3.1 · p. 17, §2.3.2 · p. 17

2.3.2 Basic Algorithms

  • Q-learning learns a parameterized Q-function, using compact representations such as neural networks when tabular state-action values are impractical.DQN uses deep neural networks, including multilayer perceptrons, convolutional networks, and recurrent networks (Tesauro, 1995; Mnih et al., 2015; Hausknecht and Stone, 2015; Li et al., 2015).Source: §2.3.2 · p. 18
  • Q-learning updates parameters from temporal differences, while experience replay and a fixed target network can improve sample efficiency and stabilize learning (Mnih et al., 2015).Replay reuses stored transitions; the target network supplies a fixed Q-function for computing temporal differences during updates.Source: §2.3.2 · p. 18
  • Policy-gradient methods directly optimize a parameterized policy, and REINFORCE estimates its gradient from sampled trajectories.REINFORCE can have high variance because its estimate depends on rewards across the trajectory; actor-critic methods use an estimated value function as a critic to reduce variance (Sutton et al., 1999a; Konda and Tsitsiklis, 1999).Source: §2.3.2 · pp. 18–19

2.3.3 Exploration

  • Exploration balances choosing apparently strong actions with trying novel ones; ϵ-greedy selects the best-looking action with probability 1 − ϵ and a random action with probability ϵ.The text notes that this simple strategy is effective in many problems, though not necessarily optimal.Source: §2.3.3 · p. 19

Question Answering and Machine Reading Comprehension

This section surveys conversational question answering across structured knowledge bases and documents, covering symbolic and neural approaches to KB-QA, multi-step reasoning, conversational QA, and neural text-QA and machine reading comprehension. It also addresses dataset and model design and the challenges of answering questions in dialogue context.

Question Answering and Machine Reading Comprehension

  • Single-turn QA is less studied than multi-turn conversational QA but remains an indispensable building block for dialogue systems.The section also reviews text-QA through MRC models that encode questions and passages and reason over them to generate answers, alongside conversational extensions to MRC.Source: Question Answering and Machine Reading… · p. 20

3.2 Semantic Parsing for KB-QA

  • Symbolic KB-QA maps questions to logical forms and matching KB paths, but paraphrase mismatches and exponentially growing path searches limit its robustness and scalability.The system retrieves end nodes from paths matching a translated query; paraphrases can mismatch KB predicates, while multi-step candidate paths grow exponentially with path length.Source: §3.2 · pp. 21–22

3.3 Embedding-based Methods

  • KB embeddings represent entities and relations as continuous vectors, supporting paraphrase matching and generalization to unseen triples and path queries.Compositions of relation embeddings, such as matrix multiplication in the bilinear model, represent multi-step paths.Source: §3.3 · p. 22

3.4.1 Symbolic Methods

  • PRA learns relational paths through bounded-depth random walks, but its discrete search can yield millions of paths, harming generalization and making reasoning prohibitively expensive.Heuristics, L1 regularization, and clustering paths using relation embeddings are described as ways to reduce the path burden.Source: §3.4.1 · pp. 23–24

3.4.2 Neural Methods

  • Neural KBR methods such as IRN and Neural LP achieve state-of-the-art benchmark results, reasoning over compact semantic memory to handle paraphrases and avoid exhaustive search (Shen et al., 2016, 2017a; Yang et al., 2017a).IRN uses a controller to iteratively attend to differentiable shared memory and decide when to stop; its symbolic query and path interpretations remain difficult to recover.Source: §3.4.2 · pp. 24–25

3.4.3 Reinforcement Learning based Methods

  • RL-based KBR agents traverse knowledge graphs with continuous states, combining neural methods’ paraphrase robustness with symbolic methods’ explicit paths (Xiong et al., 2017; Das et al., 2017b).M-Walk uses Monte Carlo Tree Search to address sparse terminal rewards by generating simulated paths with more positive terminal rewards.Source: §3.4.3 · pp. 25–26

3.5 Conversational KB-QA Agents

  • Conversational KB-QA adds a dialogue manager that tracks context and state, queries the KB using dialogue history, and chooses attribute questions or ranked entity results.Its components include belief tracking, KB lookup, belief-state summarization, and a dialogue policy, which may be programmed or trained (Wu et al., 2015; Wen et al., 2017; Dhingra et al., 2017).Source: §3.5 · pp. 26–27
  • Entropy Minimization Dialogue Management asks for the attribute with maximum entropy over remaining KB entries and is optimal without language-understanding errors, but may request information users do not know (Wu et al., 2015).A movie release ID can uniquely identify a movie but may be unknown to regular users.Source: §3.5 · p. 27
  • KB-InfoBot replaces symbolic KB queries with an induced posterior over entries and, when integrated with RL, improves task success and reward in simulations and with real users.It is trained entirely from user feedback and asks easy-to-answer questions; whether it scales to KBs with millions of entities remains unverified. SQA decomposes complex table questions into simpler related questions, while CSQA contains 200K dialogues requiring KB reasoning and coherent QA turns.Source: §3.5 · pp. 27–28

3.6 Machine Reading for Text-QA

  • SQuAD uses crowdworker questions on Wikipedia and requires answer spans, whereas MS MARCO uses real queries and Web documents, includes unanswerable queries, and generates answers from multiple passages.The paper characterizes MS MARCO as more challenging; TREC’s LiveQA track also evaluates real, fresh user questions answered in real time, unlike the earlier automated QA track’s non-real-time questions.Source: §3.6 · pp. 28–29

3.7 Neural MRC Models

  • Neural MRC typically encodes questions and passages, reasons over vectors to identify an answer, then decodes the answer; BiDAF is a widely used baseline and SAN was highly ranked on SQuAD1.1.The paper describes SAN as the best documented MRC model on the SQuAD1.1 leaderboard as of December 19, 2017.Source: §3.7 · p. 30

3.7.1 Encoding

  • Most MRC encoders combine lexical embeddings, contextual embeddings, and attention to form query-aware passage representations for reasoning.Lexical vectors can include character and linguistic features; contextualization commonly uses BiLSTMs, while BERT supports fine-tuning for MRC and a convolution/self-attention encoder was reported to train an order of magnitude faster than an RNN-based model on GPU clusters.Source: §3.7.1 · pp. 30–31

3.7.2 Reasoning

  • Dynamic multi-step reasoning outperforms single-step and fixed multi-step models on SQuAD and MS MARCO, but requires instability-prone reinforcement learning; SAN combines fixed steps with backpropagation.SAN averages predictions across steps at decoding and uses stochastic dropout over predictions during training, improving robustness and overall accuracy.Source: §3.7.2 · pp. 32–33

3.7.3 Training

  • Neural MRC models are trained end to end with backpropagation and SGD, minimizing the average negative log probabilities assigned to the gold answer-span start and end positions.Source: §3.7.3 · p. 33

3.8 Conversational Text-QA Agents

  • Conversational text-QA must resolve references and pragmatic meaning across turns, challenges that single-turn MRC models do not directly address.Conversational MRC encodes dialogue history and can generate answers beyond passage spans; one approach combines DrQA’s evidence-span selection with PGNet’s free-form generation.Source: §3.8 · pp. 33–34

Task-oriented Dialogue Systems

Task-oriented dialogue systems help users complete goal-directed, often domain-dependent tasks such as hotel reservations or movie-ticket bookings. This chapter focuses on their foundations and algorithms, including recent neural and end-to-end approaches.

Task-oriented Dialogue Systems

  • These systems assist with task completion, such as hotel reservations or movie-ticket bookings, rather than simply answering questions or providing information.They typically have a specific goal and are domain dependent.Source: Task-oriented Dialogue Systems · p. 35
  • Unlike chatbots, task-oriented dialogues typically pursue a specific goal and are domain dependent.Source: Task-oriented Dialogue Systems · p. 35
  • The chapter focuses on foundations and algorithms for raw-text dialogue, discusses industry applications elsewhere, and covers evaluation, neural system components, and end-to-end methods enabled by deep learning and reinforcement learning.Many techniques and discussions can also be adapted to spoken dialogue systems.Source: Task-oriented Dialogue Systems · p. 35

4.1 Overview

The section briefly overviews task-oriented dialogue systems, focusing on aspects that support later discussions and directing readers to Jurafsky and Martin (2018, Chapters 25 & 26) for historical development.

4.1 Overview

  • The overview selects task-oriented dialogue-system aspects to facilitate later discussion and refers readers to Jurafsky and Martin (2018, Chapters 25 & 26) for historical context.Source: §4.1 · p. 35

Slot-filling Dialogues

Slot-filling dialogues collect values for a predefined set of slots so a task, such as booking tickets, can be completed. Their slots are defined for each application domain and can be informable, requestable, or both.

Slot-filling Dialogues

  • These dialogues collect the information needed to complete a task, such as booking tickets, by filling values for predefined slots.The chapter focuses on this practical dialogue class, also called form-filling.Source: Slot-filling Dialogues · p. 35
  • Slot sets are domain-specific and defined by experts; examples include movie, restaurant, and air-ticket booking slots.Movie slots can include film, theater, time, date, ticket price, and number of tickets.Source: Slot-filling Dialogues · p. 35
  • Informable slots constrain the conversation, requestable slots can be queried, and a slot such as movie name may be both.Phone number and ticket price illustrate informable and requestable slots, respectively.Source: Slot-filling Dialogues · pp. 35–36

Dialogue Acts

Dialogue-act theory formalizes conversational interaction as actions that affect the mental states of both participants, with acts such as greeting, requesting, and informing expressed through domain-specific dialogue acts.

Dialogue Acts

  • Dialogue-act theory formalizes the analogy to reinforcement learning: a user utterance is an observation and the system utterance is the agent’s selected action.The interaction is compared with an RL agent acting in an environment.Source: Dialogue Acts · p. 36
  • User and agent utterances are actions that can change both participants’ mental states and the conversation, serving functions such as suggesting, informing, or requesting information.A greet act, for example, lets the system greet the user and start a conversation.Source: Dialogue Acts · p. 36
  • Dialogue acts may take slots or slot-value pairs as arguments; examples include requesting ticket quantity and informing the city as inform(city=‘‘seattle’’), and act sets differ by domain.The examples contrast a movie-booking interaction with the distinct act set of a restaurant domain.Source: Dialogue Acts · p. 36

Dialogue as Optimal Decision Making

Multi-turn task-oriented dialogue is modeled as an RL interaction in which the system tracks state, acts, and receives user responses and turn-level rewards. Reward design balances task success and dialogue length, and may also account for qualities such as diversity and coherence.

Dialogue as Optimal Decision Making

  • The dialogue system acts as the RL agent and the user as its environment, with the agent tracking dialogue state and choosing user-facing or internal actions each turn.In a practical pipeline, NLU maps utterances to dialogue acts, DST tracks state, policy selects actions, and NLG verbalizes user-facing actions.Source: Dialogue as Optimal Decision Making · pp. 36–37
  • A task-oriented reward can give +20 for successful completion, −20 for failure, and −1 per intermediate turn to encourage short dialogues.These values are a simplistic illustration; more sophisticated rewards may measure conversation diversity and coherence.Source: Dialogue as Optimal Decision Making · p. 37

Dialogue Manager

The section reviews traditional dialogue management from a decision-theoretic perspective, focusing on methods for handling uncertainty and the limitations of separately optimized modules. It contrasts these methods with end-to-end neural reinforcement learning, which enables joint optimization and may improve conversation quality.

Dialogue Manager

  • The chapter surveys only a small subset of traditional approaches, directing readers to earlier work and surveys for broader coverage.Its selection is guided by the paper’s decision-theoretic perspective.Source: Dialogue Manager · p. 37
  • Early reinforcement-learning dialogue managers assumed finite, fully observable dialogue states, assumptions often violated by user-utterance ambiguity and NLU errors (Walker, 2000; Singh et al., 2002).Levin et al. (2000) framed conversation as a decision-making problem.Source: Dialogue Manager · p. 37
  • POMDPs model uncertainty by treating user utterances as observations and maintaining a belief state over unobserved dialogue states.Exact POMDP optimization is computationally intractable, prompting approximation techniques and alternative information-state representations; these approaches often require more engineered domain knowledge than neural approaches.Source: Dialogue Manager · pp. 37–38
  • Separately optimized dialogue modules make it difficult to assign credit for poor responses, whereas differentiable neural models with reinforcement learning enable end-to-end optimization.End-to-end optimization may lead to higher conversation quality.Source: Dialogue Manager · p. 38

4.2 Evaluation and User Simulation

Dialogue-system evaluation combines task success, interaction cost, and other quality goals, but whole-system assessment is more difficult than optimizing individual components. The section reviews simulated, human-based, and self-play evaluation, including their trade-offs and limitations.

4.2.1 Evaluation Metrics

  • Whole-system evaluation is challenging, so reinforcement-learning rewards often combine task success, dialogue cost, and other quality metrics.Task success can reward a successful dialogue with +1 and an unsuccessful one with −1 at the final turn; turn-count rewards can be −1 per turn, while coherence, diversity, and personal style remain less-investigated reward dimensions.Source: §4.2.1 · p. 38

4.2.2 Simulation-Based Evaluation

  • Agenda-based simulators begin with a hidden user goal and use a stack agenda to track pending intentions and update simulated behavior as dialogue information emerges.Goals include inform-slots that constrain the dialogue and request-slots whose values are initially unknown; domain constraints can require particular slots, and agenda updates may use learned or expert-defined rules.Source: §4.2.2 · p. 39
  • Model-based simulators use dialogue contexts to generate user actions, but policies overfitted to a simulator may fail with other simulators or real people.The example starts with a randomly generated goal and constraints fixed during the conversation; an LSTM or sequence-to-sequence model can generate the next user utterance from human-human dialogue data, and combining learned and rule-based techniques often works well.Source: §4.2.2 · pp. 39–40

4.2.3 Human-based Evaluation

  • Recruited-subject studies measure task completion and turns but are costly, while actual users bring real tasks and can support larger-scale evaluation at the risk of disrupting service.Small recruited samples can miss statistically significant small differences, limit system comparisons, and make interactive RL impractical except in simpler applications; actual-user metrics are more reliable for real tasks, with A/B testing and counterfactual estimation among available techniques.Source: §4.2.3 · p. 40

4.2.4 Other Evaluation Techniques

  • Dialogue self-play can generate simulated-user conversations at low cost, and hybrid evaluation can validate or fine-tune policies with human users.The approach has shown promising results in negotiation and task-oriented dialogues, and is presented as a way to avoid human evaluation costs and overfitting to untruthful simulated users.Source: §4.2.4 · p. 41

4.3 Natural Language Understanding and Dialogue State Tracking

The section reviews neural methods for interpreting utterances as domains, intents, and slots, and for tracking dialogue state to guide system actions. It describes gains from neural and context-aware NLU alongside neural trackers that represent candidate slot-values or compact belief spans.

4.3.1 Natural Language Understanding

  • NLU typically pipelines domain detection, intent determination, and slot tagging, evaluated with accuracy and F1; its quality affects downstream dialogue-system quality (Li et al., 2017d).Domain and intent are commonly framed as classification from predefined candidates, while slot tagging assigns semantic labels to utterance subsequences.Source: §4.3.1 · p. 41
  • Neural multi-class methods outperform traditional statistical methods for domain and intent classification; RNNs and context-aware RNN/CNN models also show effectiveness, including better results on several benchmarks (Ravuri and Stolcke, 2015; 2016; Lee and Dernoncourt, 2016).The context-aware models use preceding text, which can help classify short utterances whose meaning depends on conversation context.Source: §4.3.1 · p. 41
  • Recurrent neural networks improve slot-tagging accuracy over CRF and SVM baselines, and incorporating prior word information can yield further gains.Slot tagging is treated as sequence classification; an IOB representation assigns semantic tags to each word.Source: §4.3.1 · p. 41
  • Context-aware slot taggers improve on context-free models, while joint, multidomain, and zero-shot NLU approaches target lower labeling needs for new domains.Hori et al. (2015) use role-dependent LSTM layers and Chen et al. (2016b) memory-network attention; Hakkani-Tür et al. (2016) and Liu and Lane (2016) study joint tasks and domains, while Bapna et al. (2017) and Lee and Jha (2019) embed slot descriptions for zero-shot learning.Source: §4.3.1 · pp. 42–43

4.3.2 Dialogue State Tracking

  • Dialogue state records what a user seeks and informs the dialogue policy’s next action; tracking has evolved from expert-created or statistical states toward neural approaches (Henderson, 2015; Henderson et al., 2013; Mrkšić et al., 2015).For slot-filling, state can include informable-slot values, requested slots, and the current search method.Source: §4.3.2 · p. 43
  • Neural Belief Tracker matches contextual utterances to candidate slot-values, while Sequicity’s belief spans support sequence-to-sequence response generation (Mrkšić et al., 2017; Lei et al., 2018).The tracker combines representations of the latest system and user utterances with a slot-value pair, then uses context modeling and semantic decoding to predict each candidate. Belief spans collect informable and requestable slot values, simplifying system design and optimization compared with pipeline approaches.Source: §4.3.2 · pp. 43–44

4.4 Dialogue Policy Learning

The section surveys reinforcement-learning approaches to dialogue-policy optimization, including DQN, efficient exploration, hierarchical and reusable policies, real/simulated-user training, and learned rewards.

4.4.1 Deep RL for Policy Optimization

  • DQN encodes dialogue state, estimates Q-values for dialogue-act/slot actions, and selects the highest-valued action; reinforcement-learning updates train the network, while exploration may require non-greedy choices.The state representation can include recent user and system acts and slots, previously filled slots, turn count, and matching knowledge-base results.Source: §4.4.1 · pp. 44–45
  • Warm starts reduce data needs through expert dialogues, imitation, or Replay Buffer Spiking, which preloads DQN replay with rule-based dialogues and was essential in simulations.Other options include recurrent networks for dialogue histories, graph networks that share parameters across slots (Chen et al., 2018), and policy-gradient methods.Source: §4.4.1 · p. 45

4.4.2 Efficient Exploration and Domain Extension

  • BBQ represents DQN weights with a Gaussian posterior and samples weights for Thompson-style action selection; experiments show more efficient domain-extension exploration than state-of-the-art dialogue baselines.The method uses a Bayesian-by-Backprop variant to quantify uncertainty over intents and slots, encouraging exploration of new ones while avoiding those already learned.Source: §4.4.2 · pp. 45–46

4.4.3 Composite-task Dialogues

  • Composite tasks enlarge state spaces through many interdependent slots and produce sparse, delayed success rewards; hierarchical options use top-level subtask selection and low-level subtask policies.The Subgoal Discovery Network identifies bottleneck states from successful dialogue demonstrations, and hierarchical DQN with discovered subgoals is competitive with expert-designed subgoals.Source: §4.4.3 · p. 46

4.4.4 Multi-domain Dialogues

  • Multi-domain dialogues lack cross-task slot constraints but enlarge the state space, motivating reusable policies: BCM aggregates recommendations, NDQN switches specialist DQNs, and domain features can enable unseen-domain handling (Papangelis et al., 2018a).The feature-based approach can reduce the need for domain knowledge to design the ontology.Source: §4.4.4 · pp. 47–48

4.4.5 Integration of Planning and Learning

  • Human-in-the-loop experiments show Deep Dyna-Q (DDQ) efficiently improves dialogue policies through real-user interaction while combining direct learning, world-model learning, and simulated-user planning.To filter simulated experience, Discriminative Deep Dyna-Q (D3Q) trains a discriminator and uses simulated experiences for policy training only when they resemble real-user experiences (Su et al., 2018b).Source: §4.4.5 · p. 48

4.4.6 Reward Function Learning

  • Because user ratings are costly and hand-designed rewards require domain knowledge, learned reward functions aim to reflect satisfaction, expert demonstrations, interaction quality, or human-like dialogue discrimination.Examples include neural success estimators (Su et al., 2015), Gaussian-process active learning (Su et al., 2016a; 2018a), multi-objective reward learning, and adversarial reward learning that outperforms hand-designed rewards.Source: §4.4.6 · pp. 48–49

4.5 Natural Language Generation

Natural language generation turns a dialogue manager’s communication goal into an utterance, shaping dialogue naturalness and user experience. The section describes the limits of hand-crafted generation and the move toward corpus-based and neural methods, including SC-LSTM’s semantic planning mechanism.

4.5 Natural Language Generation

  • NLG converts a dialogue manager’s selected communication goal into natural language and affects system naturalness and user experience.Source: §4.5 · p. 49
  • Template- and rule-based generators rely on expert-crafted rules and heuristics, making adaptation costly and generation quality dependent on those templates and rules.Machine learning can train some components, but does not remove the cost of writing and maintaining templates and rules.Source: §4.5 · p. 49
  • Corpus-based methods optimize generation from data, usually with supervised learning, while Rieser and Lemon (2010) use reinforcement learning to trade off sentence length and information revealed.Source: §4.5 · p. 49
  • SC-LSTM work expanded through stacked cells and reverse-input reranking, while later studies explored multi-domain learning and hierarchical linguistic patterns (Wen et al., 2016; Su et al., 2018c).Multi-domain learning aims to reduce the data needed to train a neural generator; hierarchical methods leverage linguistic patterns to improve generation.Source: §4.5 · pp. 49–50
  • SC-LSTM adds a sentence-planning component that updates dialogue acts so generated utterances represent the intended meaning.Its reading gate controls which information is retained for future steps.Source: §4.5 · pp. 49–50

4.6 End-to-end Learning

Traditional dialogue systems often optimize components separately, so improvements in individual modules may not improve the whole system. Neural systems can instead jointly optimize differentiable components, with end-to-end learning pursued through supervised and reinforcement learning.

4.6 End-to-end Learning

  • End-to-end learning can optimize a dialogue system’s differentiable components together by back-propagation against metrics of overall system quality.Neural models can also support speech recognition and synthesis in spoken dialogue systems.Source: §4.6 · p. 51

Supervised Learning

  • Supervised end-to-end systems train multiple dialogue components on collected desired responses to maximize prediction accuracy (Bordes et al., 2017; Wen et al., 2017; Yang et al., 2017b; Eric et al., 2017; Madotto et al., 2018; Wu et al., 2018).Bordes et al. (2017) report that memory networks and supervised embedding models outperform standard baselines on simulated dialogue tasks; Madotto et al. (2018) use pointer-network mechanisms in Mem2Seq to incorporate knowledge-base information.Source: §4.6 · p. 51
  • Wen et al. (2017) train components separately because non-differentiable knowledge-base lookups block joint training, while Dhingra et al. (2017) address this with “soft” lookups.Eric et al. (2017) propose an end-to-end trainable Key-Value Retrieval Network with attention-based retrieval over knowledge-base entries.Source: §4.6 · p. 51

Reinforcement Learning

  • Supervised methods may require costly training data and cannot explore policies beyond those of expert response-providing policies, motivating end-to-end reinforcement learning (Zhao and Eskénazi, 2016; Williams and Zweig, 2016; Dhingra et al., 2017; Li et al., 2017d; Braunschweiler and Papangelis, 2018; Strub et al., 2017; Liu and Lane, 2017; Liu et al., 2018a).Source: §4.6 · p. 51
  • Zhao and Eskénazi (2016) use an LSTM-based recurrent DQN to map user utterances to semantic system actions, jointly optimizing policy, language understanding, and state tracking.The model compresses an utterance sequence to infer an internal dialogue state, extending beyond standard supervised learning.Source: §4.6 · p. 51
  • Hybrid Code Networks jointly optimize state tracking and policy with LSTMs, retain business rules and action templates, and learn faster than several end-to-end techniques.The LSTM approach avoids engineering a separate state-tracking step.Source: §4.6 · pp. 51–52
  • Strub et al. (2017) apply policy gradients end-to-end to GuessWhat?!, a visually grounded dialogue game in which the system locates a user-selected image object through yes-no questions.Source: §4.6 · p. 52
  • Liu et al. (2018a) combine supervised pretraining on human-human dialogues, DAgger fine-tuning with human teachers, and reinforcement learning from online user feedback.Source: §4.6 · p. 52

4.7 Further Remarks

The section closes the task-oriented dialogue survey by sampling research directions beyond slot filling, including broader dialogue settings, weaker learning signals, and related evaluation and text-game work.

Beyond Slot-filling Dialogues

  • Beyond slot-filling dialogue, information-seeking and navigation tasks, multimodal verbal and visual inputs, and modality integration present broader challenges (Dhingra et al. (2017); Papangelis et al. (2018b); Bohus et al., 2014; DeVault et al., 2014; de Vries et al., 2017; Zhang et al., 2018a).Task-oriented dialogues may also involve mixed initiative, negotiation, or multiple parties; these settings make turn-taking harder and motivate game-theoretic views beyond single-agent MDPs (Horvitz, 1999; Barlier et al., 2015; Lewis et al., 2017; Bohus and Horvitz, 2009, 2011).Source: §4.7 · p. 52

Weaker Learning Signals

  • Preferential feedback that identifies which of two dialogues is better may be cheaper, easier to obtain, and more reliable than absolute labels or rewards for policy quality.This is proposed as a weaker learning signal that could facilitate dialogue management at scale.Source: §4.7 · p. 52

Related Areas

  • Off-policy evaluation on real human-human conversation corpora remains challenging, but could provide a more robust alternative to user simulation for evaluating and optimizing dialogue systems (Precup et al., 2000; Jiang and Li, 2016; Thomas and Brunskill, 2016; Liu et al., 2018c).The section identifies evaluation as a major research challenge and notes that off-policy evaluation is an active area of work.Source: §4.7 · p. 52
  • Deep reinforcement learning for text games may inform task-oriented dialogue, particularly through methods for natural-language actions and interpretable policies (Narasimhan et al., 2015; He et al., 2016; Cˆot´e et al., 2018; Chen et al., 2017c).Text-game scenarios resemble conversations in many ways, though their scenarios are predefined by the game designer.Source: §4.7 · p. 52

Fully Data-Driven Conversation Models and Social Bots

This section surveys fully data-driven approaches to open-domain conversation, including end-to-end models and reinforcement learning beyond supervised training, alongside data resources, infrastructure, and emerging open benchmarks for studying dialogue. It covers architectural remedies and grounded models, as well as challenges in response appropriateness, rewards, reproducibility, and evaluation.

Fully Data-Driven Conversation Models and Social Bots

  • Data-driven end-to-end models are especially suited to social bots because chitchat needs few external dependencies and open-domain datasets support conversation on many topics.Task-completion systems often depend on APIs such as knowledge-base access, while social bots can more readily be trained end to end.Source: Fully Data-Driven Conversation Models… · p. 53

5.1 End-to-End Conversation Models

  • Early response models cast dialogue turns as translation pairs, but query-response training limited context; RNN-based seq2seq models later exploited longer histories.LSTM models are widely used for seq2seq response generation and can encode textual contexts up to 500 words (Khandelwal et al., 2018).Source: §5.1 · p. 53, §5.1.1 · p. 54, §5.1.2 · p. 54

5.1.2 The HRED Model

  • HRED models dialogue with word- and turn-level recurrent networks, linking each turn to the previous one to capture longer spans and mitigate vanishing gradients.The cited response-generation implementation uses GRUs rather than LSTMs (Serban et al., 2016).Source: §5.1.2 · p. 54

5.1.3 Attention Models

  • Dialogue attention is less effective than in translation because source spans need not align with target spans, while CopyNet improves over RNNs by copying rare or repeated input words.CopyNet selects each output either from a fixed vocabulary or from the source sequence (Gu et al., 2016).Source: §5.1.3 · p. 55, §5.1.4 · p. 55

5.2.1 Response Blandness

  • Neural response models often produce bland replies because maximizing p(T|S) favors generic responses; MMI rescoring, adversarial training, latent variables, and retrieval offer remedies.MMI rescoring can mitigate but not eliminate blandness; VHRED uses a high-dimensional latent variable for response-level properties, while retrieval sacrifices flexibility because its response pool is fixed.Source: §5.2.1 · pp. 56–57

5.2.2 Speaker Consistency

  • Speaker embeddings condition responses on a persona to reduce one-to-many inconsistency, while related approaches use author-specific losses or train personas without their own conversational data.Li et al. (2016b) jointly learns speaker embeddings and uses a specified speaker ID at inference; Luan et al. (2017) ties seq2seq and autoencoder decoders to support personas lacking conversational data.Source: §5.2.2 · pp. 57–58

5.2.3 Word Repetitions

  • Because dialogue does not require one-to-one source-target alignment, translation-style attention often fails to prevent repetitions; decoder self-attention tracks generated content to reduce them.Shao et al. (2017) proposed self-attention to support longer, more coherent responses while mitigating word repetition.Source: §5.2.3 · p. 59

5.3 Grounded Conversation Models

  • Ungrounded models can produce structurally plausible but factually inappropriate replies; grounded systems add persona, textual, visual, or affective context to improve relevance.Ghazvininejad et al. (2018) retrieve relevant facts and encode them separately from dialogue; this lets responses reflect environmental changes without retraining and supports venues absent from conversational training data.Source: §5.2.4 · p. 59, §5.3 · pp. 59–60

5.4 Beyond Supervised Learning

  • Supervised models face a mismatch between human-to-human training data and human-computer use, optimize one response at a time, and lack well-defined reinforcement-learning success metrics for chitchat.These limitations can produce bland responses that fail to sustain long-term engagement.Source: §5.4 · pp. 60–61
  • Li et al. (2016c) trained the first fully end-to-end response generator through a user simulator with policy gradients and rewards for ease of answering, information flow, and meaningfulness.The simulator is a seq2seq model, avoiding the expense of training with thousands or tens of thousands of real-user dialogue turns; the model is initialized with supervised seq2seq parameters.Source: §5.4 · pp. 61–62

5.5 Data

  • Social media provides abundant conversational data, but restricted access and tweet deletions impede public, stable datasets and reproducible response-generation research.Researchers generally create their own fixed subsets; the existing standard-set effort cited here uses a relatively small dataset.Source: §5.5 · p. 62
  • Common end-to-end dialogue datasets include Twitter, Reddit, OpenSubtitles, Ubuntu, and Persona-Chat, which differ in scale, organization, domain, and persona design.Twitter and Reddit are practically unbounded; Ubuntu dialogues focus on operating-system goals, while Persona-Chat was crowdsourced around distinct user personas and contains about 160k utterances.Source: §5.5 · pp. 62–63

5.6 Evaluation

  • Galley et al. (2015) found BLEU and deltaBLEU Spearman’s ρ rose from 0.1 at sentence level to nearly 0.5 over corpora of 100 responses; Lowe et al. (2017) achieved 0.42 sentence-level correlation.The learned metric uses a VHRED variant, but trainable metrics raise risks of overfitting and gaming.Source: §5.6 · p. 63
  • Trainable evaluation metrics can be gamed, while adversarial approaches may be unstable, leaving automatic evaluation for end-to-end dialogue unresolved.A metric optimized directly by a generator can become its adversarial target; stable, reproducible evaluation remains important.Source: §5.6 · pp. 63–64

5.7 Open Benchmarks

  • Open benchmarks include DSTC end-to-end and grounded conversation tasks, ConvAI non-goal-oriented dialogue, NTCIR short-text conversation, and the Alexa Prize’s real-user social-bot competition.The Alexa Prize collects user ratings of conversations, providing indirect supervision.Source: §5.7 · p. 64

Conversational AI in Industry

Commercial conversational AI spans question answering, task-oriented dialogue, customer support, and chatbots. The surveyed systems use different combinations of neural, symbolic, retrieval-based, and engineered components to address their domains.

6.1 Question Answering Systems

  • Industry question-answering systems include web-scale agents such as Bing QA and knowledge-base agents such as Satori QA, while commercial systems’ internals are often not public.Search companies have added multi-turn QA to improve conversational experiences, particularly on mobile; the survey describes commercial examples with publicly described architectures.Source: §6.1 · p. 65

6.1.1 Bing QA

  • Bing QA generates direct answers from retrieved web passages, while its CQU module rewrites context-dependent follow-up queries for search.Web QA must handle billions of noisy, potentially contradictory or obsolete documents, add no more than 10 milliseconds of MRC latency, and adapt answers to devices. Bing ranks documents and passages before MRC extracts an answer span; CQU detects context dependence and rewrites queries to include prior-session information.Source: §6.1.1 · pp. 65–66

6.1.2 Satori QA

  • Satori QA combines neural and symbolic methods over Microsoft’s knowledge graph to answer factual questions.For scalability, robustness, and efficiency, complex questions can be decomposed into simpler questions and the final answer recomposed from their answers (Talmor and Berant, 2018).Source: §6.1.2 · p. 66

6.1.3 Customer Support Agents

  • Customer support agents are multi-turn conversational KB-QA systems that recommend pre-compiled solutions or involve a human when users describe problems.They clarify problems over multiple turns while navigating a knowledge base, and often accept text and voice input.Source: §6.1.3 · pp. 66–67

6.2 Task-oriented Dialogue Systems (Virtual Assistants)

  • Virtual assistants handle tasks through voice, text, or images, and platforms such as Microsoft’s TCP support multi-domain dialogue systems.TCP uses language understanding, state tracking, and a policy; its TaskForm configuration language separates individual task definitions from the overarching policy. LUIS provides natural-language understanding, Azure Bot Service supports building and deploying dialogue systems, and DialogFlow facilitates dialogue modules and Google Assistant actions.Source: §6.2 · p. 68

6.3 Chatbots

  • Commercial chatbots often combine statistical and hand-crafted methods; Replika combines neural generation and retrieval, and Alexa Prize systems include data-driven and modular designs.Replika conditions responses on images and uses persona-based generation (Fedorenko et al., 2017; Mostafazadeh et al., 2017; Li et al., 2016b). The 2017 winning Alexa Prize system, Sounding Board, combined chitchat with miniskills for tasks and topics.Source: §6.3 · pp. 69–71
  • XiaoIce combines a dialogue policy, empathetic computing, and neural generation with retrieval to produce persona-consistent responses and decide when to drive conversation or listen.Its user-experience layer supports simultaneous voice conversations and turn-taking text messages; its data layer stores conversational data, knowledge graphs, and user profiles.Source: §6.3 · pp. 69–70

Conclusions and Research Trends

The survey frames dialogue systems for question answering, task completion, chitchat, and recommendation as optimal decision processes, with neural methods improving performance across tasks and domains. It concludes that commercial systems often combine symbolic and neural methods, while identifying unresolved research challenges in unified modeling, end-to-end dialogue, data, empathy, training, commonsense, and interpretability.

Conclusions and Research Trends

  • Neural approaches improve dialogue agents across tasks, while commercial systems commonly combine symbolic methods and neural models.KB-QA supports conversational knowledge-base queries, while Text-QA uses neural MRC for concise-answer queries. Task-oriented research explores holistic DL/RL optimization, and chatbots increasingly use end-to-end neural generation beyond chitchat, including recommendation.Source: Conclusions and Research Trends · p. 72

Unified modeling framework for dialogues

  • Unified dialogue modeling remains unproven as a development framework, while XiaoIce and McCann et al. (2018) illustrate unified-model efforts.XiaoIce aims to maximize long-run user engagement, measured by expected conversation turns per session; the paper says its design will be presented in future work.Source: Conclusions and Research Trends · p. 72

Fully end-to-end dialogue systems

  • Grounded conversation models are preliminary, and fully data-driven pipelines still face the challenge of enabling API calls.Grounding models in the user's environment is presented as a step toward goal-oriented conversations.Source: Conclusions and Research Trends · p. 73

Dealing with heterogeneous data

  • Heterogeneous dialogue data requires methods that leverage plentiful chitchat data and scarce goal-oriented data, as well as non-conversational sources.Luan et al. (2017) motivates multi-task learning; Zhao et al. (2017) use data augmentation to combine chitchat and task-oriented data, improving robustness to goal-oriented dialogues. Ghazvininejad et al. (2018) is cited on exploiting sources such as Wikipedia.Source: Conclusions and Research Trends · p. 73

Incorporating EQ (or empathy) into dialogue

Scalable training for task-oriented dialogues

  • Task-oriented dialogue agents need scalable updates for changing environments, including adapting to new slots and detecting unseen intents without labeled data or retraining.Lipton et al. (2018) proposed efficient exploration for gradual domain extension; Chen et al. (2016b) proposed zero-shot learning for unseen intents in a new domain.Source: Conclusions and Research Trends · p. 73

Commonsense knowledge

Model interpretability

Related papers

Loading 1809.08267v3…