PaperlayerView exact revision on arXiv

Source-linked AI summary

Challenges in Building Intelligent Open-domain Dialog Systems

Minlie Huang, Xiaoyan Zhu, Jianfeng Gao

arXiv:1905.05709v3cs.CLcs.AI

TL;DR

Open-domain dialog systems still struggle to converse coherently and engagingly across varied topics like humans. This paper reviews neural approaches addressing semantics, consistency, and interactiveness, alongside evaluation metrics and corpora, and identifies these as central challenges for progress.

  • Problem

    Building open-domain dialog systems that converse coherently and engagingly across varied topics like humans remains challenging.

  • Method

    The paper surveys neural approaches to semantics, consistency, and interactiveness, and reviews dialog evaluation metrics and development corpora.

  • Results

    The review organizes open-domain dialog research around semantics, consistency, and interactiveness while covering evaluation metrics and frequently used corpora.

  • Takeaways & Limitations

    The paper highlights knowledge grounding and improved automatic evaluation as important directions for developing more capable open-domain dialog systems.

  • Takeaways & Limitations

    The review is incomplete, and existing methods remain insufficient for modeling psychological personality in dialog systems.

Abstract

from arXiv · show

There is a resurgent interest in developing intelligent open-domain dialog systems due to the availability of large amounts of conversational data and the recent progress on neural approaches to conversational AI. Unlike traditional task-oriented bots, an open-domain dialog system aims to establish long-term connections with users by satisfying the human need for communication, affection, and social belonging. This paper reviews the recent works on neural approaches that are devoted to addressing three challenges in developing such systems: semantics, consistency, and interactiveness. Semantics requires a dialog system to not only understand the content of the dialog but also identify user's social needs during the conversation. Consistency requires the system to demonstrate a consistent personality to win users trust and gain their long-term confidence. Interactiveness refers to the system's ability to generate interpersonal responses to achieve particular social goals such as entertainment, conforming, and task completion. The works we select to present here is based on our unique views and are by no means complete. Nevertheless, we hope that the discussion will inspire new research in developing more intelligent dialog systems.

1 INTRODUCTION

The introduction frames open-domain dialog as a harder, socially oriented alternative to task-oriented systems and identifies semantics, consistency, and interactiveness as central challenges for neural approaches.

  • Three challenges: The paper organizes neural open-domain dialog research around three challenges: semantics, consistency, and interactiveness.These challenges structure the remainder of the paper’s discussion.
  • Open-domain versus task-oriented dialog: Open-domain dialog systems aim to build emotionally connected, long-term relationships with users rather than complete predefined tasks.Their open-ended goals and less-defined rewards make them more difficult to develop than task-oriented systems.
  • Semantics: Semantics requires understanding conversational content and users’ personality, persona, emotion, sentiment, profile, and background.The challenge extends beyond textual meaning because conversations may involve multiple modalities and social information.
  • Consistency: Consistency requires responses and behaviors to remain coherent with the system’s persona and previous dialog history to build user trust and confidence.The introduction highlights persona consistency and coherence across temporal, causal, and logical dependencies.
  • Interactiveness: Interactiveness concerns optimizing multi-turn behaviors and dialog strategies to satisfy social goals such as entertainment, conformity, emotional affection, and belonging.Improving interactiveness requires understanding users’ emotional states and producing interpersonal responses.

2 FRAMEWORKS FOR BUILDING OPEN-DOMAIN DIALOG SYSTEMS

Open-domain dialog systems are typically built around an end-to-end response-generation engine that scores candidate responses from the current input and dialog context. This framework unifies retrieval-based, generation-based, and hybrid approaches while supporting contexts such as conversation history, persona, emotion, and knowledge graphs.

  • Framework: An end-to-end response-generation engine scores candidate responses conditioned on the current user input and dialog context, then selects the highest-scoring candidate.The candidate space is denoted by Ω, while Pθ scores responses and an argmax search selects the best one.
  • Framework: The formulation unifies retrieval-based, generation-based, and hybrid systems, differing mainly in how they construct the candidate-response search space and score candidates.Retrieval systems use responses from pre-collected human conversations and matching or ranking functions; generation systems construct responses directly.
  • Framework: Dialog context enables single-turn or multi-turn modeling and can encode persona, emotion labels, or knowledge graphs for personalized, emotional, or knowledge-aware responses.Context may be empty, consist of preceding utterance-response pairs, or include non-content information.
  • Retrieval-based methods: Neural retrieval models use shallow or deep interaction networks, with deep interactions occurring earlier between query context and candidate response and generally performing better than shallow interactions.Shallow networks encode inputs independently before limited interaction, while deep networks build representations through early query-response interaction.
  • Retrieval versus generation: Retrieval-based methods return responses from human-conversation repositories, whereas generation-based methods produce novel responses but often generate unsafe or low-quality outputs.Retrieval quality depends critically on repository scale, and retrieval cannot produce unseen responses.

3 SEMANTICS

The semantics challenge is producing contentful, interesting, and interpersonal responses grounded in dialog content, user emotion, and relevant real-world knowledge. The reviewed neural approaches improve encoder-decoder components and incorporate external knowledge to address bland, generic responses.

  • Semantics failures produce bland, generic replies, including “I don’t know,” “thank you,” “OK,” or repetitions of the user’s words.
  • Encoder: Encoder-decoder methods improve response semantics by enriching encoded context with longer history, persona, hidden topics, and topic words.Topic-aware models jointly attend to the dialog query and extracted topic words, while topic words can also model topic transitions.
  • Intermediate Representation: More flexible intermediate representations, including latent variables, address dialog’s one-to-many nature while improving interpretability and response-generation control.
  • Decoder and Candidate Ranker: Controllable decoding increases the probability of desirable words, while candidate ranking selects diverse responses using information unavailable or costly during decoding.CopyNet is a notable example that copies desirable infrequent words from the input into the output.
  • Knowledge-grounded dialog: Knowledge-grounded dialog systems focus primarily on selecting appropriate knowledge and injecting it into generated responses.These systems aim to connect entities and topics in user input with real-world facts and background information.

4 CONSISTENCY

Consistency requires dialog systems to maintain coherent persona, style, and context to build user confidence and trust. Existing work addresses persona consistency, stylistic generation, and contextual consistency, but psychological personality and contextual modeling remain limited.

  • Consistency spans persona, style, and context, supporting user confidence and trust in human-like dialog systems.
  • Persona consistency modeling: Persona-consistency methods divide into implicit personalization using persona representations and explicit personalization using predefined user profiles.Implicit methods learn persona features, whereas explicit methods control generation through key-value profiles.
  • Persona consistency modeling: Existing persona models generally fail to capture psychological traits such as extroversion or introversion.
  • Stylistic response generation: Stylistic response generation must disentangle content from style and obtain paired training data with identical content expressed in different styles.
  • Contextual consistency: Contextual consistency remains comparatively underexplored, with early hierarchical context representations and newer formulations based on natural language inference.

5 INTERACTIVENESS

Interactiveness concerns optimizing dialog behavior and strategies for sustained engagement and complex social goals, requiring emotion awareness and effective multi-turn interaction. Despite neural advances, complex emotion expression, social-skill integration, and realistic evaluation remain challenging.

  • 5 INTERACTIVENESS: Interactiveness requires understanding users’ emotions alongside dialog content and optimizing behavior and interaction strategies across multi-turn conversations.The goals include long-term engagement and complex outcomes such as emotional comfort or psychological counseling.
  • Emotion perception and expression: Neural approaches have expanded open-domain emotion perception and expression, replacing earlier rule-based or small-data systems limited to controlled environments.Emotion perception and expression are described as vital to human-like dialog systems.
  • Emotion perception and expression: Complex emotion generation remains difficult because emotion representations can be implicit and unexplainable, while users’ emotion transitions are often not explicitly modeled.Detecting negative emotional change could enable responses such as shifting to more comfortable topics and cheering users up.
  • Social skills: Engaging open-domain conversation requires social skills such as topic planning and dialog policies that manage stalled conversations and active listening.These skills support the eventual establishment of long-term emotional connections with users.
  • Social skills: Existing work incorporates proactive behavior by detecting conversational stalemates and retrieving or suggesting responses connected to the user’s input.One retrieval-based method uses rules to detect stalemates and retrieves responses containing relevant entities.
  • Future research: Future research needs richer human-human conversation data and realistic multi-strategy settings, since current studies often model atomic strategies and evaluate single-turn interactions.Online-forum and psychological-counseling data are identified as valuable, while broader scenario data remain scarce.

6 OPEN-DOMAIN DIALOG EVALUATION

Open-domain dialog evaluation is difficult because conversations are open-ended and often admit many plausible responses. Manual and automatic metrics each have important limitations, motivating evaluation methods that better capture conversational quality and the one-to-many nature of dialogue.

  • Open-domain evaluation is challenging because a single input can have hundreds of distinct but plausible responses.This open-endedness makes it difficult to define and measure response quality.
  • Manual evaluation uses human judges and predefined guidelines, but it is expensive, time-consuming, and not always reproducible.
  • Automatic evaluation commonly uses retrieval metrics such as precision@k, MAP, and nDCG, or generation metrics such as perplexity, BLEU, and distinct-n.These metrics are especially frequently used during early development because manual evaluation is costly.
  • Trainable metrics such as ADEM and RUBER attempt to evaluate responses using learned quality, overlap, and input-response correlation signals.ADEM was claimed to correlate better with human evaluation than BLEU and ROUGE, while RUBER combines referenced and unreferenced components.
  • Existing metrics remain unreliable because ADEM can be fooled by reversing word order, while models often repeat or contradict statements and mishandle specificity, relatedness, and question-asking.
  • Automatic dialog evaluation is unsolved because dataset-trained metrics cannot fully account for appropriate responses that are absent from limited observed references.The one-to-many nature of open-domain dialogue makes automatic metrics inherently questionable.

7 OPEN-DOMAIN DIALOG CORPORA

Open-domain dialog corpora have accelerated neural conversation-generation research, spanning diverse topics, sources, languages, scales, and design features. The surveyed datasets support research on one-to-many generation, personalization, intent and emotion, technical problem solving, and knowledge-grounded conversation.

  • The surveyed corpora differ in topic, source, language, data scale, and design features, reflecting varied requirements for open-domain dialog research.
  • STC contains 219,905 posts and 4,308,211 training responses, enabling one-to-many dialog modeling; ESTC adds six emotion classes with 62.3% classifier accuracy.
  • PersonalDialog supports personalized conversation modeling with 20.83M conversations, 8.47M user profiles, and five personality traits per profile.Its Weibo dialogues contain posts and replies from different users, with profiles including gender, age, location, interest tags, and self-description.
  • DailyDialog provides 13,118 multi-turn dialogs with manual intent and emotion annotations, while Ubuntu Dialog Corpus targets two-party technical problem-solving conversations.DailyDialog averages 7.9 turns per dialog and 14.6 words per turn; Ubuntu conversations end when the technical problem is solved.
  • Knowledge-grounded corpora connect conversations to documents, web pages, Wikipedia topics, or knowledge graphs for grounded response generation and conversational reasoning.Examples include CMU DOG, Wizard of Wikipedia, DSTC7, Topical-Chat, OpenDialKG, DuConv, and DyKgChat.

8 DISCUSSIONS AND FUTURE TRENDS

The paper reviews neural approaches addressing semantics, consistency, and interactiveness in open-domain dialog, while identifying sustained, coherent, and engaging conversation as an unresolved challenge. Future directions emphasize grounding, empathy, personality, and controllable generation.

  • Research Scope: The review centers on neural approaches to semantics, consistency, and interactiveness, alongside evaluation metrics and corpora for open-domain dialog.It also discusses improving automatic evaluation metrics.
  • Future Challenges: Open-domain systems aim for long, free-form social conversations, but sustaining coherent and engaging interaction remains challenging despite recent progress.They differ from early task-oriented assistants designed for short, domain-specific exchanges.
  • Topic and Knowledge Grounding: Knowledge grounding in real-world topics and entities can provide contentful conversations and partially support language understanding in dialog context.The paper identifies knowledge bases as one grounding resource, while noting that existing evidence remains preliminary.
  • Empathetic Computing: Future empathetic machines should perceive users’ changing emotions, produce emotionally influential conversations, and evaluate the emotional impact of their actions.Existing work is described as addressing emotion only through superficial expression.
  • Controllability of Dialog Generation: Controllable generation remains difficult because probabilistic neural models frequently produce repetitive, bland, illogical, or unethical responses.The paper connects controllability with interpretability and robustness and points to hybrid neural-symbolic modeling as a potential direction.
Loading 1905.05709v3…