Source-linked AI summary
Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, Kam-Fai Wong
TL;DR
Composite task-completion dialogue requires coordinating multiple subtasks under cross-subtask constraints, a challenge for flat reinforcement learning. The paper formulates this problem with options over MDPs and proposes a hierarchical deep reinforcement learning dialogue manager. Experiments with simulated and real users show that the hierarchical agent significantly outperforms flat RL and rule-based agents while improving dialogue coherence.
Problem
Composite task-completion dialogues require multiple subtasks to be completed collectively, while flat reinforcement learning faces sparse rewards, large state-action spaces, and incoherent subtask switching.
Method
The paper uses options over MDPs and a hierarchical dialogue manager with top-level subtask selection, low-level primitive-action selection, and global state tracking.
Results
The hierarchical RL agent significantly outperforms flat RL and rule-based agents on simulated and real users, while its hierarchical structure improves dialogue-flow coherence.
Takeaways & Limitations
Hierarchical policy structure and intrinsic rewards support more efficient exploration and less frequent switching between subtasks.
Takeaways & Limitations
The approach uses a two-level hierarchy, while more complex tasks may require deeper hierarchies, and designing task hierarchies requires substantial domain knowledge and time.
Abstract
from arXiv · showhide
Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to collectively complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of options over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.
1 Introduction
The paper targets composite task-completion dialogues, where multiple subtasks and cross-subtask constraints make flat reinforcement learning difficult. It proposes a hierarchical deep reinforcement learning dialogue manager and validates it on simulated and real users.
- Problem: Composite tasks require multiple subtasks to be fulfilled collectively while satisfying cross-subtask slot constraints.Travel planning may require coordinating flights, hotels, and car rentals, including timing and ticket-count constraints.
- Challenges: Flat reinforcement learning faces reward sparsity, larger state-action spaces, longer dialogues, and frequent subtask switching.The resulting incoherent conversations are associated with poor user experience and dialogue-session failure.
- Approach: The proposed dialogue manager uses a top-level policy for selecting subtasks, a low-level policy for primitive actions, and a global state tracker for cross-subtask constraints.The formulation uses options over MDPs and combines deep reinforcement learning with hierarchical task decomposition.
- Approach: Intrinsic rewards encourage completing a subtask before switching, making exploration more efficient and reducing subtask switching.The internal critic estimates how likely a subtask is completed from the global state tracker’s current state.
- Contributions: The work presents itself as the first effort to develop a composite task-completion dialogue agent.Its contributions include formulating the problem with options over MDPs and proposing hierarchical deep reinforcement learning.
- Evaluation: The approach is evaluated for travel planning with simulated and real users.The paper explicitly reports validation on both user types.
2 Related Work
Prior task-completion dialogue research largely addresses single-domain settings, while multi-domain work typically assigns domains to separate agents. The paper distinguishes composite-domain dialogues by their collective subtask-completion requirement.
- Single-domain dialogue: Earlier reinforcement-learning dialogue systems primarily focus on single-domain task completion.The cited work includes both conventional and deep reinforcement-learning approaches.
- Multi-domain dialogue: Multi-domain dialogue systems commonly handle each domain with a separate agent and focus on transfer learning or domain selection.These dialogues normally involve one domain during a conversation rather than collectively completing several subtasks.
- Composite-domain dialogue: Composite-domain dialogue differs because multiple subtasks must be completed collectively for the overall task to succeed.This distinction motivates a different technical treatment from ordinary multi-domain dialogue.
3 Dialogue Policy Learning
The paper models composite task completion as hierarchical decision making: a top-level policy selects subtasks, while a low-level policy executes primitive actions under global state tracking. It trains both policies with deep Q-learning, using intrinsic rewards for subtask completion and extrinsic rewards for overall task progress.
- Dialogue manager architecture: The dialogue manager combines language understanding, state tracking, policy selection, and natural-language generation.The implementation uses an LSTM-based language understanding module and a model-based natural language generator.
- Options over MDPs: Composite task completion has a natural hierarchy: the top level selects subtasks, and the low level completes each selected subtask through primitive actions.Options represent multi-step actions whose intra-option policies execute sequences of primitive actions until termination.
- Hierarchical policy learning: The hierarchical learner uses a top-level policy πg for subtasks and a shared low-level policy πa,g for primitive actions conditioned on the current subtask.The subtask remains a constant input to the low-level policy until a terminal state is reached.
- Hierarchical policy learning: Global state tracking preserves information across subtasks, supporting satisfaction of cross-subtask constraints during dialogue.The state contains global information for all subtasks, while the internal critic indicates whether the current subtask has been solved.
- Reward design: The low-level policy optimizes cumulative intrinsic rewards, whereas the top-level policy optimizes cumulative extrinsic rewards when new subtasks begin.The intrinsic reward is provided by the internal critic; the extrinsic reward is received from the environment at subtask starts.
- Optimization: Both policies can be learned with deep Q-learning, using separate value functions, replay buffers, target networks, and stochastic-gradient updates.The top-level update spans the number of low-level steps needed to accomplish a subtask and then considers the next subtask.
4 Experiments and Results
Experiments evaluate the hierarchical reinforcement-learning agent on travel planning with simulated and real users, against handcrafted and flat-RL baselines. Across simulated user types, HRL produced more robust policies, learned faster, and required fewer turns; with real users, it also achieved higher success and ratings than flat RL.
- Experimental setup: The travel-planning evaluation used simulated users derived from human-human multi-domain dialogue data, augmented with inter-subtask constraints and user preferences.The study also evaluated agents with human users.
- Experimental setup: HRL was benchmarked against a handcrafted Rule agent, an exhaustive Rule+ agent, and a flat DQN agent trained with extrinsic rewards only.The comparison used success rate, average rewards, and average turns per dialogue session.
- Simulated-user results: Across all simulated user types, HRL outperformed the rule-based and flat-RL agents on success rate and needed fewer turns than those baselines.Success rate measures the fraction of dialogues completed successfully within the maximum turns.
- Simulated-user results: HRL significantly outperformed flat RL, while its hierarchical structure and intrinsic rewards helped address sparse rewards, long horizons, and larger search spaces.For types B and C, HRL performance did not drop much despite users revising slots; flat RL showed a large performance drop as task complexity increased.
- Simulated-user results: HRL learned faster than flat RL and reached the same performance level with fewer simulation examples, demonstrating greater sample efficiency on complex tasks.The paper connects this result to reduced sample complexity from hierarchical dialogue policies.
- Simulated-user results: Rule+ and flat RL had comparable success rates, but Rule+ required more turns, whereas flat RL achieved similar success with fewer turns.The authors conclude that HRL outperformed all agents in success rate across the simulated user types.
- Human evaluation: With real users, HRL consistently achieved higher success rates and user ratings than flat RL, while switching among subtasks much less frequently.The human evaluation collected 225 dialogue sessions from 12 users.
5 Discussion and Conclusions
The paper studies composite task-completion dialogues and finds that hierarchical reinforcement learning outperforms flat and rule-based agents while improving dialogue coherence. It also identifies deeper hierarchies and automated hierarchy learning as future directions.
- Hierarchical reinforcement learning significantly outperforms flat reinforcement learning and rule-based agents on composite task-completion dialogues.
- The hierarchical structure improves the coherence of the dialogue flow.
- The approach shows strong adaptation ability to tailor dialogue policies to different user types.
- The current agent uses two policy levels, while more complex tasks may require deeper hierarchies.
- Designing task hierarchies requires substantial domain knowledge and is time-consuming, motivating research on automatically learning hierarchies.
A User Simulator
The user simulator first generates a feasible goal, then produces user actions using an agenda-based state representation that tracks constraints and requests. It can also model noisy communication from the NLU component.
- The simulator begins by generating a feasible user goal containing request slots and inform slots.
- Inform slots represent known slot-value pairs, with multiple values treated as soft constraints and single values as hard constraints.
- The user state is represented by an agenda and a goal consisting of constraints and requests.
- At each time step, the simulator generates the next user action from its current state and the previous agent action, then updates its state.
- When no NLU module is used, an error model simulates NLU noise and noisy communication between user and agent.
B Algorithms
The training algorithm uses separate replay buffers and Q-networks for the meta-controller and controller. It alternates subtask selection with primitive action selection, combining intrinsic and extrinsic rewards during learning.
- The algorithm initializes separate experience replay buffers D1 and D2 for the meta-controller and controller.
- It initializes Q1 and Q2 networks with random weights and loads a dialogue simulator with its knowledge base.
- Training proceeds while the dialogue state is nonterminal, using exploration probabilities for both subtask and action selection.
- The meta-controller selects a subtask g, then the controller selects actions conditioned on the selected subtask until it is achieved or the dialogue terminates.
- The controller obtains intrinsic rewards from an internal critic and samples minibatches from D1 to update Q1.
- The meta-controller stores transitions with extrinsic rewards in D2 and updates Q2 using sampled transitions.