Source-linked AI summary
Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space
Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, Han Liu
TL;DR
Existing DRL methods typically handle either discrete or continuous action spaces, leaving hybrid actions that combine both types to approximation or relaxation. The paper introduces P-DQN, which directly integrates DQN-style discrete selection with DDPG-style continuous parameter handling. Across several environments, empirical comparisons indicate that P-DQN is efficient and effective, and more efficient and robust than a DDPG-based relaxation method.
Problem
Most existing DRL frameworks handle either discrete or continuous action spaces, while hybrid spaces require approximation by discretization or relaxation into a continuous set.
Method
P-DQN directly handles hybrid actions by extending DQN with a deterministic function for continuous parameters and an action-value function over finite hybrid actions.
Results
Empirical experiments across several environments indicate that P-DQN is more efficient and robust than a method that relaxes the hybrid space and applies DDPG.
Takeaways & Limitations
P-DQN provides a hybrid-action DRL framework that avoids approximation or relaxation while combining discrete-action and continuous-action methods.
Takeaways & Limitations
Discretizing the continuous components can lose their natural structure and may require a huge number of discrete actions when they form Euclidean regions.
Abstract
from arXiv · showhide
Most existing deep reinforcement learning (DRL) frameworks consider either discrete action space or continuous action space solely. Motivated by applications in computer games, we consider the scenario with discrete-continuous hybrid action space. To handle hybrid action space, previous works either approximate the hybrid space by discretization, or relax it into a continuous set. In this paper, we propose a parametrized deep Q-network (P- DQN) framework for the hybrid action space without approximation or relaxation. Our algorithm combines the spirits of both DQN (dealing with discrete action space) and DDPG (dealing with continuous action space) by seamlessly integrating them. Empirical results on a simulation example, scoring a goal in simulated RoboCup soccer and the solo mode in game King of Glory (KOG) validate the efficiency and effectiveness of our method.
1 Introduction
Existing DRL methods generally assume either discrete or continuous actions, whereas this paper addresses hierarchical actions combining a discrete choice with a continuous parameter. It proposes P-DQN to operate directly on this hybrid space and reports greater efficiency and robustness than a continuous relaxation baseline.
- Motivation: Most existing DRL methods require action spaces to be either finite and discrete or continuous.Examples include DQN for discrete spaces and DPG/DDPG for continuous spaces.
- Problem setting: Hybrid actions first select a high-level action k from [K], then select an associated continuous parameter x_k from X_k.This hierarchical structure defines the discrete-continuous action space considered in the paper.
- Existing approaches: Discretizing each continuous parameter can lose its natural structure and require a huge number of discrete actions for Euclidean regions.The paper identifies this as one straightforward way to apply existing DRL approaches, but notes its approximation burden.
- Existing approaches: Relaxing the hybrid space into a continuous set is another existing approach, exemplified by applying DDPG to an approximate space.Hausknecht and Stone’s formulation uses continuous variables to select the discrete action.
- Proposed method: P-DQN directly handles the discrete-continuous hybrid space without approximation or relaxation by combining a deterministic parameter function with a hybrid action-value function.The method extends DQN while using a deterministic policy for continuous action parameters.
- Empirical evaluation: Empirical study indicates that P-DQN is more efficient and robust than Hausknecht and Stone’s method, which relaxes the hybrid space and applies DDPG.The reported comparison concerns experiments across several environments.
2 Background
Reinforcement learning methods differ in how they represent and optimize actions, with DQN-style value methods suited to finite spaces and policy-gradient methods suited to continuous spaces. Continuous-action value maximization is generally computationally difficult, motivating deterministic policy-gradient approaches and structured-action methods for hybrid spaces.
- An MDP specifies states, actions, transitions, initial-state distribution, discounting, and rewards, with agents seeking policies that maximize expected discounted return.
- Value-based methods estimate Q* and derive a greedy policy, whereas policy-based methods directly optimize the expected-return objective J(π).
- DQN approximates Q* with a neural network when the state space is too large for tabular storage, and several extensions improve its training or estimation.
- Continuous Action Space: For continuous actions, maximizing a neural-network Q-function is generally intractable because the network is nonconvex in the action and global optimization is NP-hard in the worst case.
- Continuous Action Space: Continuous-action methods address this challenge through analytic maximization under quadratic advantage parameterization or deterministic policy gradients implemented by DPG and DDPG.
- Hybrid Action Space: Structured-action work combines finite action choices with continuous parameters, using relaxed continuous representations or separate on-policy updates for discrete and continuous components.
3 Parametrized Deep Q-Networks (P-DQN)
P-DQN handles parametrized hybrid actions by learning continuous parameters for each discrete action and selecting the discrete type through explicit Q-value maximization. Its Bellman formulation reduces the hybrid backup to a discrete maximization once parameter-producing policies are available, while training uses DQN-like value updates and deterministic-policy optimization.
- Action Representation: P-DQN represents each action as a discrete type k paired with a continuous parameter x_k, and defines Q(s,k,x_k) over this hybrid space.
- Bellman Equation: For each discrete action, P-DQN computes an approximately maximizing continuous parameter x*_k and then selects the largest resulting Q value across discrete actions.
- Bellman Equation: The method replaces continuous optimization in the Bellman backup with deterministic policy networks x_k(s;θ), making the resulting equation resemble a classical discrete-action Bellman equation.
- Optimization: P-DQN trains its value network with a DQN-like least-squares Bellman loss and updates the deterministic policy network using stochastic gradients on a two-timescale schedule.
- Training Procedure: The experience-replay algorithm uses epsilon-greedy exploration over parameterized actions, stores transitions, samples minibatches, and supports asynchronous distributed training.
- Comparison: Unlike relaxed DDPG-style methods, P-DQN selects discrete actions directly by Q maximization and is off-policy; unlike prior separate-update methods, it can use human players’ data.
- Training Procedure: When n>1, n-step P-DQN is no longer off-policy, although the n-step bootstrap can improve convergence speed for delayed-reward or long-episode problems.
4 Experiments
The experiments evaluate P-DQN on a simulation task, simulated RoboCup soccer, and King of Glory, comparing it with continuous relaxation and discrete approximation baselines. Across these settings, P-DQN is reported to converge quickly and stably while learning effective policies.
- Experimental setup: P-DQN is evaluated in a simulation example, simulated RoboCup soccer, and King of Glory’s solo mode, with comparisons against Hausknecht and Stone [2016] and DQN where applicable.The experiments use fair-condition comparisons across the three scenarios.
- Simulation example: P-DQN converges in less than 150k iterations in the simulation example, with test mean episode reward measured over 100 trials without ϵ-greedy exploration.The figure distinguishes noisy training curves from their running averages.
- HFO: In HFO, P-DQN first learns to approach and kick the ball, then learns to score as quickly as possible after 250k iterations.The episode-length increase during early training is attributed to episodes ending when the ball is not kicked within 100 frames.
- Simulation example: P-DQN converges much faster and more stably than the precedent method in the simulation example, while DQN reaches a sub-optimal solution with high variance from direction discretization.Performance is evaluated using mean reward, mean goal percentage, and mean episode length from 100-trial averages.
- HFO: P-DQN scores more accurately and quickly than DDPG with more stable performance, requiring about 1 hour on two Intel Xeon CPUs versus three days for DDPG training on a Titan-X GPU.The HFO comparison uses independently trained P-DQN agents and baseline results from Hausknecht and Stone [2016].
- King of Glory: In King of Glory, P-DQN learns the value and policy networks much faster than Hausknecht and Stone [2016], while episode rewards increase during training and testing.Game length first rises as player strengths approach parity, then falls as the learned agent defeats the opponent more easily.
5 Conclusion
The conclusion presents P-DQN as a hybrid-action method that extends DQN with a deterministic policy for continuous action parameters. Empirical comparisons across several experiments demonstrate its reported efficiency and effectiveness.
- Conclusion: P-DQN extends classical DQN with a deterministic policy for the continuous part of discrete-continuous hybrid actions.It operates on the hybrid action space without approximating it by a discrete set or relaxing it into a continuous set.
- Conclusion: Several empirical experiments with comparisons against other baselines demonstrate the efficiency and effectiveness of P-DQN.
A.1 More information on King of Glory
King of Glory is a MOBA and RTS-related game in which heroes fight to destroy the opposing base. The paper studies its one-versus-one solo mode, where a single hero makes rapid decisions involving movement, attacks, and purchases.
- Game setting: King of Glory is a MOBA game with two opposing teams whose objective is to destroy the enemy base.Heroes can move, kill, perform skills, and purchase equipment, while computer-controlled units advance along three lanes.
- Solo mode: The experiments focus on solo mode, a one-versus-one setting in which both players control the hero Lu Ban against the game’s internal AI.Only the middle lane is active in solo games.
- Solo mode: A typical solo game lasts about 10 to 20 minutes and requires instantaneous decisions across attack, movement, and purchasing actions.The paper identifies the large state space as one of the mode’s main reinforcement-learning difficulties.
A.2 Parameter setting
The experiments use distinct training configurations for simulation, HFO, and KOG, with KOG additionally using frame skipping and constrained exploration.
- Simulation: Simulation uses B = 32, a 10k replay memory, and networks sized 64-32 and 64-32-32.The exploration parameter ϵ is annealed from 1 to 0.1 over the first 30k iterations.
- KOG: KOG uses 256-128-64 hidden-layer networks, a 256-128-64-64 Q network, and frame skipping with actions taken every 0.2 second.Frame skipping reduces the action frequency to every 3 frames.
- KOG: Figure 6 depicts the MOBA map, the solo KOG map with only the middle lane active, and a screenshot distinguishing algorithm-controlled and computer-controlled units.The full map has three lanes and three towers per lane for each side.
- KOG: KOG training uses ϵ = 0.255, assigns probabilities to sampled action types, and replaces infeasible sampled actions with the greedy feasible policy.The effective exploration rate is therefore less than ϵ.