Source-linked AI summary
Action Branching Architectures for Deep Reinforcement Learning
Arash Tavakoli, Fabio Pardo, Petar Kormushev
TL;DR
High-dimensional discrete and discretized continuous action spaces make conventional discrete-action reinforcement learning computationally difficult because possible actions grow combinatorially. The paper proposes a shared decision module with one branch per action dimension and instantiates it as BDQ, which scales to challenging tasks and competes with DDPG. The experiments also identify the shared module as important for coordinating distributed action branches.
Problem
Discrete-action reinforcement learning faces combinatorial growth in explicitly represented actions as action dimensionality increases, especially when continuous actions are finely discretized.
Method
The paper distributes action control across network branches while retaining a shared decision module, implementing the design in Branching Dueling Q-Network as a branching Dueling DDQN.
Results
BDQ scales robustly to high-dimensional control tasks, competes with DDPG, and outperforms it on a task with approximately 6.5 × 10^25 discretized action tuples.
Takeaways & Limitations
Action branching enables discrete-action algorithms to be applied to high-dimensional discrete or continuous action spaces while preserving shared coordination across action branches.
Takeaways & Limitations
The experiments focus on deep Q-learning; in simpler tasks, DDPG performs better or on par with BDQ, and prioritized replay remains insufficiently characterized.
Abstract
from arXiv · showhide
Discrete-action algorithms have been central to numerous recent successes of deep reinforcement learning. However, applying these algorithms to high-dimensional action tasks requires tackling the combinatorial increase of the number of possible actions with the number of action dimensions. This problem is further exacerbated for continuous-action tasks that require fine control of actions via discretization. In this paper, we propose a novel neural architecture featuring a shared decision module followed by several network branches, one for each action dimension. This approach achieves a linear increase of the number of network outputs with the number of degrees of freedom by allowing a level of independence for each individual action dimension. To illustrate the approach, we present a novel agent, called Branching Dueling Q-Network (BDQ), as a branching variant of the Dueling Double Deep Q-Network (Dueling DDQN). We evaluate the performance of our agent on a set of challenging continuous control tasks. The empirical results show that the proposed agent scales gracefully to environments with increasing action dimensionality and indicate the significance of the shared decision module in coordination of the distributed action branches. Furthermore, we show that the proposed agent performs competitively against a state-of-the-art continuous control algorithm, Deep Deterministic Policy Gradient (DDPG).
Introduction
The paper addresses the exponential action-space growth that limits discrete-action reinforcement learning in high-dimensional discrete and discretized continuous domains. It proposes action branching with shared coordination and introduces BDQ, which scales to challenging high-dimensional control tasks.
- Problem: Discrete-action reinforcement learning must explicitly represent Q_N d=1 n_d possible actions, producing exponential growth with action dimensionality.Such large action spaces are difficult to explore efficiently and can make discrete-action algorithms intractable.
- Approach: The proposed architecture distributes action control across individual branches while retaining a shared decision module for coordination.The shared module encodes a latent input-state representation, and each branch controls one action dimension.
- Approach: Action branching changes network-output growth from combinatorial to linear as action dimensionality increases.This decomposition is intended for high-dimensional discrete or continuous action spaces using neural network function approximators.
- Evaluation: BDQ is a branching variant of Dueling Double DQN evaluated on complex control problems through fine-grained discretization of continuous actions.The agent is presented as an example of combining action branching with a discrete-action algorithm.
- Evaluation: BDQ scaled robustly to high-dimensional benchmark domains and outperformed DDPG on the most challenging task with approximately 6.5 × 10^25 discretized action tuples.The result concerns a discrete-action problem previously considered intractable for discrete-action algorithms.
Related Work
Related approaches address large discrete action spaces through continuous embeddings, autoregressive prediction, or cooperative multiagent decompositions. Each approach introduces a different structural assumption or learning setup for managing multidimensional actions.
- Large discrete action spaces: Wolpertinger combines DDPG with approximate nearest-neighbor lookup to achieve logarithmic-time selection over embedded discrete actions.The method leverages prior information about discrete actions by embedding them in a continuous space.
- Autoregressive approaches: Autoregressive Q-learning predicts action values sequentially across dimensions but requires manual dimension ordering and can accumulate Q-value noise.The sequential structure becomes more problematic as the number of action dimensions increases.
- Multiagent approaches: Cooperative multiagent approaches assign one agent to each action dimension, with agents observing the global state and sharing a team reward.Independent Q-learning is used so each agent simultaneously learns its own action-value function.
Action Branching Architecture
The action branching architecture permits partially independent optimization across action dimensions while preserving shared state information for coordination. BDQ demonstrates this design by adapting Dueling DDQN to branched action outputs.
- BDQ architecture: Figure 2 shows BDQ evaluating a common state value and factorized action advantages before aggregating them into Q-values for each action dimension.The factorized Q-values are queried to generate a joint-action tuple.
- Action Branching Architecture: Action branching distributes value-function or policy representations across branches while retaining a shared module that encodes the common input state.The shared module addresses challenges associated with naively using independent function approximators.
- Action Branching Architecture: The shared decision module is hypothesized to support coordination among branches while allowing a degree of independence for each action dimension.The architecture targets a substantial reduction in required network outputs for multidimensional action spaces.
- BDQ architecture: BDQ incorporates action branching into Dueling Double Deep Q-Network as a proof-of-concept discrete-action agent.The architecture is presented as an example that could be combined with other discrete-action algorithms.
- Scope: The empirical verification uses deep Q-learning, while the authors describe potential applicability to a broader spectrum of discrete-action algorithms.The broader applicability remains a stated potential rather than the scope of the reported experiments.
Branching Dueling Q-Network
The proposed BDQ adapts DQN extensions to an action-branching architecture for high-dimensional action spaces. Its design combines shared state-value estimation with branch-specific action advantages, aggregation choices, and distributed error handling.
- Action-branching architecture: BDQ adapts DQN and its extensions to an action-branching architecture with multiple action branches.The section describes adaptations for temporal-difference targets, loss aggregation, prioritized replay, and gradient propagation.
- Action-branching architecture: A common state-value estimator is shared across all action branches, generally yielding better performance.This adapts the dueling architecture to action branching by combining one state-value estimate with branch-specific advantages.
- Value aggregation: Each branch’s Q-value combines the common state value with its corresponding sub-action advantage after local mean-advantage subtraction.The paper reports that this aggregation performed better than the naïve and local maximum alternatives, although it does not resolve identifiability.
- Learning targets and loss: The best-performing temporal-difference target replaces a maximum over action branches with a mean operator.The authors compare separate branch targets, a global maximum-based target, and the selected mean-based target.
- Learning targets and loss: The loss uses mean squared temporal-difference error across branches, while prioritized replay aggregates absolute branch errors for each transition.The unified prioritization error is the sum of a transition’s absolute distributed TD errors.
- Evaluation: Figure 3 reports smoothed evaluation reward against training episodes using averages over 3 runs and shaded standard deviations.Evaluations occurred every 50 training episodes for 30 episodes with a greedy policy.
Experiments
Experiments evaluate BDQ on custom reaching tasks and MuJoCo benchmarks with increasing action dimensionality and discretization granularity. BDQ remains effective in high-dimensional settings, while shared branching supports stable learning and simpler-task performance can favor DDPG.
- Custom Reaching Environments: BDQ is evaluated against Dueling DDQN on custom reaching tasks with 3, 4, and 5 action dimensions and different discretization granularities.The tasks are physical manipulation variants simulated with the MuJoCo physics engine.
- Custom Reaching Environments: In the N = 5 reaching task, BDQ converges to good policies robustly across discretization granularity, whereas Dueling DDQN with n = 5 has poor sample efficiency.Dueling DDQN with n = 9 could not be run because explicitly representing 9^5 ≈ 6 × 10^4 actions was computationally expensive.
- Standard Benchmark Environments: BDQ significantly outperforms DDPG on Humanoid-v1, the most challenging benchmark with 17 action dimensions and approximately 6.5×10^25 possible actions for n = 33.This result is reported for a highly combinatorial discrete action space.
- Ablations and Limitations: The shared decision module is significant for coordinating distributed branches, while removing prioritized replay substantially degrades learning stability.The ablation compares BDQ with a shared network module against IDQ without one; the role of prioritized replay remains for future research.
- Standard Benchmark Environments: BDQ achieves similar final performance for n = 17 and n = 33 despite the substantial difference in total possible actions.The comparison indicates efficient learning across the two discretization resolutions.
Experiment Details
The experiments use a common DQN-based training setup, with agent-specific exploration strategies and architecture configurations. BDQ uses Gaussian exploration, while Dueling DDQN uses annealed ϵ-greedy exploration and IDQ removes shared layers.
- Common training setup: Training began after the first 10^3 steps and continued at every subsequent time step without reward-scaling tuning.All agents used each domain’s raw rewards.
- BDQ: BDQ used Adam with learning rate 10^-4, minibatches of 64, discount factor γ = 0.99, and target-network updates every 10^3 steps.Its hidden layers used ReLU activations, with two shared layers containing 512 and 256 units.
- Exploration: DDPG used temporally correlated Ornstein-Uhlenbeck exploration noise, whereas BDQ used fixed Gaussian noise around greedy actions.The paper notes that random exploration can be inefficient in physical environments with momentum.
- Exploration: BDQ sampled actions from a Gaussian distribution centered on greedy actions, using standard deviation 0.2 during training and zero during evaluation.This strategy was selected to encourage lifelong exploration in physical environments.
- Dueling DDQN: Dueling DDQN used average value–advantage aggregation and linearly annealed ϵ-greedy exploration after Gaussian exploration performed moderately worse.Its dueling-stream gradients were rescaled by 1/2 before entering the shared feature module.
- IDQ: IDQ matched BDQ’s layer sizes but used separate networks without shared initial hidden layers, applying dueling architecture independently to each network.IDQ served as a baseline for investigating the significance of shared representations.
- DDPG: DDPG followed the rllab implementation and reported Lillicrap et al. hyperparameters, except that L2 weight decay on Q was omitted.The originally proposed penalty of 10^-2 deteriorated performance.
Conclusion
The paper concludes that action branching distributes policy or value representations across branches while retaining a shared module for implicit centralized coordination. This architecture enables DQN extensions in high-dimensional discrete or continuous action spaces and motivates further investigation.
- Architecture: The proposed architecture distributes policy or value representations across network branches while retaining a shared module for implicit centralized coordination.The paper adapts DQN and several notable extensions to this architecture.
- Implications: Action branching enables discrete-action algorithms to operate in high-dimensional discrete or continuous action spaces previously considered intractable.The conclusion presents this as evidence of the architecture’s effectiveness and potential generality.
- Future work: The reported scaling performance is described as promising enough to motivate further theoretical and empirical investigations.