Source-linked AI summary
Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space
Zhou Fan, Rui Su, Weinan Zhang, Yong Yu
TL;DR
Parameterized action spaces require coordinated discrete action and continuous parameter selection, challenging methods designed for only one action-space type. The paper proposes a hybrid actor-critic architecture with separate sub-actors and a global critic, and instantiates it as H-PPO using PPO. Across four parameterized-action tasks, H-PPO learns stably and outperforms previous parameterized-action reinforcement-learning methods, while broader hierarchical-action evaluation remains future work.
Problem
Parameterized action spaces combine discrete actions with continuous parameters, whereas most reinforcement-learning models are designed for only one action-space type.
Method
The hybrid actor-critic architecture uses parallel sub-actors for action-selection components and a global state-value critic; H-PPO implements it with PPO.
Results
H-PPO achieves stable learning in all four parameterized-action tasks and outperforms previous parameterized-action reinforcement-learning methods.
Takeaways & Limitations
The architecture supports separate training of discrete action and continuous parameter policies while retaining a shared critic, and can be extended to hierarchical action spaces.
Takeaways & Limitations
The extended architecture for general hierarchical action spaces was only briefly presented, and more experiments are needed to evaluate it.
Abstract
from arXiv · showhide
In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-actor networks. While this paper is mainly focused on parameterized action space, the proposed architecture, which we call hybrid actor-critic, can be extended for more general action spaces which has a hierarchical structure. We present an instance of the hybrid actor-critic architecture based on proximal policy optimization (PPO), which we refer to as hybrid proximal policy optimization (H-PPO). Our experiments test H-PPO on a collection of tasks with parameterized action space, where H-PPO demonstrates superior performance over previous methods of parameterized action reinforcement learning.
1 Introduction
Parameterized action spaces combine discrete action selection with continuous parameter selection, creating hierarchical structures that standard discrete- or continuous-action algorithms do not directly address. The paper proposes a hybrid actor-critic architecture and an H-PPO instance for these spaces.
- Action-space structure: Parameterized actions pair a discrete action with a continuous real-valued parameter vector.The agent selects both an action and the parameter associated with that action.
- Applications: Parameterized actions model settings such as choosing Kick in HFO and specifying its power and direction, and they also arise naturally in robotics.Robotics can use higher-level meta-actions controlled by fine-grained parameters.
- Action-space structure: Hierarchical action spaces can include multiple discrete layers, discretized parameters, or taxonomies that reduce extremely large action sets.Parameterized action spaces are presented as a special case with a discrete layer followed by a continuous layer.
- Proposed approach: The proposed hybrid actor-critic architecture uses multiple parallel sub-actors for separate action-selection layers and one global critic to update them.Its flexible structure is intended to extend beyond parameterized spaces to other hierarchical action spaces.
- Proposed approach: H-PPO instantiates the hybrid architecture with proximal policy optimization and outperforms previous methods on a collection of parameterized-action tasks.The paper positions H-PPO as an empirical application of the proposed architecture.
2 Related Work
Parameterized action reinforcement learning must coordinate discrete action selection with continuous parameter selection, unlike methods designed for a single action-space type. Prior approaches discretize parameters, relax discrete choices into continuous outputs, alternate learning stages, or use hierarchical and hybrid value-based designs.
- Challenges: Parameterized action spaces are harder than flat discrete or continuous spaces because they require both hierarchical action selection and mixed discrete-continuous decisions.Most reinforcement learning models are designed for only one of these action-space types.
- Standard RL methods: Q-learning updates the Q-function using the Bellman equation, while DQN approximates that function with a deep neural network for discrete action spaces.Policy-gradient, DPG, DDPG, TRPO, and PPO provide alternative policy-optimization approaches across discrete or continuous settings.
- Parameterized-action methods: Discretizing continuous parameters preserves a discrete formulation but loses fine-grained control and can create an extremely large action space.The passage describes this as a straightforward approach to mixed action spaces.
- Parameterized-action methods: Relaxing discrete action selection into a continuous space can significantly increase action-space complexity.Hausknecht and Stone’s method uses an actor to output values for discrete actions together with continuous parameters.
- Parameterized-action methods: Q-PAMDP alternates Q-learning for discrete action selection with policy-search updates for parameter-selection policies.Other methods use hierarchical conditioning or combine DQN and DDPG, with joint learning reported as potentially unstable.
3 Methodologies
The methodology introduces a hybrid actor-critic architecture for parameterized actions, using parallel actors for action components and a shared state-value critic. H-PPO instantiates this design with PPO and extends it to general hierarchical action spaces.
- Parameterized Action Spaces: Parameterized-action Markov decision processes combine discrete action selection with continuous parameter selection.A complete action is represented by a discrete action and its associated continuous parameter set.
- Hybrid Actor-Critic Architecture: The hybrid architecture uses separate parallel actors for discrete action selection and continuous parameter selection, while a single critic guides both.The discrete actor learns πθd, the continuous actor learns πθc, and the executed action pairs the selected discrete action with its parameter.
- Critic Design: The architecture uses a state-value critic because an action-value critic would be over-parameterized by irrelevant parameters for unselected actions.The state-value function depends only on the state and is used to compute an advantage estimator.
- H-PPO: H-PPO applies PPO separately to the discrete and continuous policies using their respective clipped surrogate objectives.Although both policies jointly determine the complete action, their optimization objectives treat them as separate distributions rather than an explicitly joint distribution.
- General Hierarchical Action Spaces: For general hierarchical action spaces, the architecture assigns one actor to each discrete or continuous action-selection sub-problem and retains one shared critic.Actors may share initial state-encoding layers and are updated as separate policies with methods such as PPO.
4 Experiments
The experiments evaluate H-PPO on four parameterized-action tasks against three baseline algorithms, measuring success rate and mean episode reward during training. H-PPO generally learns stably and outperforms the other methods across the test environments.
- Environments: Four parameterized-action environments are used to evaluate H-PPO, including Catching Point, Moving, and Half Field Football.The experiments also include Chase and Attack; each task has a winning state indicating episode success.
- Experiment Settings: H-PPO is compared with extended DDPG, P-DQN, and DQN using a discretized parameterized action space.The networks use the same hidden-layer sizes, while DQN discretization produces a particularly large action space for Half Field Football.
- Metrics: The evaluation tracks success rate and mean episode reward during training, with Table 1 also reporting success-rate standard deviation.Success rate is the percentage of episodes ending in the winning state.
- Results: H-PPO shows stable learning and achieves a high success rate across all four tasks.The results are reported after the methods undergo the same number of learning iterations.
- Results: H-PPO outperforms the other methods by a large margin in three of four environments, while all three algorithms have similar success rates in Chase and Attack.In Chase and Attack, H-PPO has the lowest variance; overall, it generally achieves faster convergence and lower variance.
- Policy Behavior: In a Half Field Football episode, H-PPO coordinates TURN with an angle, DASH with power and direction, and KICK toward the goal across successive frames.The example illustrates coordinated discrete action selection and continuous parameter selection during an episode.
5 Conclusion and Future Work
The paper introduces a hybrid actor-critic architecture and its PPO-based instance, H-PPO, for parameterized action spaces. H-PPO learns stably across four tasks and outperforms previous parameterized-action reinforcement-learning methods, while broader hierarchical action spaces remain insufficiently tested.
- Conclusion: The hybrid architecture trains discrete-action and continuous-parameter policies in parallel as separate actors with a global critic.H-PPO implements this architecture using PPO.
- Conclusion: H-PPO achieves stable learning on all four parameterized-action tasks and outperforms previous parameterized-action reinforcement-learning methods.
- Future Work: The extension to general hierarchical action spaces is only briefly presented, and more experiments are needed to evaluate its performance there.