Source-linked AI summary
Deep Reinforcement Learning in Parameterized Action Space
Matthew Hausknecht, Peter Stone
TL;DR
Deep reinforcement learning had not previously been shown to handle structured continuous action spaces combining discrete action types with continuous parameters. This paper extends DDPG to that setting in simulated RoboCup soccer, achieving reliable learned goal scoring and outperforming the 2012 champion in scoring reliability, while requiring reward engineering and remaining limited to simpler tasks.
Problem
Previous work had not succeeded at using deep neural networks in structured, parameterized continuous action spaces.
Method
The paper extends DDPG to parameterized action spaces and bounds action-space gradients to support stable learning.
Results
The best learned agent scores goals more reliably, though slower, than the hand-coded 2012 RoboCup champion.
Takeaways & Limitations
Deep reinforcement learning can learn RoboCup policies from scratch in parameterized action spaces and demonstrates capability beyond purely continuous action settings.
Takeaways & Limitations
The paper does not address scoring on a goalie or cooperating with a team, and reward engineering is necessary because sparse goal rewards impede exploration.
Abstract
from arXiv · showhide
Recent work has shown that deep neural networks are capable of approximating both value functions and policies in reinforcement learning domains featuring continuous state and action spaces. However, to the best of our knowledge no previous work has succeeded at using deep neural networks in structured (parameterized) continuous action spaces. To fill this gap, this paper focuses on learning within the domain of simulated RoboCup soccer, which features a small set of discrete action types, each of which is parameterized with continuous variables. The best learned agent can score goals more reliably than the 2012 RoboCup champion agent. As such, this paper represents a successful extension of deep reinforcement learning to the class of parameterized action space MDPs.
1 INTRODUCTION
The paper applies deep reinforcement learning to RoboCup HFO, a domain with continuous states and structured actions combining discrete types with continuous parameters. It extends DDPG for this setting and demonstrates learned goal-scoring policies from scratch.
- 1 INTRODUCTION: The paper extends Deep Deterministic Policy Gradients (DDPG) into a parameterized action space.It also documents bounding action-space gradients for stable learning in this domain.
- 1 INTRODUCTION: The agents learn from scratch, using a single reward function to locate and approach the ball, dribble to the goal, and score on an empty goal.The learned policies operate in low-level continuous states and parameterized-continuous actions.
- 1 INTRODUCTION: The best learned agent scores goals more reliably than the hand-coded 2012 RoboCup champion, although it is slower.The champion agent provides a difficult comparison baseline.
- 1 INTRODUCTION: HFO combines a low-level continuous state space with parameterized-continuous actions, requiring discrete action selection followed by continuous parameter specification.This structure differs from a purely continuous action space.
2 HALF FIELD OFFENSE DOMAIN
Half Field Offense abstracts RoboCup soccer into a focused decision-making environment with continuous observations and structured actions. Its reward design supports learning toward ball approach, kicking, and goal scoring, while reward engineering remains necessary because sparse goal rewards impede exploration.
- 2 HALF FIELD OFFENSE DOMAIN: HFO abstracts full RoboCup difficulties and focuses experiments on core decision-making, especially scoring and defending goals.Full games are lengthy, high-variance, and require specialized rule handling.
- 2 HALF FIELD OFFENSE DOMAIN: Each agent receives 58 continuously valued, egocentric state features describing positions, velocities, orientations, stamina, and distances or angles to relevant objects.Relevant objects include the ball, goal, field corners, penalty-box corners, teammates, and opponents.
- 2 HALF FIELD OFFENSE DOMAIN: At each timestep, the agent selects one of four discrete actions—Dash, Turn, Tackle, or Kick—and specifies that action’s continuous parameters.Dash and Kick use power in [0, 100], while directions are bounded in [−180, 180] degrees.
- 2 HALF FIELD OFFENSE DOMAIN: The hand-crafted reward combines movement toward the ball, an initial close-enough-to-kick bonus, and movement of the ball toward the goal.The reward uses changes in distances between the agent, ball, and goal.
- 2 HALF FIELD OFFENSE DOMAIN: Reward engineering is necessary because random behavior is exceedingly unlikely to score even one goal within a reasonable time.The paper identifies better exploration of large state spaces as a future direction.
3 BACKGROUND: DEEP REINFORCEMENT LEARNING
The paper presents deep actor-critic reinforcement learning for continuous actions, using critic gradients to improve the actor and stabilizing updates with target networks and replay memory.
- Actor-Critic Architecture: Actor-critic networks separate action selection from value estimation: the actor outputs actions while the critic estimates their scalar Q-value.The actor maps state s to action a, and the critic evaluates state-action pairs.
- Critic Updates: In continuous action spaces, the actor supplies the next-state action because maximizing the critic over all possible actions is intractable.This produces a critic loss whose target uses a′ = µ(s′|θµ).
- Actor Updates: The critic guides actor improvement through action gradients, avoiding an explicit optimization over the continuous action space.A backward pass through the critic yields ∇aQ, which is backpropagated through the actor to update its parameters.
- Stable Updates: Critic updates assume the actor approximates the optimal policy, while actor updates assume the critic’s improvement directions remain valid in the environment.Errors in the critic can create destructive feedback and cause divergence, motivating stabilizing techniques.
- Stable Updates: Replay memory reduces bias by sampling mini-batches uniformly from stored experiences, while target networks change more slowly than the learned networks.Together, these techniques allow the critic loss and actor update to be expressed more stably.
- Implementation: The implemented networks process 58 state inputs through fully connected layers and produce separate outputs for four discrete actions and six continuous parameters.The architecture uses layers of 1024, 512, 256, and 128 units with ReLU activations.
4 PARAMETERIZED ACTION SPACE ARCHITECTURE
A parameterized action space combines a discrete action choice with continuous parameters. The architecture factors these outputs while training the critic and actor over all action types and parameters.
- Parameterized Action Spaces: A parameterized action space MDP contains discrete actions, each paired with its own set of continuous parameters.Actions are represented as tuples consisting of an action type and its parameter values.
- HFO Architecture: In HFO, the actor factors the action space into outputs for Dash, Turn, Tackle, and Kick plus six continuous parameters.This mirrors the domain’s structured action representation.
- Action Selection: At each timestep, the actor selects the maximally valued discrete action and pairs it with the associated continuous parameters.Thus one network simultaneously chooses the action type and how to parameterize it.
- Training: During training, the critic receives outputs for all four discrete actions and all six parameters rather than an action-specific subset.Actor updates likewise use gradients for every discrete action and continuous parameter.
- Exploration: Exploration samples a random discrete action and its continuous parameters with probability ϵ, which is annealed from 1.0 to 0.1 over 10,000 updates.The associated parameters are sampled uniformly.
5 BOUNDED PARAMETER SPACE LEARNING
Bounded continuous parameters can become unstable during learning unless updates respect their allowed ranges. The paper compares gradient-handling strategies and emphasizes inverted gradients as a general solution.
- The Bounded-Parameter Problem: HFO bounds direction parameters to [−180, 180] and power parameters to [0, 100], but unconstrained updates quickly exceed these limits.The authors observed parameters trending toward astronomically large values after continued updates.
- Gradient-Bounding Methods: Zeroing gradients suppresses updates that would push a parameter farther beyond its upper or lower bound.The method examines each parameter’s critic gradient relative to its current limit.
- Gradient-Bounding Methods: Squashing gradients uses tanh to bound parameter activations before rescaling them into their intended ranges, but saturation can cause problems.Unlike zeroing, it does not require manually modifying gradients.
- Gradient-Bounding Methods: Inverting gradients downscales updates near parameter boundaries and reverses them when parameters exceed their permitted range.If the critic continually recommends increasing a parameter, it can converge to the upper bound without saturation.
- Scope: These three approaches apply to bounded-continuous action spaces beyond HFO or parameterized action spaces, and all are evaluated empirically.The paper frames the methods as general mechanisms for enforcing continuous action bounds.
6 RESULTS
The experiments compare three gradient-bounding strategies for parameterized actions. Only inverting gradients produced robust soccer learning, while squashing saturated and zeroing exhibited overflow or instability.
- 3 million iterations trained two agents independently for each gradient-bounding approach in the parameterized HFO task.The task involved approaching the ball and scoring a goal; each agent required approximately 20,000 episodes.
- Only inverting gradients produced robust learning: both agents reliably approached the ball and scored goals.The four squashing- or zeroing-gradient agents did not reliably approach the ball or score.
- Squashing gradients kept parameters within bounds but quickly saturated, causing agents to repeat one discrete action with extreme parameters.The saturated functions never recovered as critic gradients pushed parameters toward larger or smaller values.
- Zeroing gradients allowed parameters to overflow their bounds and produced instability, potentially because gradients on other parameters indirectly changed the bounded parameter.One observed dash-power value reached 120 despite a maximum of 100.
- One zeroing-gradient agent became unstable, while the other clearly failed to learn; Figure 3 captures these outcomes through Q-values and critic losses.The figure reports astronomically high Q-values and unstably large loss for zeroing, whereas squashing maintained stable Q-values but found little reward.
- Inverting gradients respected action bounds without saturation, with observed dash power reaching 98.8 out of 100, allowing the critic to shape the actor’s policy.The results motivate non-saturating functions that enforce action bounds.
7 SOCCER EVALUATION
The paper evaluates inverting-gradient DDPG agents against the 2012 Helios champion and a SARSA baseline. The learned agents were competitive in scoring reliability, although they generally took longer.
- The Helios comparison used a hand-coded agent that won the 2012 RoboCup-2D world championship as a high-performance baseline.The baseline was independently created by the Helios RoboCup-2D team.
- The SARSA baseline learned from continuous features in a simplified HFO domain with high-level discrete actions and tile-coded states.SARSA is described as model-free, on-policy reinforcement learning.
- DDPG agents scored nearly as reliably as the expert baseline but required more time to do so.Figure 4 shows phases of approaching the ball, kicking toward the goal, and beginning to score around episode 10,000.
8 RELATED WORK
Related work spans parameterized-action reinforcement learning, learned RoboCup behaviors, and deep-learning control. This paper differs by learning a monolithic policy over structured actions in a more integrated setting.
- Earlier RoboCup learning work used genetic programming and sequences of reward functions to develop increasingly capable soccer behaviors.The sequence encouraged approaching the ball, kicking, scoring, and finally winning.
- Masson and Konidaris formulated parameterized-action MDPs using separate policies for discrete action selection and continuous parameter selection.Their approach alternates optimizing discrete action selection and continuous parameter selection with Q-learning.
- Masson and Konidaris studied a simplified, partly hand-crafted RoboCup abstraction, making direct difficulty comparisons with this paper’s task difficult.Their domain began with the agent and ball co-located and used only parameterized kick actions, though they also examined scoring on a keeper.
- The Brainstormers integrated neural networks across many soccer decisions, but their pre-deep-reinforcement-learning work required more constrained training environments for individual skills.This paper instead learns approaching, kicking, and scoring within one monolithic policy.
- Deep learning had been applied to discrete and continuous control and robot manipulation, while this work targets action spaces with latent structure and continuous parameters.The paper positions parameterized-continuous actions as distinct from purely continuous action spaces.
9 FUTURE WORK
Future work extends evaluation to harder defended scoring and to multi-agent RoboCup settings. The paper also identifies critic state-gradients as a possible resource for model-based control.
- Scoring against a goalie remains a harder task left for future work.
- Future RoboCup research could study ad hoc teamwork and true multi-agent collaboration among deep reinforcement learning agents.Prior multi-agent learning solutions may translate to these settings.
- Gradients of the critic with respect to state inputs could indicate improvement directions that a forward model might exploit for favorable transitions.The paper connects this possibility to developments showing that detailed next-state models are feasible.
10 CONCLUSION
The paper demonstrates deep reinforcement learning from scratch for RoboCup goal-scoring in a parameterized action space. It extends DDPG with bounded action-space gradients and identifies important boundaries for the demonstrated setting.
- The agent learns from scratch to approach the ball, kick toward goal, and score using deep reinforcement learning.
- The best learned agent scores goals more reliably than a hand-coded expert policy.
- The demonstrated scope excludes scoring against a goalie and cooperation with a team.
- The method extends DDPG with an approach for bounding action-space gradients suggested by the critic.