Source-linked AI summary
Reinforcement Learning with Parameterized Actions
Warwick Masson, Pravesh Ranchod, George Konidaris
TL;DR
Parameterized action MDPs require choosing both a discrete action and its continuous parameters. The paper introduces Q-PAMDP, an alternating model-free method, proves local-optimum convergence under conditions, and finds it outperforms comparison methods in two domains.
Problem
Reinforcement learning must represent action types and fine-grained action choices when actions are discrete with continuous parameters.
Method
Q-PAMDP alternates learning parameter-selection policies with Q-learning for discrete action-selection policies in parameterized action MDPs.
Results
Q-PAMDP(1) and Q-PAMDP(∞) outperformed eNAC and fixed parameter SARSA in the goal-scoring and Platform domains.
Takeaways & Limitations
Parameterized actions support simple representations of discontinuous policies while retaining continuous action adaptability and distinct action kinds.
Takeaways & Limitations
Q-PAMDP(∞) has weaker convergence properties because it requires a globally convergent parameter-update method.
Abstract
from arXiv · showhide
We introduce a model-free algorithm for learning in Markov decision processes with parameterized actions-discrete actions with continuous parameters. At each step the agent must select both which action to use and which parameters to use with that action. We introduce the Q-PAMDP algorithm for learning in these domains, show that it converges to a local optimum, and compare it to direct policy search in the goal-scoring and Platform domains.
1 Introduction
Parameterized actions combine discrete action kinds with continuous parameters, addressing limitations of purely discrete or purely continuous action spaces. The paper introduces Q-PAMDP for learning action-selection policies and reports stronger empirical performance than direct policy search and fixed parameter SARSA.
- Discrete action spaces preserve distinct action kinds, whereas continuous spaces allow fine state-dependent tuning but require all actions to share one vector representation.
- Parameterized actions attach a real-valued vector to each discrete action, so agents choose both the action type and its execution parameters.The paper motivates this structure with actions such as kicking, passing, or running, each parameterized differently.
- Q-PAMDP alternates learning action-selection and parameter-selection policies for predefined parameterized actions.
- The authors show Q-PAMDP converges to a local optimum and compare it with direct policy search in goal-scoring and Platform domains.
- Q-PAMDP outperformed direct policy search and fixed parameter SARSA in the reported experiments.
2 Background
The background frames reinforcement learning as optimizing discounted return in an MDP. It contrasts model-free Q-learning with the difficulty of maximizing Q over continuous actions and motivates direct policy search.
- An MDP specifies states, actions, transition probabilities, rewards, and a discount factor, with the goal of maximizing expected discounted return.
- The value function measures expected discounted return from a state, while the action-value function measures return after taking an action and following a policy.
- Q-learning supports model-free control by learning action values and selecting actions that maximize Qπ(s, a).
- In continuous action spaces, selecting the Q-maximizing action requires finding a global maximum over a continuous domain.
- Direct policy search instead optimizes parameters θ of a policy class against an objective function J(θ).
Parameterized Tasks
Parameterized tasks define episodes through a fixed task vector and learn policies conditioned on task variation. The paper notes that such task policies can also serve as parameterized actions.
- A parameterized task is defined by a task vector τ supplied at the start of each episode and fixed throughout that episode.
- Prior work learned task-dependent policy or motor-primitive parameters and mappings from task parameters to policy parameters.
- A learned parameterized task can be reused as a parameterized action, such as kick-to(τ) for kicking a ball to position τ.
3 Parameterized Action MDPs
PAMDPs use continuous states with finite discrete actions, each having its own continuous parameter space. The paper selects action types and parameters in two tiers, then alternates parameter-policy and discrete-action-value updates.
- 3 Parameterized Action MDPs: A PAMDP has continuous state space S ⊆ R^n, finite discrete actions Ad, and an action-specific continuous parameter space Xa ⊆ R^ma.
- 3 Parameterized Action MDPs: Each complete action is a tuple (a, x), combining a discrete action with parameters drawn from that action’s parameter set.
- 3 Parameterized Action MDPs: The policy uses two tiers: πd(a|s) selects the discrete action, then πa(x|s) selects its parameters.
- 3 Parameterized Action MDPs: Direct policy search optimizes the expected-return objective J with respect to the parameter-policy parameters θ and action-policy parameters ω.
- 3 Parameterized Action MDPs: Q-PAMDP alternates parameter-policy updates with Q-learning on a corresponding discrete-action MDP induced by the fixed parameter policy.
- 3 Parameterized Action MDPs: Q-PAMDP(1) performs one θ update before relearning ω to convergence, while Q-PAMDP(∞) uses repeated parameter updates before Q-learning.
4 Theoretical Results
The theoretical results establish local—and under stronger conditions global—convergence for Q-PAMDP variants, while bounding Q-learning updates under continuity and bounded parameter updates.
- Q-PAMDP(1): If W(θ) is continuous and P-UPDATE converges globally or locally, Q-PAMDP(1) reaches the corresponding global or local optimum.The result depends on the continuity of W and the convergence behavior of P-UPDATE.
- Q-PAMDP(1): Q-PAMDP(1) converges to a local optimum for J when it locally optimizes H and maintains ω = W(θ).If θ converges globally for H, Q-PAMDP(1) converges globally for J.
- Bounded updates: Bounded θ updates and continuity of W allow the learning rate α to keep successive ω updates within any chosen ϵ.The required bound follows because sufficiently small α makes ||ωt+1 − ωt||2 < ϵ.
- Policy-gradient updates: When P-UPDATE is a policy-gradient step, Q-PAMDP converges locally and Q-LEARN requires a fixed number of updates under the stated conditions.The gradient of H equals the gradient of J when ω = W(θ), enabling gradient-based policy search for θ.
- Q-PAMDP(∞): Q-PAMDP(∞) converges to a local optimum through direct alternating optimization, but requires a globally convergent P-UPDATE and therefore has weaker convergence properties than Q-PAMDP(1).Its full optimization steps may nevertheless bypass nearby local optima.
5 Experiments
Experiments evaluate Q-PAMDP in robot soccer and Platform domains against direct policy search and fixed-parameter baselines. Q-PAMDP improves performance in both domains, while Q-PAMDP(1) is less suited to the Platform domain's discontinuous value changes.
- Goal domain: The soccer task requires choosing between left- and right-shooting actions because the optimal shooting policy is discontinuous around the keeper.The parameterized actions include kick-to(x, y) and shoot-goal(h), with separate shoot-goal-left and shoot-goal-right actions used to represent this behavior.
- Goal domain: In robot soccer, Q-PAMDP(1) and Q-PAMDP(∞) increased goal probability to roughly 35%, versus 15% for eNAC and 20% for fixed-parameter SARSA.Both Q-PAMDP variants strongly outperformed fixed-parameter SARSA, while eNAC did not.
- Goal domain: A converged Q-PAMDP(1) soccer policy draws the keeper to one side and shoots immediately after overtaking the keeper.The illustrated episode shows the player running to one side before shooting when the goal is open.
- Platform domain: The Platform task uses three parameterized actions—run(dx), hop(dx), and leap(dx)—to navigate enemies and gaps, with episodes ending at the goal, enemy contact, or a fall.The player hops over enemies and leaps over gaps in successful episodes.
- Platform domain: In Platform, Q-PAMDP(1) and Q-PAMDP(∞) reached average distances of 50% and 65%, respectively, outperforming fixed-parameter SARSA at 40%.Direct eNAC reached 10% in the same comparison.
- Platform domain: Q-PAMDP(1) may be poorly suited to Platform because small parameter changes around the first successful jump produce large action-value changes.Q-PAMDP(1) is better suited to domains where action values change smoothly with the parameter policy, whereas Platform requires drastic changes after reaching a new platform.
6 Related Work
Related work addresses hybrid or parameterized actions through planning, sampling, symbolic dynamic programming, and hierarchical MDPs. The paper distinguishes its model-free learning setting from these approaches and their domain or representation assumptions.
- Hybrid action methods: Hauskrecht et al. study hybrid discrete-continuous action spaces, but their actions combine components rather than providing distinct actions with action-specific parameter dimensions.They also assume a compact factored representation and consider only planning.
- Parameterized-action planning: Rachelson’s XMDPs extend time-dependent continuous-time MDPs with parameterized wait actions, using a planning perspective in a time-dependent domain.The parameter space has the same size for all parameterized actions.
- Sampling methods: Hoey et al. use POMCP with domain-specific adjustments to compute continuous action components in Bayesian affect control theory.They characterize discrete control as the “what” and continuous control as the “how.”
- Symbolic dynamic programming: Zamani et al. apply symbolic dynamic programming to discrete parameterized actions with different parameter spaces, representing value functions with XADDs.Their setting is relational or first-order MDP planning.
- Hierarchical methods: Hierarchical MDPs can represent parameterized actions by treating parameter selection as a subtask, suggesting MAXQ as a possible value-function decomposition method.The passage presents MAXQ as a possibility for learning action values in parameterized-action problems.
7 Conclusion
The PAMDP formalism supports model-free learning with parameterized actions, and Q-PAMDP variants achieve convergence guarantees and outperform comparison methods in tested domains.
- Parameterized actions combine continuous adaptability with distinct action types and can represent discontinuous policies without complex parameterizations.
- Q-PAMDP(1) with an appropriate P-UPDATE method converges to a local or global optimum, while Q-PAMDP(∞) with global optimization converges to a local optimum.
- Q-PAMDP(1) and Q-PAMDP(∞) outperformed eNAC and fixed parameter SARSA in the goal-scoring domain and outperformed eNAC and fixed SARSA in the Platform domain.
- Q-PAMDP(1) and Q-PAMDP(∞) performed similarly in goal scoring, learning policies that scored goals roughly 35% of the time.