Discretizing Continuous Action Space for On-Policy Optimization

Yunhao Tang; Shipra Agrawal

Source-linked AI summary

Discretizing Continuous Action Space for On-Policy Optimization

Yunhao Tang, Shipra Agrawal

arXiv:1901.10500v4Jan 29, 2019cs.LGcs.AI

TL;DR

Continuous-control optimization must handle continuous actions while retaining tractable, expressive policies. The paper discretizes actions with factorized distributions and adds an ordinal parameterization, finding gains over baseline policies especially on complex high-dimensional tasks. It also reports a numerical-stability limitation for Beta-distribution comparisons.

Problem
The paper studies whether continuous-control tasks can use simple discrete policies without the combinatorial explosion caused by discretizing multiple action dimensions.
Method
It discretizes each action dimension, factorizes the joint discrete policy across dimensions, and introduces an ordinal parameterization that encodes natural class ordering.
Results
Discrete policies significantly outperform Gaussian policies on high-dimensional tasks with complex dynamics, while ordinal policies further improve PPO and TRPO performance.
Takeaways & Limitations
Action discretization and ordinal parameterization improve baseline on-policy algorithms across the evaluated benchmark control tasks.
Takeaways & Limitations
Beta-distribution training can produce numerical errors under aggressive PPO or TRPO updates, while conservative updates reduce errors but greatly degrade learning performance.

Abstract

from arXiv · show

In this work, we show that discretizing action space for continuous control is a simple yet powerful technique for on-policy optimization. The explosion in the number of discrete actions can be efficiently addressed by a policy with factorized distribution across action dimensions. We show that the discrete policy achieves significant performance gains with state-of-the-art on-policy optimization algorithms (PPO, TRPO, ACKTR) especially on high-dimensional tasks with complex dynamics. Additionally, we show that an ordinal parameterization of the discrete distribution can introduce the inductive bias that encodes the natural ordering between discrete actions. This ordinal architecture further significantly improves the performance of PPO/TRPO.

1. Background

The paper asks whether continuous-control tasks can retain the simplicity of discrete actions. It proposes factorized discretization to avoid the combinatorial explosion and studies its effect on on-policy optimization.

Continuous-control tasks use continuous actions, whereas discrete actions are simpler to analyze but have mainly been applied to games.
Discretizing each of M action dimensions into K atomic actions creates M K combinations, becoming intractable as dimensionality grows.
A factorized joint distribution across action dimensions keeps the discrete policy tractable for sampling and training.
The paper systematically evaluates discretization in simulated environments against on-policy optimization baselines.

2. Background

This section introduces Markov decision processes and policy optimization, then reviews trust-region algorithms and policy-class choices relevant to comparing discrete and continuous policies.

2.1. Markov Decision Process: An MDP maps states to actions, rewards, and transitions, while a policy maps states to distributions over actions.
2.2. On-Policy Optimization: Policy optimization searches over parameterized policies using policy gradients and incremental parameter updates.
2.2. On-Policy Optimization: Vanilla policy-gradient training can take occasionally large steps and fail to recover from a bad policy, motivating more stable variants.
2.2. On-Policy Optimization: TRPO constrains policy changes with KL divergence, while PPO approximates trust-region updates by clipping likelihood ratios; ACKTR uses K-Fac to invert the constraint matrix.
2.3. Related Work: The paper separates on-policy optimization algorithms from policy distributions to identify the net effect of policy-class expressiveness.
2.3. Related Work: Prior work has converted continuous problems into discrete ones, but some approaches were demonstrated only on relatively low-dimensional or limited task sets.
2.3. Related Work: Ordinal modeling incorporates internal ordering among discrete variables, while earlier approaches either lacked a proper probabilistic model or required extra inference techniques.

3. Discretizing Action Space for Continuous Control

The paper discretizes each continuous action dimension into equally spaced atomic actions and factorizes the joint policy across dimensions. The resulting policy can represent multimodal distributions and may outperform Gaussian policies, but K trades control capacity against training difficulty and cost.

Each action dimension in A = [−1, 1]^m is discretized into K equally spaced atomic actions.
The joint discrete policy factorizes into one categorical distribution per action dimension, making sampling and training tractable.
With sufficient atomic actions, discrete policies can represent multimodal action distributions that Gaussian policies cannot represent by design.
In a K = 11 bandit experiment, 4 out of 5 Gaussian policies became stuck at a suboptimal policy, while all discrete policies achieved optimal rewards.
Discrete policies can capture multimodality in complex reward landscapes and achieve better exploration to bypass suboptimal policies.
Choosing K trades control capacity against training difficulty because small K makes discretization coarse while larger K increases policy-gradient variance.
Model parameters and computational costs grow linearly in K.

4. Discrete Policy with Ordinal Architecture

The ordinal architecture injects ordering information into discrete action distributions while retaining a proper probabilistic model. Its parameter dependencies encode graded differences between neighboring and more distant classes, improving policy optimization in practice.

Because discretization discards information about the underlying continuous space, the ordinal architecture incorporates continuity when parameterizing discrete distributions.
The architecture transforms logits with sigmoids, computes final logits from the transformed values, and obtains output probabilities with a softmax.
Its stick-breaking-like parameterization is motivated by ordered classes whose samples can be separated according to their class order.
The ordinal loss uses cumulative target vectors, making the discrepancy between class k and k + 1 smaller than that between k and k + 2.
Unlike one-hot encoding, the ordinal architecture introduces internal class ordering while maintaining the probabilistic properties of discrete distributions.
The architecture introduces dependencies between logits and generally brings significant performance gains during policy optimization.

5. Experiments

Experiments compare discrete, ordinal, Gaussian, and alternative policy architectures across on-policy benchmarks. Discrete policies are strongest on complex, high-dimensional tasks, while ordinal parameterization further improves performance and robustness.

Benchmark design: Experiments compare discrete policies with Gaussian policies across PPO, TRPO, and ACKTR on MuJoCo, rllab, Roboschool, and Box2D tasks.The study makes minimal changes to the original on-policy algorithms and varies the number of discrete bins.
Benchmark performance: Discrete policies significantly outperform Gaussian policies on high-dimensional tasks with complex dynamics, although advantages are smaller or absent on simpler tasks.For PPO, discrete learning is comparable on HalfCheetah and slightly lower on Ant, but substantially better on Humanoid; TRPO gains are consistently significant.
Benchmark performance: Discrete policies outperform Gaussian policies across all Roboschool Humanoid tasks for both PPO and TRPO, with the largest gains under TRPO.The Gaussian policy barely progresses in some TRPO settings, whereas the discrete policy has stable learning curves.
Architecture controls: Increasing Gaussian network size does not match discrete-policy performance, indicating that the gains are attributed to the distribution class rather than larger networks.The comparison trains Gaussian policies with substantially more hidden units per layer.
Comparison with off-policy methods: PPO with discrete or ordinal actions achieves comparable or better performance than state-of-the-art off-policy baselines on Humanoid tasks after 10^7 training steps.The comparison includes DDPG, SQL, SAC, and TD3, with off-policy results approximated from prior figures.
Alternative architectures: Beta policies often encounter numerical errors under aggressive updates, while conservative updates reduce errors but greatly degrade learning performance.Reported Beta results use the final 10 iterations before potentially premature termination, so their potential advantages are offset by unstable training.
Discrete Policy vs. Ordinal Policy: Ordinal policies significantly improve asymptotic performance and convergence speed over discrete policies across the evaluated high-dimensional tasks.The ordinal policy also achieves comparable performance to efficient off-policy methods on Humanoid tasks and improves over discrete policies for most tasks.
Discrete Policy vs. Ordinal Policy: The ordinal architecture is interpreted as providing an inductive bias that improves PPO and TRPO optimization, while task-specific ordering remains future work.The study also reports that discrete policies are generally more robust to sampled hyper-parameters than Gaussian policies.

6. Conclusion

The paper finds that action discretization substantially improves baseline on-policy algorithms, particularly on high-dimensional tasks with complex dynamics, while ordinal parameterization can boost performance further.

Action discretization greatly improves baseline on-policy algorithms, especially on high-dimensional tasks with complex dynamics.
Ordinal parameterization encodes natural ordering in the discrete distribution and can further boost baseline algorithm performance.

A. Hyper-parameters

The experiments use PPO, TRPO, and ACKTR implementations based on OpenAI baselines, with policy classes including Gaussian, discrete, ordinal, Gaussian+tanh, and Beta policies.

PPO, TRPO, and ACKTR implementations are based on OpenAI baselines, while environments use OpenAI Gym, rllab, and Roboschool.
Gaussian policies use neural-network means with diagonal covariance whose standard deviations are shared across states.
Discrete policies factorize categorical distributions across action dimensions, using K evenly spaced atomic actions and neural-network logits per dimension.
Ordinal policies add ordinal parameterization to discrete policies without changing their parameter count.
Gaussian+tanh policies add a final tanh transformation to constrain the mean to [−1, 1].
Beta policies can become numerically unstable near action boundaries, while reducing the trust-region size stabilizes training but degrades performance.The reported instability occurs when αθ(s) or βθ(s) must approach infinity to represent boundary actions.
Value-network sizes and selected PPO, TRPO, and ACKTR hyper-parameters are tuned separately, with other settings taken from baseline implementations.

B. Effects of the Number of Atomic Actions

The number of atomic actions creates a trade-off: larger K increases control capacity but also raises policy-gradient variance and computational cost, with best performance reported at 7 ≤K ≤15.

For each action dimension, K logits are converted by soft-max into probabilities over K atomic actions.
The variance analysis compares gradients of shared state-encoding parameters in a one-step bandit with uniform high-entropy initialization.
Policy-gradient variance increases with finer discretization, making optimization harder under the same hyper-parameters.
Control capacity rises with K, but small K limits performance and large K makes training harder.
The best performance is obtained when 7 ≤K ≤15; K = 2 degrades performance from insufficient control capacity, while K ≈50 causes only slight degradation.
Model parameters and training costs scale linearly with K, with cost increases potentially more severe at higher action dimensions.

C.1. PPO

PPO results vary with task complexity: discretization is not consistently better on simpler MuJoCo tasks, while ordinal policies outperform discrete policies on high-dimensional tasks.

Discrete policy does not necessarily outperform factorized Gaussian on simpler MuJoCo tasks.
PPO learning curves compare Gaussian and discrete policies using K = 7, 11, and 15 bins on MuJoCo tasks.
TRPO results on simpler MuJoCo tasks show discrete policies significantly outperform Gaussian on Reacher and Double Pendulum.

C.3. ACKTR

ACKTR evaluations show that discretization offers little gain on relatively simple dynamics but significant gains over Gaussian policies on Humanoid tasks.

Discrete policies significantly improve performance over Gaussian policies on ACKTR Humanoid tasks.
For ACKTR tasks with relatively simple dynamics, discrete-policy performance gains are not significant.
Gaussian policies with larger networks generally perform worse than smaller networks in the reported TRPO and PPO comparisons.

D. Illustration of Benchmark Tasks

The benchmark suite covers simulated locomotion tasks using sensory states and actuator actions, with performance visualized through learning curves and sensitivity plots.

All benchmark tasks use sensory data as states and actuator controls as actions.
ACKTR benchmark curves show cumulative rewards against time steps and average results over 5 random seeds with mean ± std performance.
PPO Gaussian-policy network comparisons use larger and smaller two-layer networks, with smaller networks generally performing better.
PPO sensitivity plots show final performance across benchmark tasks in OpenAI MuJoCo, rllab, and Roboschool.
PPO experiments train Reacher for 10^6 steps, Hopper for 2 · 10^6 steps, and other tasks for about 5 · 10^6 steps.
Figure 12 illustrates locomotion tasks from OpenAI Gym, rllab, MuJoCo, and Roboschool.