PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO

Yiyang He, Zhichun Zhou, Ziwei Wang, Tao Xue, Haolin Fei

arXiv:2608.24488v1cs.LG

TL;DR

Bounded-action PPO often optimizes an unbounded Gaussian before execution, raising whether boundary-heavy behavior reflects variance, mean placement, or both. The paper separates these factors, changes only the entropy measurement space, analyzes the resulting gradients, and tests the mechanism in MyoLeg and an independent Dog-Stand implementation. It finds that entropy measurement changes coupled mean–variance geometry, while task return alone does not characterize bounded-policy geometry.

  • Problem

    Continuous-control policies can use unbounded Gaussian distributions for bounded actuators, but return-only evaluation provides limited evidence about the resulting executed-action geometry.

  • Method

    The paper separates mean placement from dispersion, compares latent and executed-action entropy with matched PPO interventions, analyzes direct gradients, and replicates the comparison independently.

  • Results

    Across MyoLeg and Dog-Stand, entropy measurement changed mean–variance geometry: latent entropy favored dispersion and extruded means, while executed-action entropy produced more inward means; direct mean penalties could also center means.

  • Takeaways & Limitations

    Entropy measurement space is a coupled mean–variance design choice, so bounded-action PPO should measure mean placement, dispersion, and task return separately.

  • Takeaways & Limitations

    The MyoLeg environment contains a velocity-reward implementation quirk that contributes a constant reward term, although identical reward code is used across entropy conditions.

Abstract

from arXiv · show

Many continuous-control policies are optimized as unbounded Gaussians and then mapped into bounded actions. We show that where entropy is measured changes the policy geometry learned by proximal policy optimization (PPO). In an 80-muscle MyoLeg task, a clipped Gaussian executes 89.07% of actions within 5% of a bound. A same-state decomposition shows that this is not due to variance alone: setting variance to zero still leaves 83.83% of actions near a bound, while 82.12% of state-conditioned means lie outside the executable interval. Replacing clipping with a tanh map does not remove the high-variance regime. For latent Gaussian entropy H(u), the entropy loss has zero gradient with respect to the mean and a constant variance-increasing gradient. For executed-action entropy H(a), the transform Jacobian adds an inward gradient on the mean. Across three matched MyoLeg seeds, near-boundary occupancy is 71.42%, 29.76%, and 18.83% under latent entropy, no entropy, and executed-action entropy. A 38-dimensional Dog-Stand replication with an independent CleanRL-based PPO implementation reproduces the ordering in mean geometry, which also survives shared-state evaluation and boundary margins from 1% to 10%. Direct mean penalties can match or exceed the centering produced by H(a), showing that interior means are not unique to executed entropy. However, matched mean geometry can coexist with substantially different variance and return. Entropy measurement space is therefore a coupled mean-variance design choice, and task return alone does not characterize bounded-policy geometry.

1 Introduction

Bounded-action PPO can learn boundary-dominated policies because both state-conditioned mean placement and dispersion matter. Measuring entropy before or after the action transform changes the resulting mean–variance geometry, a mechanism tested across MyoLeg and an independent Dog-Stand replication.

  • Actuator-level phenomenon: 89% of executed muscle commands were within 5% of an action bound, while removing activation filtering or symmetrizing time constants left this regime essentially unchanged.A 14-actuator motor policy showed about 9.5% near-boundary occupancy under the same margin.
  • Mean placement and dispersion: 83.8% near-boundary occupancy remained after setting variance to zero, and 82.1% of state-conditioned means lay outside the executable interval.Large dispersion was also sufficient, but the policy mean itself was strongly extruded.
  • Entropy measurement space: Latent Gaussian entropy has no direct mean gradient and increases variance uniformly, whereas executed-action entropy adds an inward mean gradient through the transform Jacobian.For the tanh map, the variance gradient also becomes distribution-dependent.
  • Entropy measurement space: Across three MyoLeg seeds, latent entropy produced the largest dispersion and most extruded means, while no entropy reduced dispersion but left means closer to bounds than executed-action entropy.Thus the policy with the smallest exploration-driven boundary occupancy did not have the smallest total boundary occupancy.
  • Independent replication: An independent 38-dimensional Dog-Stand PPO replication reproduced the mean-geometry ordering across seeds and checkpoints, shared states, and 1%–10% boundary margins.Direct mean penalties could match or exceed H(a) centering while producing different variance and return regimes.
  • Study logic: The study connects entropy measurement space to trained PPO policy geometry through controlled decompositions, gradient analysis, and independent replication rather than proposing a new entropy formula.The intervention changes the entropy objective while retaining the latent Gaussian sampling family and PPO importance ratio.

2 Related Work

Prior work addresses bounded actions through bounded distributions, corrected gradients, transformed densities, and structured exploration. This paper instead keeps the squashed-Gaussian parameterization fixed and studies how PPO entropy-space choices affect policy geometry.

  • Bounded-action methods include Beta policies, clipping-aware policy gradients, marginal policy gradients, and tanh-squashed Gaussians.
  • SAC evaluates transformed action density with the tanh Jacobian, corresponding here to entropy measured in executed action space.
  • The paper isolates entropy measurement space while retaining the squashed-Gaussian parameterization and decomposing boundary-heavy behavior into mean placement and dispersion.
  • PPO implementation choices, including entropy bonuses and standard-deviation parameterization, can materially affect learned behavior and reported performance.
  • Direct mean penalties are used as mechanism controls to test whether executed entropy’s mean-centering effect can be reproduced without changing entropy definition.

3 Policy Geometry in a Bounded Action Space

The paper formalizes latent and executed action spaces, their boundary criteria, and a same-state decomposition separating mean and variance contributions. It also shows that executed-action entropy adds inward mean pressure while changing the variance gradient.

  • PPO produces a diagonal Gaussian latent action with state-independent log σ, while the environment executes a = f(u) through clipping or tanh.
  • A 5% near-boundary margin on [0, 1] uses thresholds a ≤0.05 or a ≥0.95.
  • Dog-Stand uses y = tanh u ∈(−1, 1) and classifies actions as near-boundary when |y| ≥0.9.
  • The paper distinguishes absolute-distance margins from range-fraction margins; they coincide on [0, 1] but differ on symmetric [−1, 1] action boxes.
  • Same-state counterfactual cells separate mean-only, centered-baseline, and variance-related contributions to near-boundary occupancy.
  • The interaction term can be large and negative when mean and variance effects are redundant under the 100% ceiling, so the four cells are reported directly.
  • Latent entropy has no direct mean force and a variance-increasing gradient independent of current variance.
  • For the tanh map, the Jacobian contributes an inward mean effect and makes the variance gradient finite and distribution-dependent.

4 Experimental Design

The experiments compare bounded-policy geometry across MyoLeg and Dog-Stand while varying entropy objectives, evaluation states, boundary margins, and direct mean penalties. They also document task definitions, training configurations, gradient diagnostics, statistical units, and reproducibility controls.

  • MyoLeg task: 80-muscle MyoLeg commands lie in [0, 1]80, while 14 motor commands lie in [−1, 1]14; observations have 101 dimensions.The environment runs at 250 Hz from a 1 ms physics timestep and frame skip of 4.
  • Reward and statistical specification: The velocity subreward is constant because its velocity-difference variable is set to zero, and the same reward code is used across MyoLeg entropy conditions.The training seed is the method-level comparison unit; Dog return comparisons remain descriptive with three seeds.
  • Entropy conditions: The MyoLeg design compares latent entropy, no entropy, and executed-action entropy while keeping the latent Gaussian family and PPO ratio identical.The primary tanh comparison uses separate 512–512 ReLU policy and value networks and three training seeds.
  • External replication: A 38-dimensional Dog-Stand replication uses an independent CleanRL-based PPO implementation, with only the action transform, entropy objective, and diagnostics changed across primary conditions.Each condition is trained from scratch for 3,000,320 steps with seeds 0, 1, and 2.
  • Mean-penalty controls: Direct mean penalties are trained without entropy by matching the H(a) entropy-loss mean-gradient magnitude at the 5% boundary threshold.The controls use fixed coefficients λtanh2 = 0.0105263158 and λµ2 = 0.0012226438, with the same Dog seeds and evaluation protocol.
  • Gradient diagnostics: Frozen-gradient decomposition separately estimates PPO and entropy contributions for µ and log σ, while Dog terminal log-variance intervals are used only to bound long-horizon variance interpretations.Dog inward-mean diagnostics use 30 independent buffers per late-training checkpoint and seed.
  • Robustness checks: Shared-state evaluation and boundary-margin sweeps from 1% to 10% test whether mean-geometry ordering depends on visited states or a particular threshold.The same states are used within each margin sweep, and the shared-state audit scores every policy on every donor set.
  • Reproducibility: The archival code package includes training implementations, entropy estimators, evaluators, audits, gradient diagnostics, frozen result JSONs, and SHA-256 manifests.The paper treats the training and analysis code as part of the experimental specification.

5 Results

The results separate bounded-policy boundary occupancy into mean placement and dispersion, then show that entropy measurement space changes both through distinct gradients. These geometric effects replicate across tasks and remain separable from return, state visitation, and direct mean regularization.

  • 5.2 Mean extrusion and dispersion are both sufficient: 89.065% ± 0.102% actual occupancy falls to 83.825% ± 1.103% when variance is zero, while centering the mean leaves 77.515% ± 0.428%.The interaction is about −72 pp, indicating strong redundancy near the 100% ceiling.
  • 5.2 Mean extrusion and dispersion are both sufficient: 82.12% ± 1.26% of state-conditioned means lie outside [0, 1], so reducing exploration variance alone cannot make the controller interior.Hard clipping adds exact atoms, but the latent policy geometry is already boundary oriented before clipping.
  • 5.3 Entropy measurement space separates the trained tanh policies: Smooth tanh mapping preserves the high-variance latent regime: H(u) ends at +0.8835 ± 0.0499 mean log σ versus −0.6920±0.0110 without entropy and −0.5748±0.0069 under H(a).The local gradient decomposition constrains but does not fully explain the long-horizon training trajectories.
  • 5.3 Entropy measurement space separates the trained tanh policies: No entropy has lower P01 than H(a), 1.53% versus 2.43%, but higher total occupancy, 29.76% versus 18.83%, because P10 is 25.67% instead of 11.62%.The smallest exploration-driven boundary occupancy is therefore not the smallest total occupancy.
  • 5.4 The direct gradient on the mean is measurable: H(a) adds a direct inward mean gradient with cosine +0.979, whereas H(u) and no entropy have exactly zero entropy gradient with respect to the mean.The PPO surrogate has only a weak, heterogeneous radial component, so latent entropy leaves mean extrusion unconstrained while H(a) opposes it.
  • 5.5 External replication in Dog-Stand: Dog-Stand reproduces the mean-geometry ordering across 18 of 18 state-set–seed comparisons and all 48 task × metric × margin × seed cells.The pooled H(u)–H(a) P10 separation is 35.23 points, exceeding the largest donor-induced span of 6.37 points.
  • 5.7 Direct mean regularization reproduces centering but not the same operating regime: Direct mean penalties reduce P10 to 0.09% and 1.60%, below H(a) at 1.91%, showing that interior means are not unique to executed-action entropy.Mean-only regularization can also alter the long-horizon variance regime through coupled PPO optimization dynamics.
  • 5.7 Direct mean regularization reproduces centering but not the same operating regime: L2 and H(a) have similar mean geometry but differ in mean log σ by 0.600 and stochastic return by 129 points.Their P10 values are 1.60% and 1.91%, respectively, while pooled shared-state values remain close at 4.13% and 4.47%.

6 Discussion

The paper shows that entropy measurement changes bounded-policy mean–variance geometry, while return, mean placement, variance, and execution behavior can vary independently across tasks and controls.

  • Mechanism: Latent entropy changes variance but is blind to mean placement, whereas executed entropy adds an inward mean regularizer and state-dependent variance term.These mechanisms produce distinct mean–variance regimes without changing the latent Gaussian sampling family or PPO importance ratio.
  • Controls: Direct mean penalties can match or exceed H(a)'s centering, showing that executed entropy is not uniquely capable of preventing mean extrusion.The controls target current policy means rather than action-rate smoothness, unlike CAPS.
  • Interpretation: Matched interior means can coexist with substantially different dispersion and Dog return, so mean geometry does not determine the final variance or performance regime.The comparison does not establish that higher retained variance causes higher return because objectives also alter the optimization trajectory.
  • Cross-task evidence: The geometric ordering is consistent across MyoLeg and Dog-Stand, but return ordering is not: no entropy leads MyoLeg while H(a) leads Dog-Stand.Selecting methods by return alone therefore gives different conclusions about action geometry across the two tasks.
  • Diagnostics: A four-cell decomposition recommends reporting P10 and P01 alongside return and total near-boundary occupancy to distinguish mean placement from dispersion.Large P10 indicates that variance reduction alone cannot produce an interior mean.
  • Scope: The experiments cover PPO with diagonal Gaussian policies, bounded actions, state-independent learned log σ, and MuJoCo-family simulation; state-dependent exploration and hardware remain untested.Dog conclusions are also tied to the attained performance regime and canonical stochastic returns of roughly 464–594.

7 Conclusion

In the studied PPO settings, return alone can hide bounded-policy geometry: mean extrusion and dispersion jointly shape boundary occupancy, while executed entropy changes both mean and variance objectives.

  • Conclusion: Latent entropy raises the relative variance regime without constraining mean placement, whereas executed entropy adds a direct inward mean regularizer and changes the variance objective.The ordering in mean geometry is reproduced in 38-dimensional Dog-Stand, shared-state evaluation, and margins from 1% to 10%.
  • Conclusion: Direct mean penalties can match or exceed H(a)'s centering, yet mean placement, dispersion, and task return remain distinct controller properties.All three should therefore be measured explicitly for bounded-action PPO.

A Boundary Margin Sensitivity

The boundary-margin analysis evaluates P11/P10 across four margins and finds the primary geometry ordering robust, while identifying one threshold-sensitive variance comparison.

  • Evaluation: The margin sweep uses the same states and action-noise convention as canonical evaluation, with no action resampling for boundary probabilities.This design makes the 5% cells reproduce Tables 2 and 3 exactly.
  • Caveat: No-entropy seed 2 exceeds H(a) in P01 at 1% and 2.5% margins but falls below it at 5% and 10%, so this comparison is not headline evidence.The reported pairs are 0.2610% versus 0.1204%, 0.7309% versus 0.6857%, 1.8284% versus 2.3493%, and 5.2824% versus 7.5610%.

B Common-State Dog Audit

The common-state Dog audit tests whether policy geometry depends on the evaluated state distribution and finds the primary ordering across all donor sets and seeds.

  • Common-state evaluation: The audit evaluates every trained Dog policy on every policy's raw donor states, using each policy's own frozen observation normalizer.An equal-mixture pool contains equal numbers of states from each donor condition.
  • Results: The ordering H(u) > no entropy > H(a) holds on all six evaluated state sets in every seed: 18 of 18 comparisons.Each policy also has its lowest seed-averaged P10 on its own donor distribution in 14 of 15 policy–seed comparisons.
  • Interpretation: The donor-distribution pattern is treated as descriptive evidence of policy–state-distribution co-adaptation, not as a causal result.

C Gradient Details

The late-training MyoLeg gradient decomposition characterizes local log-σ gradients, while its interval evidence is explicitly limited as a full-trajectory causal explanation.

  • Table 6 reports mean per-dimension gradients with respect to log σ over 10 independently collected frozen-policy buffers for each entropy condition.The analysis uses the late-training MyoLeg seed-0 checkpoint.
  • Only the net interval under no entropy excludes zero.The intervals are 95% t-intervals over buffers, and positive loss gradients lower σ.
  • The local decomposition constrains the late-training MyoLeg mechanism but is not a complete causal explanation of the full trajectories.

D Training Configuration and Executed-Entropy Estimator

The primary experiments use verified tanh-policy configurations and an executed-entropy estimator that combines analytic latent entropy with a sampled transform Jacobian and pathwise gradients.

  • Training Configuration: The two primary implementations use separate actor and critic MLPs with a state-independent learnable log σ vector.Tables 7 and 8 provide the verified configurations and architectures.
  • Executed-Entropy Estimator: The executed-entropy path includes the MyoLeg affine map to [0, 1] in the Jacobian, while Dog’s later constant affine scaling is omitted from policy gradients.
  • Executed-Entropy Estimator: The estimator computes σ = exp(log σ) and exact latent entropy, then samples two independent Gaussian noises and their antithetic negatives.
  • Executed-Entropy Estimator: For each sampled noise, it forms the reparameterized latent action u_k = µ + σ ⊙ϵ_k and evaluates a stable componentwise log-Jacobian for a = (tanh u + 1)/2.
  • Executed-Entropy Estimator: The actor loss adds the negative cent-weighted minibatch mean of the log-Jacobian to analytic Gaussian entropy.Reparameterization provides pathwise gradients in µ and log σ; K = 4 is fixed for H(a) runs.
  • Reproducibility: The archival package and manifest link training implementations, canonical evaluators, analysis scripts, frozen outputs, evaluator revisions, checkpoint hashes, and seed conventions.Reported returns come from canonical evaluators rather than training-time rolling diagnostics.

E Reproducibility Notes

The final analysis uses audited, step-matched checkpoints and versioned evaluators, with detailed software, hardware, control-loop, and training-provenance records.

  • The final analysis uses step-matched checkpoints and versioned canonical evaluators after an audit changed earlier values or statistical interpretations.The audit identified issues including non-compute-matched reward-selected checkpoints and replayed rollouts in an initial gradient analysis.
  • Experiments ran with a recorded Windows, Python, PyTorch, simulation, and RL software stack on a single RTX 5090 system.The release manifest also records the full pip freeze, driver report, checkpoint digests, and resolved configuration fields.
  • MyoLeg uses a 4 ms control period at 250 Hz, while Dog-Stand uses a 0.015 s control period at 66.67 Hz.The MyoLeg timing was verified from the live environment with a 1 ms physics timestep and frame skip of 4.
  • No-entropy and H(a) seed-0 runs resume 30M-step pilots, symmetric-τ seeds 1 and 2 continue 100M-step runs, and filter and override seed-0 policies use earlier single-run drivers.
Loading 2608.24488v1…