Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints

1 INTRODUCTION

• Existing entropy bonuses can alter the primary optimization landscape, while LLM entropy-collapse remedies are described as heuristic, domain-specific, and lacking theoretical guarantees.Entropy collapse in LLM policy-gradient methods is associated with reduced diversity and performance degradation; direct entropy bonuses have been unstable or ineffective in that setting.Source: §1 · p. 1
• ERA applies designed activations to model outputs to impose an entropy constraint while decoupling it from optimization of the primary objective.The paper states that ERA has provable entropy guarantees and can be integrated with existing algorithms as a non-invasive module.Source: §1 · p. 2
• ERA improves SAC on high-dimensional DeepMind Control Suite tasks like Humanoid and Dog by over 25%, and improves Qwen-2.5-Math-7B by 9.0% on AIME-24 and 37.4% on AIME-25.The figure also summarizes gains for language-model, control, and image-classification benchmarks and methods.Source: §1 · p. 2, Figure 1 · p. 2
• The paper establishes a theoretical framework with provable entropy guarantees and designs ERA instantiations for continuous control, discrete image classification, and LLMs.The LLM version is specialized and adaptive; experiments report improvements over strong baselines and examine properties such as parameter sensitivity.Source: §1 · p. 2

Policy learning in control

• PPO and SAC introduce entropy terms into their objectives, and SAC’s dynamically adjusted temperature can lead to instability.Other control approaches implement maximum entropy in different ways or optimize state entropy directly; the paper says these methods modify the cumulative reward objective and can lead to suboptimal performance.Source: §2 · p. 2

RL for LLMs

• LLM entropy-collapse approaches include reward shaping, loss re-weighting, and clip-higher regularization, whereas ERA is presented as cross-domain and theoretically guaranteed.The passage describes entropy collapse as decay of exploratory behavior that often leads to performance plateaus.Source: §2 · p. 3

Policy optimization

• GRPO estimates a generated response’s advantage from a set of K samples, and the policy is updated by maximizing its objective.The passage refers to the GRPO advantage expression and the policy update objective.Source: §3 · pp. 3–4
• The clipped surrogate objective takes the minimum of the probability-ratio-weighted advantage and its clipped counterpart.The ratio compares current-policy and old-policy action probabilities.Source: §3 · p. 3

Policy entropy

• Policy entropy measures stochasticity; the preliminaries describe token-level entropy for discrete actions and entropy for squashed or truncated Gaussian continuous policies.For truncated Gaussian policies, the passage defines standardized bounds and a normalization constant using the standard normal PDF and CDF.Source: §3 · p. 3

Maximum entropy reinforcement learning

• Maximum-entropy RL maximizes the reward objective subject to a minimum entropy constraint, and SAC solves the Lagrangian dual while updating a soft Q-function and policy.The SAC target includes reward and a next-state soft-Q term adjusted by policy log-probability; its target network is updated by exponential moving average.Source: §3 · pp. 3–4

4.1 THE CORE IDEA: ENTROPY CONSTRAINT VIA OUTPUT ACTIVATION

• ERA enforces expected entropy through an activation on output parameters, leaving θ's training objective free of explicit entropy terms and mitigating gradient conflicts.The final policy is formed from the transformed parameters z′.Source: §4.1 · p. 4

4.2 INSTANTIATIONS FOR CONTINUOUS AND DISCRETE SPACES

• The framework is instantiated for bounded-Gaussian policies in continuous control and softmax policies in discrete spaces.Source: §4.2 · p. 4

4.2.1 CONTINUOUS CONTROL WITH BOUNDED GAUSSIAN POLICIES

• For bounded Gaussians, ERA compensates for nonnegative bounding bias by raising underlying Gaussian entropy via σ while keeping σ in [σmin, σmax].Architectural entropy constraints also allow SAC's critic target and actor loss to omit an explicit entropy term.Source: §4.2.1 · pp. 4–5

4.2.2 DISCRETE CLASSIFICATION WITH SOFTMAX POLICIES

• For softmax classification, ERA transforms logits to enforce minimum predictive entropy, allowing structured input-dependent uncertainty unlike fixed, uniform label smoothing.Source: §4.2.2 · p. 5

4.3 INSTANTIATIONS FOR RL IN LARGE LANGUAGE MODELS

• LLM ERA leaves sampling unchanged and transforms sampled-token probabilities during updates, targeting entropy among the response's top 20% highest-entropy tokens.Under reasonable assumptions, policy entropy remains above H0; the design avoids imposing high entropy on every token, which could introduce unintended tokens and corrupt responses.Source: §4.3 · pp. 5–6

5.1 EXPERIMENTS ON CONTINUOUS CONTROL

• ERA improves sample efficiency and final performance across diverse continuous-control algorithms and benchmarks.Across four DMC tasks, SAC-ERA also outperforms SAC throughout the tested range of entropy targets, including a comparison with SAC without its entropy term.Source: §5.1 · pp. 6–7

5.2 EXPERIMENTS ON IMAGE CLASSIFICATION

• ERA outperforms ResNet-50 across ImageNet and CIFAR-10 with and without standard data augmentation.Accuracy remains relatively insensitive to the tested minimum-entropy values, without extensive tuning.Source: §5.2 · p. 7, Table 1 · p. 7, Figure 3 · p. 8

5.3.1 MAIN RESULTS

• ERA achieves the best results on most of six mathematical reasoning benchmarks, outperforming strong entropy-based baselines by significant margins.For AIME’24, AIME’25, and AMC’23, results average 16 sampled responses; the evaluation samples from the original policy before ERA.Source: §5.3.1 · p. 8, Table 2 · p. 8

5.3.2 EXTENSION TO MORE MODELS AND ALGORITHMS

• ERA yields significant gains on Qwen2.5-Math-1.5B and GSPO, including a 14.1% average improvement on Qwen2.5-Math-1.5B.These extensions test ERA across a smaller model and another algorithm.Source: §5.3.2 · p. 8

5.3.3 ANALYSIS ON ENTROPY AND REASONING CAPACITY BOUNDARY

• ERA maintains a stable entropy level while GRPO undergoes entropy collapse, and achieves higher pass@k on AIME’24 and AIME’25.The authors relate retaining multiple candidate reasoning trajectories to improved likelihood of successful solutions under pass@k evaluation.Source: §5.3.3 · p. 9, Figure 4 · p. 9

5.3.4 OUT-OF-DISTRIBUTION GENERALIZATION

• ERA outperforms GRPO by 16.9% on average across three hard out-of-distribution benchmarks.The benchmarks are ARC-C, GPQA-Diamond, and MMLU-Pro.Source: §5.3.4 · p. 9, Figure 5 · p. 9

6 CONCLUSIONS

• ERA treats output activations as a direct medium for entropy regularization, with theoretical analysis and empirical results across diverse, challenging domains.The authors report that it surpasses prominent baselines without significant computational overhead and identify robust, efficient learning agents as a research avenue.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

• The appendix provides derivations and proofs, experimental details, and core ERA code; the authors also report a full open-source codebase.The authors state these materials are intended to support straightforward reproduction.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1.1 CODE IMPLEMENTATION OF ERA IN CONTINUOUS CONTROL

• The continuous-control actor applies ERA at its output, using a target entropy that may be constant or learnable.The implementation passage defines log-standard-deviation bounds, action dimension, and the actor’s raw output.Source: §A.1.1 · p. 15, Figure 6 · p. 15

A.1.2 ENVIRONMENTS

• Experiments use standard Mujoco Gym and DMC configurations, while SAC and OBAC HumanoidBench runs use action repeat 2 and disabled episode termination.FastSAC uses standard HumanoidBench tasks; reported seed counts vary by algorithm.Source: §A.1.2 · pp. 15–16, Figure 8 · p. 16

A.1.3 PSEUDO CODE OF SAC-ERA

• The appendix presents SAC adapted with ERA as a representative algorithmic example.The pseudocode covers critic and policy updates.Source: §A.1.3 · p. 16, §A.2.1 · p. 19

A.1.4 HYPERPARAMETERS

• The appendix identifies the codebases and selected settings used for SAC, OBAC, PPO, TD-MPC2, and FastSAC implementations.It notes that FastSAC-ERA differs from the method in the original paper.Source: §A.1.4 · pp. 16–18

A.1.5 FASTSAC-ERA

• FastSAC-ERA retains FastTD3’s noise mechanism, removes Delayed Policy Updates, and adds ERA at the actor output; it outperforms FastTD3.Other implementation details and hyperparameters are kept identical to the original FastTD3 configuration.Source: §A.1.5 · p. 18

A.2.1 CODE IMPLEMENTATION OF ERA IN IMAGE CLASSIFICATION

• The image-classification implementation transforms logits before softmax; its C_H parameter corresponds to the target entropy, and τ is set to 4 without tuning.The implementation passage identifies n_dims as the number of classes.Source: §A.2.1 · p. 18

A.2.2 TRAINING SETUP

• ImageNet training used four A100 GPUs; CIFAR-10 used three runs on separate machines, each with four A40 GPUs.The authors report 95% confidence intervals for both datasets, computed from the dataset for ImageNet and from the three runs for CIFAR-10.Source: §A.2.2 · p. 19

A.2.3 COMMANDS USED FOR EXPERIMENTS

• ImageNet and CIFAR-10 use the same timm-sourced commands, with and without augmentation, and no dataset-specific hyperparameter tuning.The two settings use a consistent experimental setup.Source: §A.2.3 · p. 19

A.3.1 CODE IMPLEMENTATION OF ERA IN LLM

• The LLM implementation retains the 20 largest logits with a top-k filter; advantage scaling is omitted because it did not affect performance.The implementation maps era_lb, era_ub, and era_k to ωlow, ωhigh, and k.Source: §A.3.1 · p. 20

A.3.2 HYPERPARAMETERS

• GRPO-based LLM experiments use batch and mini-batch sizes of 256, a learning rate of 10^-6, and dynamic sampling.Qwen2.5-Math-1.5B uses MATH problems at levels 3–5 for training.Source: §A.3.2 · p. 20
• ERA uses ωlow = 0.45, ωhigh = 3.0, and k = 2 across settings, except Qwen2.5-Math-7B’s second stage, which uses ωlow = 0.2 and ωhigh = +∞.For the other settings, the fixed values reference initial response entropy Hresp ≈1.5.Source: §A.3.2 · p. 20

• Residual entropy δ is the minimum additional entropy needed to ensure the transformed Gaussian’s final entropy is at least H0.It accounts for entropy lost through bounding; the proof uses H0 + δ as the pre-transformation threshold.Source: §B.1 · p. 21
• ERA can learn a state-independent δ̂ by minimizing Eq. 12, using an objective identical to SAC’s and a dual formulation whose convergence follows from strong duality.Source: §B.1 · p. 21
• When residual entropy δ̂ ≥ δ, the continuous ERA policy has entropy H(π) ≥ H0 and σ′ bounded within [σmin, σmax].The proof requires pre-transformation Gaussian entropy of at least H0 + δ.Source: §B.1 · p. 21

• Because −p ln p is non-monotonic on [0, 1], ERA uses probabilities divided by τ > e, where the entropy mapping is strictly increasing and invertible.The inverse is approximated numerically, then the recovered probabilities are normalized.Source: §B.2 · p. 22
• The entropy loss during normalization is bounded, so enforcing an initial entropy above CH0 guarantees final discrete entropy above CH0.Source: §B.2 · p. 22
• ERA’s discrete policy has entropy H(π) ≥ H0 for target entropy H0 and hyperparameter τ ≥ e.This is the stated guarantee for the policy defined by Eq. 15.Source: §B.2 · p. 22

B.3 PROOF OF ENTROPY BOUND IN LLMS

• When γBk − α > β, ERA updates keep expected response entropy above a positive constant H0 under the stated assumptions.Below ωlow, the adaptive KL regularizer produces a positive first-order entropy increase; the observed lower bound is higher than the theoretical bound because the derivation uses a loose Markov inequality and tokens outside the top 20% also gain entropy.Source: §B.3 · pp. 24–26

C.1.1 TRUNCATED GAUSSIAN IS MORE STABLE THAN TANH GAUSSIAN

• With learned δ, the Tanh Gaussian suffers catastrophic training failures, while the Truncated Gaussian is more stable and has minimal performance difference between learned and constant δ.The experiments used SAC on four DMC Dog and Humanoid tasks, five seeds, and 1M steps; the authors therefore use constant δ = 0 for simplicity.Source: Figure 9 · p. 26, §C.1.1 · pp. 26–27

C.1.2 BATCH-LEVEL ENTROPY REGULARIZATION V.S. STATE-LEVEL ENTROPY REGULARIZATION

• State-level and batch-level entropy regularization achieve similar performance and both outperform SAC on DMC Dog and Humanoid locomotion tasks.The authors attribute the minimal difference to these tasks requiring exploration but not high precision.Source: Figure 10 · p. 27, §C.1.2 · p. 27

C.1.3 SAC-ERA ON MUJOCO GYM ENVIRONMENTS

• SAC-ERA shows only slight advantages over SAC on Mujoco Gym environments, where relatively low action-space dimensionality may reduce the impact of constraint schemes.The evaluated environments include HalfCheetah-v4, Hopper-v4, Walker2d-v4, Ant-v4, Humanoid-v4, and Swimmer-v4.Source: §C.1.3 · p. 27, Figure 11 · p. 28

C.1.4 APPLICABILITY OF LLM RL TECHNIQUES TO CONTINUOUS CONTROL

• On HalfCheetah-v4, Selective High-Entropy Training and Clip-Higher failed to provide higher policy entropy than standard PPO and yielded no significant or only marginal gains.The authors conclude that LLM RL techniques do not necessarily transfer to continuous control; their results were not comparable to ERA.Source: Figure 12 · p. 28, §C.1.4 · p. 28

C.1.5 COMPARING ERA WITH OTHER MAXIMUM ENTROPY RL APPROACHES

• On MuJoCo tasks, PPO-ERA outperforms EAPO and SAC-ERA outperforms MNSE, based on curves reported in the original papers.Neither comparison extends to DMC Suite or HumanoidBench because the other papers reported no results there; EAPO adds an entropy critic and MNSE an inverse dynamics model, while ERA requires no additional networks.Source: Figure 13 · p. 29, Figure 14 · p. 29, §C.1.5 · pp. 29–30

C.1.6 TIME COST OF ERA IN CONTINUOUS CONTROL

• 6%: ERA adds about 6% average training-time overhead on HumanoidBench when comparing FastTD3 and FastSAC-ERA.The paper frames this overhead as a trade-off for improved exploration and sample efficiency.Source: Figure 15 · p. 30, §C.1.6 · p. 30

C.2.1 COMPARING ERA WITH COMMON REGULARIZATION TECHNIQUES

• The CIFAR-10 comparison evaluates ERA against a baseline, label smoothing factors 0.2 and 0.3, and dropout rates 0.1, 0.2, and 0.3.The main experiment uses timm defaults: label smoothing 0.1 and no dropout; the supplied passage gives no comparative results.Source: §C.2.1 · p. 30
• ERA consistently improves CIFAR-10 Top-1 and Top-5 accuracy more than dropout or label smoothing, whose stronger smoothing harms performance and dropout yields only marginal gains.By constraining entropy while allowing uncertainty to be allocated freely among dimensions, ERA can adapt to the data’s intrinsic structure and improve generalization.Source: §C.2.1 · p. 30, Figure 16 · p. 31

C.2.2 TIME COST OF ERA IN IMAGE CLASSIFICATION

• ERA introduces almost no training-time difference from the original ResNet-50 on CIFAR-10 when data is well-parallelized.The comparison reports average training time across three machines, each equipped with four NVIDIA A40 GPUs.Source: §C.2.2 · pp. 30–31, Figure 17 · p. 31

C.3.1 DETAILED ENTROPY ANALYSIS

• After ωlow is decreased, entropy rapidly drops and stabilizes at the second-level lower bound, showing ERA enforces a non-trivial entropy floor.The entropy curve is shown in Figure 18.Source: §C.3.1 · p. 31, Figure 18 · p. 32
• With ωlow = 0.45 and ωhigh = 3.0, the bottom 80% of tokens have nearly zero entropy, while responses with Hresp > ωhigh quickly fall to zero and those with Hresp < ωlow remain within [0, 0.06].The authors report that ERA adaptively raises overly low-entropy responses to a moderate level; Figure 19 plots token entropy and response proportions.Source: §C.3.1 · p. 31, Figure 19 · p. 32

C.3.2 ABLATION STUDY ON ENTROPY BOUND

• Without the upper entropy bound ωhigh, model entropy explodes in a very short time, indicating that the upper constraint is essential for controlling training entropy.Figure 20 compares ERA with and without ωhigh.Source: §C.3.2 · p. 31, Figure 20 · p. 32

C.3.3 TIME COST OF ERA IN LLM

• 5.6% is the reported first-step time-cost overhead for ERA when computing response-token log probabilities; whole-step overhead is described as even smaller.Figure 21 compares GRPO and ERA using timing_s/old_log_prob; other training components are unaffected.Source: §C.3.3 · pp. 31, 33, Figure 21 · p. 33

C.4 TRAINING CURVES OF CONTINUOUS CONTROL TASKS

• The supplied captions show training curves for OBAC and OBAC-ERA on HumanoidBench, TD-MPC2 and TD-MPC2-ERA on DMC, and PPO and PPO-ERA on Mujoco Gym.They also identify FastTD3 and FastSAC-ERA on HumanoidBench, and SAC and SAC-ERA on HumanoidBench and DMC; the captions do not state comparative outcomes.Source: Figure 22 · p. 33, Figure 23 · p. 34, Figure 24 · p. 34

D THE USE OF LARGE LANGUAGE MODELS IN THIS PAPER

• LLMs were used for proofreading, language polishing, title inspiration, and occasional debugging, but not for methodology design, experiments, or result interpretation.The authors state that human authors conceived and developed the core contributions and reviewed and verified the content.Source: D THE USE OF LARGE LANGUAGE MODELS IN… · p. 33