PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding and Improving Hyperbolic Deep Reinforcement Learning

Timo Klein, Thomas Lang, Andrii Shkabrii, Alexander Sturm, Kevin Sidak, Lukas Miklautz, Claudia Plant, Yllka Velaj, Sebastian Tschiatschek

arXiv:2512.14202v2cs.LGcs.AI

TL;DR

Formal explanations of optimization failures in hyperbolic deep RL are lacking. The paper analyzes gradients in hyperbolic models and introduces Hyper++; its analysis links large-norm embeddings to PPO trust-region breakdowns.

  • Problem

    Formal analysis of why optimization fails in hyperbolic deep RL is lacking, despite optimization instabilities limiting its broader adoption.

    Source: §1 · p. 2

  • Method

    The paper analyzes core operations in the Poincaré Ball and Hyperboloid models and combines RMSNorm, learned feature scaling, a hyperboloid model, and categorical value loss in Hyper++.

    Source: §1 · p. 2

  • Results

    Large-norm embeddings destabilize gradients and are linked to PPO trust-region breakdowns.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The authors present Hyper++ as a way to stabilize hyperbolic deep RL training.

    Source: §7 · p. 10

    The authors cannot rule out unethical downstream uses, though such uses would require significant extensions and modifications.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

The exponential volume growth of hyperbolic geometry can embed the hierarchical relationships between states in reinforcement learning (RL) with far less distortion than Euclidean space. However, hyperbolic deep RL faces severe optimization challenges, and formal analysis of why optimization fails is lacking. We identify key factors that determine the success and failure of training hyperbolic deep RL agents. By analyzing the gradients of core operations in the Poincar\'e Ball and Hyperboloid models of hyperbolic geometry, we show that large-norm embeddings destabilize gradient-based training, leading to trust-region violations in proximal policy optimization (PPO). Based on these insights, we introduce Hyper++, a new hyperbolic deep RL agent that consists of three components: (1) feature regularization guaranteeing bounded norms while avoiding the curse of dimensionality from clipping; (2) a categorical value loss for stable critic training; and (3) a more optimization-friendly formulation of hyperbolic network layers. On ProcGen, we show that Hyper++ guarantees stable learning, outperforms prior hyperbolic agents, and reduces wall-clock time by approximately 30%. On Atari-5 with Double DQN, Hyper++ strongly outperforms Euclidean and hyperbolic baselines. We release our code at https://github.com/Probabilistic-and-Interactive-ML/hyper-rl.

1 INTRODUCTION

Hierarchical decision processes have exponentially branching state structures that hyperbolic geometry can embed with low distortion. The paper analyzes why hyperbolic RL training destabilizes and introduces HYPER++ components to improve stability and performance.

1 INTRODUCTION

  • Hyperbolic geometry’s exponential volume growth aligns with tree-like RL state branching, whereas Euclidean spaces’ polynomial volume growth forces hierarchical relationships to be distorted.The introduction links this geometric mismatch to deep RL’s data inefficiency.Source: §1 · p. 1
  • HYPER++ improves test return by 52% (PPO+ProcGen), reduces forward pass time by 30%, and transfers gains to Atari-5 Double DQN and ProcGen Phasic Policy Gradient.The introduction reports better performance, speed, and generality than prior hyperbolic agents.Source: Figure 1 · p. 1, §1 · p. 2
  • Formal gradient analysis finds that growing embedding norms destabilize training in both the Poincaré Ball and Hyperboloid models, causing PPO trust-region violations despite clipping.Existing SpectralNorm stabilization is insufficient without severely limiting network capacity.Source: §1 · p. 2
  • HYPER++ combines RMSNorm with a learned scaling layer to bound embedding norms without sacrificing capacity, and uses the Hyperboloid model to prevent large gradients from propagating.This addresses the stability-capacity trade-off of SpectralNorm and instabilities inherent to the Poincaré Ball model.Source: §1 · p. 2
  • Replacing MSE regression with a categorical value loss aligns critic outputs with hyperbolic multinomial logistic regression’s hyperplane-distance geometry and stabilizes learning under nonstationary targets.This is one of HYPER++’s three targeted components.Source: §1 · p. 2

2 BACKGROUND

The background introduces the MDP and PPO framework, then outlines hyperbolic representations and the models used to compute policy and value outputs.

2.1 REINFORCEMENT LEARNING

  • The paper formalizes RL as a discrete MDP and defines its objective as maximizing expected discounted future rewards, with γ controlling the value of future rewards (Sutton & Barto, 2018).The MDP comprises state and action spaces, a policy, a reward function, and a transition kernel.Source: §2.1 · p. 2
  • PPO replaces TRPO’s hard trust-region constraint with a clipped surrogate objective that limits policy probability ratios to [1 − ϵ, 1 + ϵ] (Schulman et al., 2017; Schulman et al., 2015).The clipping acts as an unconstrained proxy for TRPO’s KL-divergence trust region.Source: §2.1 · pp. 2–3

2.2 HYPERBOLIC REPRESENTATION LEARNING

  • The Poincaré Ball and Hyperboloid are isometrically equivalent, negatively curved Riemannian models; hyperbolic encoding maps tangent-space vectors onto the manifold with an exponential map.The exponential map at the origin follows a geodesic from the origin to the resulting manifold point.Source: §2.2 · p. 3
  • Hyperbolic multinomial logistic regression computes class probabilities from logits defined by signed distances to learnable margin hyperplanes (Lebanon & Lafferty, 2004; Shimizu et al., 2021; Bdeir et al., 2024).The model uses learnable parameters for the hyperplane normal and shift, whose definitions depend on the hyperbolic model.Source: §2.2 · p. 3

3 DIAGNOSING ISSUES WITH HYPERBOLIC PPO AGENTS

Hyperbolic PPO can become unstable when geometry-dependent gradients grow or vary with large feature norms, contributing to policy shifts and trust-region violations. The analyses identify norm regularization as important for stable training, while showing that the specific failure mechanisms depend on the hyperbolic layers used.

3.1 PPO OPTIMIZATION

  • Unregularized hyperbolic PPO loses entropy and has larger update KL and more trust-region violations, while Hyper++ achieves lower update KL and less clipping without SpectralNorm or conformal-factor instability.The instability coincides with rising entropy variance across batch states, which can drive large policy updates and cross-state interference because PPO constrains ratios only on sampled states.Source: Figure 2 · p. 4, §3.1 · p. 4

3.2 GRADIENT ANALYSIS PRELIMINARIES

  • The Poincaré Ball conformal factor can destabilize gradients, motivating curvature-aware analysis of core hyperbolic layers and maps.For the Poincaré MLR, backpropagation through the exponential map yields vanishing gradients near the boundary; PPO’s sampled-state constraints also leave unseen-state policy changes unconstrained.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.3 GRADIENT ANALYSIS FOR HYPERBOLIC NETWORK++ MLR

  • In HNN++ MLR, a denominator term causes gradient explosion near the Poincaré Ball boundary, and its exponential-map Jacobian is sensitive to growing Euclidean norms.Clipping hyperbolic embeddings can shift decision boundaries and cause performance plateaus; S-RYM scaling moderates the norms and directional variability, showing why pre-layer Euclidean feature regularization is necessary for stable hyperbolic PPO.Source: §3.3 · p. 5

3.4 GRADIENT ANALYSIS FOR HYPERBOLOID MLR

  • The Hyperboloid MLR avoids the Poincaré Ball’s conformal-factor and feature-norm gradient instabilities, but its exponential map can still destabilize gradients when Euclidean feature norms grow.Policy breakdown and large-norm encoder gradients depend on the actor and critic’s hyperbolic layers, while noisy actor-critic gradients—particularly from the critic—can exacerbate numerical instability.Source: §3.4 · p. 5

4 STABILIZING HYPERBOLIC DEEP RL

HYPER++ addresses distinct sources of instability in hyperbolic deep RL with categorical value learning, Hyperboloid MLR, RMSNorm, and learned feature scaling. These components stabilize the loss derivative, hyperbolic-layer Jacobian, and exponential-map Jacobian, respectively.

4 STABILIZING HYPERBOLIC DEEP RL

  • Categorical value loss stabilizes critic gradients and fits MLR hyperplane-distance outputs to discrete value bins, while Hyperboloid MLR and RMSNorm with feature scaling target separate Jacobian instabilities.The categorical loss addresses the loss derivative, Hyperboloid MLR the hyperbolic-layer Jacobian, and RMSNorm with feature scaling the exponential-map Jacobian.Source: §4 · p. 6

4.1 REGULARIZATION

  • RMSNorm before the encoder’s final activation bounds embeddings while preserving other layers’ expressivity; unlike SpectralNorm on every layer, it avoids global smoothness constraints and power-iteration overhead.For common 1-Lipschitz activations, the resulting bounds are dimension-independent; RMSNorm is preferred over LayerNorm to avoid mean-centering hierarchical embeddings.Source: §4.1 · p. 6

4.2 LEARNED EUCLIDEAN FEATURE SCALING

  • For d = 32, learned feature scaling yields approximately 1.2 × 10^3 more usable volume by expanding the ball radius from 0.76 to 0.95 while preventing conformal-factor explosion.The radius bound of 0.76 can sharply restrict hyperbolic representational capacity because d-dimensional ball volume scales exponentially with radius.Source: §4.2 · p. 7

4.3 HYPERBOLOID MODEL

  • Although the Hyperboloid avoids conformal-factor instability, operations become ill-conditioned far from the origin, so RMSNorm and learned scaling bound its time component and thereby its space component.The strategy transfers from the Poincaré Ball via their isometry and is applied before the last Euclidean activation.Source: §4.3 · p. 7

5 EXPERIMENTS

Experiments evaluate HYPER++ on ProcGen with PPO and PPG, ablate its components, and test DDQN performance on Atari-5.

5.2 ABLATION STUDIES

  • Removing RMSNorm and 1/d feature scaling causes complete learning failure, while SpectralNorm also fails to learn and learned scaling supports RMSNorm’s effectiveness.The failure shows large embedding norms and near-zero encoder final-layer gradients; both SpectralNorm configurations fail, underscoring RMSNorm’s role in bounded feature norms.Source: §5.2 · p. 9
  • Replacing HL-Gauss with MSE degrades performance nonuniformly across games, C51 performs worse than MSE, and the Poincaré ball yields a modest drop.The reported comparison aligns with Farebrother et al. (2024), who also find that HL-Gauss does not consistently improve performance across environments.Source: §5.2 · p. 9
  • Euclidean agents improve with RMSNorm and the full regularization combination, but the Hyperboloid HYPER++ achieves the best overall ProcGen test IQM.Euclidean+Categorical performs worse than Euclidean+MSE; Euclidean with the full method reaches IQM 0.35, slightly above Poincaré HYPER++ at 0.34, while Hyperboloid HYPER++ reaches 0.40.Source: Figure 8 · p. 9, §5.2 · p. 9

5.1 PROCGEN; 5.3 ATARI DDQN

  • HYPER++ outperforms baselines on Atari-5 and outperforms Euclidean and prior hyperbolic agents on ProcGen with PPG.On ProcGen with PPO, it also wins head-to-head against Hyper+S-RYM in 8/16 training games and 11/16 test games.Source: Figure 6 · p. 8, §5.1 · p. 8, Figure 9 · p. 10

5.3 ATARI DDQN

  • On Atari-5, HYPER++ gains most on NAMETHISGAME and Q*BERT, while its strong initial performance on PHOENIX plateaus; target-update choice has only minor effects on NAMETHISGAME.The PHOENIX plateau is consistent with plasticity loss as a confounding factor; the Atari-5 evaluation uses 10M training steps and five seeds.Source: §5.3 · p. 10

6 RELATED WORK

Prior work has developed hyperbolic neural architectures and analyzed optimization and numerical stability, while this paper studies optimization problems for hyperbolic networks in RL. It also connects PPO regularization and categorical value learning to its hyperbolic agent.

Hyperbolic deep learning

  • Hyperbolic learning has advanced from early neural networks and Riemannian optimization to fully hyperbolic architectures, while this work targets optimization problems within RL.Related work also includes RL adaptation of hyperbolic networks, reduced parameter redundancy in Poincaré Ball MLR, and independent analyses of optimization and numerical stability (Ganea et al., 2018; Bécigneul & Ganea, 2019; Cetin et al., 2023; Shimizu et al., 2021; Chen et al., 2022; Bdeir et al., 2024; Mishne et al., 2023; Guo et al., 2022).Source: §6 · p. 10

Reinforcement learning

  • PPO remains an actively studied RL algorithm, and this work uses RMSNorm regularization to avoid interference with hyperbolic representations.Prior deep RL work widely adopts LayerNorm, whereas this agent uses RMSNorm (Schulman et al., 2017; Andrychowicz et al., 2021; Moalla et al., 2024; Henderson et al., 2018; Ba et al., 2016; Lyle et al., 2023; Nauman et al., 2024b; Lee et al., 2025; Gallici et al., 2025; Zhang & Sennrich, 2019).Source: §6 · p. 10
  • This work extends categorical value-learning objectives, previously used to stabilize value-function learning, to hyperbolic PPO.Prior approaches include categorical objectives from Bellemare et al. (2017), Schrittwieser et al. (2020), Imani & White (2018), and Farebrother et al. (2024).Source: §6 · p. 10

7 LIMITATIONS AND CONCLUSION … A.3 HYPERBOLOID EXPONENTIAL MAP GRADIENTS

This section summarizes the paper’s conclusions and open questions about hyperbolic deep RL, and details gradient analyses of hyperbolic maps and network layers, including vanishing and potentially unstable gradients.

7 LIMITATIONS AND CONCLUSION

  • HYPER++ combines RMSNorm with learned feature scaling and categorical value loss, improving ProcGen performance and reducing wall-clock time versus hyperbolic PPO agents, with strong gains on Atari and DDQN.The conclusion describes these findings as indicating applicability beyond PPO.Source: §7 · p. 10
  • The analysis leaves representation structure, environment suitability, and interactions between geometric choices and deep RL algorithm designs for future work.The work focuses on training dynamics and how hyperbolic deep RL learns.Source: §7 · p. 10

Appendix

  • The code is publicly available, and the appendices provide gradient-analysis derivations and proofs, implementation details, and discussion of differences from existing results.Appendix D covers architecture, hyperparameters, implementation details, and hardware; Appendix C.3 discusses differences in results to existing works.Source: REPRODUCIBILITY · p. 11, Appendix · p. 16, Table 1 · p. 16

7 LIMITATIONS AND CONCLUSION

  • The authors report using LLMs for literature search, summarization, repetitive coding support, and writing iteration, with outputs reviewed for accuracy.They state that LLMs were not used for idea generation, experimental design, or proofs, and that mathematical expressions were derived independently.Source: USAGE OF LARGE LANGUAGE MODELS (LLMS) · p. 11
  • The authors say the work has no direct ethical implications, while noting that downstream unethical uses cannot be ruled out and would require significant extensions or modifications.They point to RL and PPO use in training LLMs and the suitability of hyperbolic embeddings for text data as relevant context.Source: ETHICS STATEMENT · p. 11

A.1 POINCAR´E EXPONENTIAL MAP GRADIENTS

  • A.1 builds on Guo et al. (2022) to derive the Jacobian of the Poincaré Ball exponential map at the origin.The passage then introduces the derivation of the second term.Source: §A.1 · p. 17
  • The Poincaré exponential-map Jacobian decays, while its directional term can vanish faster.The supplied passages do not preserve the rates after the O notation.Source: §A.1 · p. 17

A.2 HYPERBOLIC NETWORKS++ GRADIENTS

  • Although a factor in the Hyperbolic Networks++ gradient cannot blow up, gradients can become unstable for inputs near the Poincaré Ball boundary.The passage identifies the bounded factor and points to gradients in Equation 18 for samples close to the boundary.Source: §A.2 · p. 18

A.3 HYPERBOLOID EXPONENTIAL MAP GRADIENTS

  • The Hyperboloid exponential-map Jacobian is derived by combining the time- and space-component derivatives.The time-component derivative is described as a d + 1-dimensional vector with a zero first element.Source: §A.3 · p. 18

A.4 PROOFS … C.2 ATARI

The appendix supplies mathematical proofs, background on policy optimization, Double DQN, and hyperbolic classifiers, and descriptions of the ProcGen and Atari evaluation protocols. The supplied passages describe benchmark setups but do not report benchmark results.

A.4 PROOFS / 2. Then,

  • The proofs use the Lipschitz property of f and the induced matrix norm; the special ReLU case follows because ReLU is 1-Lipschitz and ReLU(0) = 0.The following proof passage begins Proposition 4.2 by bounding the normalized feature-vector norm, then bounds the conformal factor using the exponential map and a bound on the norm.Source: §A.4 · p. 19, §2 · p. 19

B.1 TRUST-REGION POLICY OPTIMIZATION (TRPO)

  • TRPO maximizes a surrogate cumulative-reward objective with an average KL-divergence constraint that keeps the updated policy close to the data-generating policy.Its theory guarantees monotonic improvement, and the passage notes practical deep-network approximations tend to retain that property (Schulman et al., 2015).Source: §B.1 · p. 19

B.2 DOUBLE DEEP Q-NETWORK

  • Double DQN separates next-action selection by the online network from value evaluation by the target network, mitigating the overestimation bias of standard DQN.Standard DQN uses replayed transitions and a periodically updated target network, but its shared selection and evaluation under noise can cause systematic overestimation (Sutton & Barto, 2018); DDQN uses parameters θ for selection and φ for evaluation (van Hasselt et al., 2016).Source: §B.2 · p. 20

B.3 HYPERBOLIC MULTINOMIAL LOGISTIC REGRESSION

  • Hyperbolic multinomial logistic regression predicts class probabilities from signed distances to margin hyperplanes, parameterized by a normal vector and shift with d + 1 parameters per hyperplane.Parameters remain in Euclidean space for established optimization algorithms, with normals parallel-transported to the hyperbolic tangent space (Shimizu et al., 2021; Bdeir et al., 2024).Source: §B.3 · p. 20

B.3.1 POINCAR´E BALL MLR

  • The Poincaré Ball MLR formulation uses Möbius addition and subtraction to express its layer.The passage identifies this as the Poincaré Ball MLR layer (Shimizu et al., 2021).Source: §B.3.1 · p. 20

B.3.2 HYPERBOLOID MLR

  • The Hyperboloid MLR represents tangent parameters in d dimensions by omitting the constrained time component, and distinguishes the space and time components of x.The passage identifies this as the Hyperboloid MLR layer.Source: §B.3.2 · pp. 20–21

C ENVIRONMENTS / C.1 PROCGEN

  • ProcGen uses 64 × 64 × 3 RGB observations, 15 discrete actions, and training on easy levels 0–199, with end-of-training evaluation on 100 parallel train and test environments.Scores are normalized per run before aggregation; Figure 10 visualizes the 16 games.Source: §C.1 · p. 22, Figure 10 · p. 22

C.2 ATARI

  • Atari observations are grayscaled, downsampled to 84 × 84, and frame-stacked; rewards are clipped to {−1, 0, +1}, with randomized no-op resets and action repeat.The Arcade Learning Environment has up to 18 discrete joystick/button combinations; most games use a subset.Source: §C.2 · p. 22

C.3 EVALUATION DIFFERENCES WITH EXISTING WORKS · D EXPERIMENTAL DETAILS · D.1 HARDWARE & RUNTIME

The authors attribute reproduction differences to seeding, implementation, software versions, and evaluation choices, and report the hardware and conditions used for runtime measurements.

C.3 EVALUATION DIFFERENCES WITH EXISTING WORKS

  • The paper builds on Cetin et al. (2023), but the authors struggled to reproduce their results and attribute this mainly to three differences.Source: §C.3 · p. 22
  • Their source code does not use seeding; because deep RL is seed-dependent, the authors find exact reproduction impossible (Henderson et al., 2018; Agarwal et al., 2021).Source: §C.3 · p. 22
  • The authors use different hyperbolic-geometry operations, which may affect results, an issue noted in the hyperbolic deep learning community (Katsman & Gilbert, 2025).Source: §C.3 · p. 22
  • They also use different versions of PyTorch and Python.Source: §C.3 · p. 22
  • Their evaluation protocol differs slightly, and they use PyTorch evaluation mode before generating results for their agents.Source: §C.3 · p. 22
  • The authors hope releasing their complete code will advance reproducibility in hyperbolic deep RL.Source: §C.3 · p. 22

D.1 HARDWARE & RUNTIME

  • Experiments used Nvidia A100 GPUs; a single 40GB GPU could train up to four ProcGen agents in parallel.Source: §D.1 · p. 23
  • Runtime measurements average 100 ProcGen forward passes and five NameThisGame seeds; full-run timing may be confounded by episode length, which better agents can increase.Source: §D.1 · p. 23

D.2 NETWORK ARCHITECTURE · D.3 HYPERPARAMETERS · E ADDITIONAL RESULTS

The supplied passages describe Hyper++’s ProcGen and Atari network architectures and summarize its hyperparameter choices and sources.

D.2 NETWORK ARCHITECTURE

  • For ProcGen, Hyper++ builds on the Impala-ResNet used by Cetin et al. (2023), with its Hyper++-specific hybrid architecture illustrated in Figure 11.The base architecture is attributed to Espeholt et al. (2018).Source: §D.2 · p. 23, Figure 11 · p. 24
  • The ProcGen network applies RMSNorm before scaling Euclidean features and uses TanH instead of ReLU as its penultimate activation.Source: §D.2 · p. 23
  • It also applies learned feature scaling before the exponential map and uses the Hyperboloid rather than the Poincaré Ball.Source: §D.2 · p. 23
  • For Atari, Hyper++ uses NatureCNN with the same modifications as in ProcGen.NatureCNN is attributed to Mnih et al. (2015).Source: §D.2 · p. 23

D.3 HYPERPARAMETERS

  • The paper lists PPO hyperparameters for ProcGen, DDQN hyperparameters for Atari, PPG hyperparameters for ProcGen, and additional Hyper++ parameters in separate tables.Source: §D.3 · p. 25, Table 4 · p. 25, Table 5 · p. 25
  • ProcGen uses latent dimension d = 64 versus d = 32 for Hyper+S-RYM; learned scaling is set to α = 0.95 without tuning across experiments.HL-Gauss uses the default parameters specified by Farebrother et al. (2024).Source: §D.3 · p. 25
  • DDQN and PPG use cleanRL hyperparameters and preprocessing, while RMSNorm and LayerNorm omit affine parameters to avoid overfitting to the training set.The passage attributes cleanRL to Huang et al. (2022) and the overfitting concern to Xu et al. (2019).Source: §D.3 · p. 25

E.1 FULL PPO RESULTS · E.2 ADDITIONAL ABLATION METRICS

The appendix documents PPO ProcGen full and ablation results across training and test evaluations, then adds diagnostic metrics on how RMSNorm, value-loss choice, and learned feature scaling affect training.

E.1 FULL PPO RESULTS

  • The full PPO ProcGen results include train and test learning curves and mean ± std result tables.The supplied captions identify the evaluation views but do not provide curve values or table entries.Source: Figure 12 · p. 26, Figure 13 · p. 27, Table 7 · p. 27
  • The PPO ProcGen ablation results include separate train and test tables reporting mean ± std.The captions do not expose the ablation entries or their comparative outcomes.Source: Table 9 · p. 29, Table 10 · p. 30

E.2 ADDITIONAL ABLATION METRICS

  • Without RMSNorm, agents suffer growing embedding norms and vanishing encoder gradients.The caption attributes this pattern to the −RMSNorm variant.Source: Figure 14 · p. 31
  • Using MSE instead of HL-Gauss leads to larger initial encoder gradients because MSE gradients scale with the loss (Imani & White, 2018).The caption identifies this as the +MSE variant.Source: Figure 14 · p. 31
  • Removing learned feature scaling produces the largest embedding norms and gradients, which RMSNorm’s gradient variance normalization quickly compensates for (Zhang & Sennrich, 2019).The caption identifies this as the −Scaling variant.Source: Figure 14 · p. 31

E.3 GRADIENT AND LOSS VARIANCE ANALYSIS … E.8 PHASIC POLICY GRADIENT

Across these supplementary analyses, HYPER++ outperforms baselines under PPG, while categorical critic loss yields smaller, lower-variance gradients despite higher loss and variance; the remaining sections report additional benchmark and ablation analyses.

E.3 GRADIENT AND LOSS VARIANCE ANALYSIS

  • Despite higher loss values and variance, HL-Gauss categorical loss yields smaller, lower-variance penultimate-layer gradients than MSE.Loss and variance were averaged over all runs; gradient statistics use the final 25% of training steps.Source: §E.3 · p. 31, Table 11 · p. 31, Figure 15 · p. 32

E.4 LEARNING CURVES FOR ATARI GAMES

  • HYPER++ outperforms the baselines on all Atari-5 environments, with particularly strong gains in NAMETHISGAME and QBERT.Learning curves are averaged over five seeds.Source: Figure 16 · p. 32

E.5 POLYAK AVERAGING EXPERIMENT

  • On NAMETHISGAME, Polyak averaging versus hard target-network replacement does not meaningfully affect algorithm performance.Runs are averaged over five seeds, with one standard deviation shown as error.Source: Figure 17 · p. 33

E.6 OFF-BATCH PPO METRICS

  • Off-batch and updated-batch PPO KL divergence and clipping-fraction trajectories are highly similar, but off-batch KL divergence has noticeably higher variance.The off-batch metrics are measured on randomly sampled on-policy states.Source: Figure 18 · p. 33

E.7 ARCHITECTURE ABLATIONS

  • The ProcGen PPO architecture ablations examine latent dimension and learned scaling α, with test results averaged over six seeds.The supplied passage identifies the ablation scope but does not provide outcome comparisons.Source: Table 12 · p. 34

E.8 PHASIC POLICY GRADIENT

  • On PPG ProcGen, HYPER++ outperforms both baselines in train and test performance and beats Hyper+S-RYM by 53% in IQM test reward.Hyper+S-RYM fails to outperform the Euclidean baseline, while HYPER++ does; results are averaged over six seeds.Source: Table 13 · p. 35, Table 14 · p. 35, §E.8 · p. 35

Related papers

Loading 2512.14202v2…