Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL

1 INTRODUCTION

• Euclidean agents can learn maze route structure, but may need substantial depth to do so; hyperbolic geometry supplies that structure as an inductive bias.The paper frames this as a potential substitute for substantial network depth.Source: §1 · p. 2
• Hyperbolic CRL replaces Euclidean embedding distances in CRL’s contrastive objective with hyperbolic geodesic distances, at a small, fixed computational cost.This gives agents an inductive bias toward tree-like route structure.Source: §1 · p. 2
• Maze route distances are approximately Euclidean at short range but increasingly tree-like at long range, unlike straight-line distance, which ignores walls.The mismatch can make straight-line proximity misleading and lead an agent down dead ends.Source: §1 · pp. 1–2
• At fixed depth, hyperbolic agents outperform Euclidean agents on most tasks; depth-64 agents outperform Euclidean baselines up to 16× deeper while training in at least 7× less wall-clock time.Source: §1 · p. 2

2 BACKGROUND

• A goal-conditioned MDP samples goals and trains a goal-conditioned policy to maximize expected return for reaching them.The goal-relevant part of each state is extracted by a function f, and the reward is the density of achieving the goal at the next step.Source: §2 · p. 2
• CRL estimates a policy’s discounted visitation density over goals by classification rather than temporal-difference learning.The Q-function is described as a density over goals, with trajectories generated by the policy defining the relevant densities.Source: §2 · p. 2
• CRL’s critic scores state-action and goal embeddings through a similarity function and trains with InfoNCE on future achieved goals as positives and other trajectories’ goals as negatives.The implementation replaces negative Euclidean distance with negative hyperbolic geodesic distance.Source: §2 · p. 3

3 RELATED WORK

• Earlier work scaled CRL to networks up to 1024 layers, while this work keeps the CRL objective and changes representation geometry.Prior scaling approaches include regularized residual critics and mixture-of-experts layers.Source: §3 · p. 3
• Quasimetric methods model asymmetric temporal distances but retain Euclidean embedding geometry; prior quasimetric CRL performed poorly on maze-like tasks.Hyperbolic geodesic distances preserve symmetry while changing curvature.Source: §3 · p. 3
• Figure 2 compares maze tree-likeness across horizons and tests how embedding distances order locations by route distance.Its panels also compare how often critics assign a larger embedding distance to the longer of two routes with similar straight-line distance.Source: Figure 2 · p. 4
• Prior hyperbolic RL work studied learned representations or benchmark performance, but did not test whether the environment itself is tree-like.The paper states it is the first to use hyperbolic representations in goal-conditioned RL.Source: §3 · p. 4

4 CASE STUDY: ANALYZING THE GEOMETRY OF A DECEPTIVE MAZE

• The study compares shortest-path route distance within the maze with Euclidean straight-line distance that ignores walls.Critics use negative embedding distance as energy, with Euclidean distance or hyperboloid geodesic distance, respectively.Source: §4 · p. 4
• At longer horizons, Ant U5-Maze route distances become increasingly tree-like while straight-line distances show the opposite trend.The maze is locally close to flat and tree-like globally; hyperbolic geometry is approximately Euclidean at small scales and tree-like at larger ones.Source: §4 · pp. 4–5
• Hyperbolic critics order goals by route distance more consistently than Euclidean critics, including pairs whose straight-line distances are similar.For the latter comparison, selected goal pairs differ in route distance by at least 10 world units.Source: §4 · p. 5
• In Ant U5-Maze, Euclidean critic gradients tend toward walls, whereas hyperbolic critics do not show this bias.The Euclidean critic also assigns high values across walls; the hyperbolic critic's values increase along the route toward the goal.Source: §4 · p. 5, Figure 3 · p. 5
• At fixed depth, hyperbolic agents outperform Euclidean agents, with particularly pronounced improvements in hard navigation environments such as Humanoid mazes.Figure 4 reports depth-16 agents and means across 10 seeds with 95% bootstrapped confidence intervals.Source: Figure 4 · p. 6

5 EXPERIMENTS

• The benchmark follows Wang et al. (2025), substitutes Ant Ball for Ant Big Maze, and uses a different Humanoid evaluation protocol.Performance is measured in steps at goal with per-environment means and 95% bootstrap confidence intervals; the main experiments use depth 16.Source: §5 · p. 6

5.1 MAIN RESULTS

• At depth 64, hyperbolic agents substantially outperform depth 1024 and depth 512 Euclidean agents on Humanoid U-Maze and Humanoid Big Maze.The strongest gains occur on the suite’s two hardest tasks.Source: §5.1 · p. 6

5.2 BEHAVIORAL EVIDENCE OF INVALID SHORTCUTS IN EUCLIDEAN AGENTS

• Euclidean agents vault over walls and often fall, while hyperbolic agents follow corridors and stay at goals; goals behind the inner wall remain difficult for both.On Humanoid U-Maze, the depth 64 hyperbolic agent stays at the goal for 240 steps, versus around 100 steps for depth 1024 Euclidean agents.Source: §5.2 · p. 7, Figure 5 · p. 7

5.3 HYPERBOLIC REPRESENTATIONS IMPROVE STITCHING

• At every tested depth, hyperbolic agents stay longer at the goal than Euclidean agents on Ant U-Maze, with the largest gap at depth 4.At depth 16, hyperbolic agents exceed the mean of depth 64 Euclidean agents; evaluation uses start–goal pairs absent from training.Source: Figure 6 · p. 7, §5.3 · pp. 7–8

5.4 PERFORMANCE SENSITIVITY ACROSS CURVATURE AND NETWORK DEPTH

• Hyperbolic agents benefit from deeper Euclidean encoders on three of four tasks, but depth narrows the gap on Arm Push Easy and Humanoid remains below Euclidean agents from depth 16.On Ant U4-Maze, a 4-layer hyperbolic agent matches 16- or 32-layer Euclidean agents.Source: Figure 8 · p. 8, §5.4 · p. 8
• Intermediate curvature works best on mazes, while Humanoid performance increases as curvature decreases; c = 1 performs worse than c = 0.5 on all four tested tasks.On Humanoid, c = 0.01 matches Euclidean performance; Arm Push Easy performs best at c = 0.1.Source: Figure 7 · p. 8, §5.4 · p. 8

5.5 OFFLINE EXPERIMENTS

• Hyperbolic agents achieve higher mean success rates on all eight OGBench tasks, with the largest relative gains on giant mazes, which neither agent reliably solves.The hyperbolic agent improves clearly on three of four navigation tasks and three of four stitching tasks.Source: Figure 9 · p. 9, §5.5 · p. 9

6 CONCLUSION

• Depth-64 hyperbolic agents outperform 1024-layer Euclidean agents on hard humanoid mazes, with gains largest when walls make route and straight-line distances diverge.The authors suggest matching representation geometry to environment structure can alleviate much of the depth needed for long-horizon goal reaching.Source: §6 · p. 9
• Dynamic curvature selection is challenging amid deep-RL noise, and symmetric geodesic distance prevents the critic from representing one-way transitions.The authors identify combining hyperbolic geometry with quasimetric structure as a natural extension.Source: §6 · p. 9
• The authors used generative AI for literature search, coding, experiments, writing, and figure generation, while stating that they designed the core idea and studies themselves.They report reviewing code and verifying that baselines reproduce literature results.Source: AI USE STATEMENT · p. 10
• The work evaluates simulated locomotion and manipulation benchmarks without human data and notes that downstream misuse would require substantial extensions beyond the evaluated setting.The authors report that code is included in the submission and will be made public upon acceptance.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

B.1 HYPERBOLIC SPACE OPERATIONS

• The hyperboloid uses its first coordinate as the time component and the remaining coordinates as space components; exponential maps project Euclidean features onto the manifold.The geodesic distance is the length of the shortest manifold path, and the architecture also uses a fully connected hyperbolic linear transformation.Source: §B.1 · p. 15, §B.1.1 · p. 15, §B.1.2 · p. 15

B.2 DELTA-HYPERBOLICITY

• Metric trees are 0-hyperbolic, while Euclidean spaces of dimension at least two are not hyperbolic for any finite constant; normalizing by diameter makes the measure scale-invariant.Relative values near 0 indicate small deviations from tree-like geometry relative to the space's diameter.Source: §B.2 · p. 16

C.1 ONLINE ENVIRONMENTS

• The online experiments use ten environments spanning arm manipulation, humanoid tasks, and ant navigation, including maze and ball tasks.The listed environments include Ant U5 Maze and Humanoid U Maze.Source: §C.1 · p. 17

C.2 DETAILS ON THE ANT U5 MAZE CASE STUDY

• The Ant U5 Maze case study compares relative hyperbolicity across task horizons using shortest-path route distances and straight-line distances on sampled maze subspaces.It also evaluates route ordering, embedding-distance correlations, and critic-gradient alignment under specified sampling and filtering procedures.Source: §C.2 · pp. 17–18

C.3 HUMANOID EVALUATION PROTOCOL

• The Humanoid results are not directly comparable to Wang et al. (2025, Figure 1) because Brax versions apply the timestep override differently.Brax 0.10.1 retains a 0.003 s timestep, while version 0.10.4 uses the overridden 0.0015 s substep.Source: §C.3 · p. 18
• The discrepancy is specific to Humanoid; other environments, including Humanoid U-Maze and Humanoid Big Maze, respect the timestep override.Source: §C.3 · p. 18
• The authors run Humanoid at a 0.0015 s substep, which they believe Wang et al. intended, and report consistency with that paper's Figure 20 left panel.Source: §C.3 · p. 18

D ARCHITECTURE & HYPERPARAMETERS

• A Euclidean encoder's feature vectors are mapped from the tangent space to the hyperboloid, followed by a hyperbolic linear layer; similarity is negative geodesic distance.Source: Figure 12 · p. 19

D.1 ONLINE CRL HYPERPARAMETERS

• Online hyperparameter tables distinguish shared settings from hyperbolic-only rows and list task-specific training-step totals.The caption states that some rows differ from the released code's defaults.Source: Table 1 · p. 19

D.2 OFFLINE CRL HYPERPARAMETERS

• For offline CRL, the agent uses Minkowski inner product energy and a hybrid hyperbolic actor regularized with Hyper++; the reference configuration is shared except for hyperbolic-only settings.The authors state that inner product energy preserves the ordering of goals by geodesic distance.Source: §D.2 · p. 20, Table 2 · p. 20

E.1 FULL CURVATURE ABLATION

• No single curvature setting works best across environments, and a per-environment setting lets hyperbolic agents match Euclidean steps-at-goal on Humanoid and Arm Push.On both tasks, the Euclidean agent outperforms the default c = 0.5 with high probability.Source: Figure 13 · p. 21, §E.1 · p. 21

E.2 BEHAVIOR ON ANT BALL

• The contrasting Ant Ball behaviors were observed consistently across 640 evaluation episodes.Source: §E.2 · p. 22
• On Ant Ball, the Euclidean agent's initial push leaves the ball 1.5 m from the target, while the hyperbolic agent pushes it back into the goal region.The hyperbolic agent pushes the ball 1.0 m past the target, circles to the far side, and pushes it back; the ball stays in the goal region from step 264 onward.Source: Figure 14 · p. 22, §E.2 · p. 22

• A hyperbolic critic with a Euclidean actor achieves higher mean success than Euclidean CRL on seven of eight tasks.Source: Table 3 · p. 22
• Adding a hyperbolic actor raises mean success further on seven of eight tasks.Source: Table 3 · p. 22
• Entries are mean OGBench success rates (%) ± standard deviation over eight seeds; bold marks hyperbolic entries whose 95% bootstrap CI does not overlap Euclidean CRL.Source: Table 3 · p. 22
• Figure 15 plots median SPS, JAX compilation time, and peak host memory, with all vertical axes on logarithmic scales.Source: Figure 15 · p. 23

• Throughput is median SPS over retained epochs, covering interaction, replay sampling, updates, and diagnostics while excluding setup, prefill, and evaluation.Source: F WALL-CLOCK TIMINGS · p. 23
• Estimated training time for N transitions multiplies the steps needed to collect N transitions by measured seconds per training step, converted to hours, and excludes compilation, setup, prefill, evaluation, and checkpointing.Source: F WALL-CLOCK TIMINGS · p. 23
• At depth 16, hyperbolic agents have 6.27% lower throughput, while the maximum throughput overhead is 14.71% at depth 4; increasing Euclidean depth from 64 to 512 lowers throughput several times and raises compilation overhead and peak memory by more than an order of magnitude.Source: F WALL-CLOCK TIMINGS · p. 23, Table 4 · p. 24
• The hyperbolic head adds 8,448 parameters irrespective of network depth.Source: Table 4 · p. 24