PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperbolic Deep Neural Networks: A Survey

Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, Guoying Zhao

arXiv:2101.04562v3cs.LGcs.CV

TL;DR

Hyperbolic deep neural networks address the challenge of representing hierarchical data with geometry suited to such structure. The survey synthesizes hyperbolic neural components, models, and applications, finding reported gains in several settings while identifying unresolved model-choice and normalization questions.

  • Problem

    Euclidean representations are limited for data with hierarchical structure, while constructing neural networks on hyperbolic spaces requires difficult manifold operations.

  • Method

    The survey reviews hyperbolic geometry, generalizes neural components, examines hyperbolic architectures and applications, and identifies open questions.

  • Results

    Reported applications show improvements over Euclidean models in restricted dimensions or capacities, while performance advantages can disappear as embedding dimension increases.

  • Takeaways & Limitations

    Hyperbolic representations can provide compact, interpretable modeling for hierarchical data across machine-learning tasks and applications.

  • Takeaways & Limitations

    The literature remains concentrated on Poincaré-based operations, while model choice and hyperbolic normalization remain unresolved.

Abstract

from arXiv · show

Recently, there has been a rising surge of momentum for deep representation learning in hyperbolic spaces due to theirhigh capacity of modeling data like knowledge graphs or synonym hierarchies, possessing hierarchical structure. We refer to the model as hyperbolic deep neural network in this paper. Such a hyperbolic neural architecture potentially leads to drastically compact model withmuch more physical interpretability than its counterpart in Euclidean space. To stimulate future research, this paper presents acoherent and comprehensive review of the literature around the neural components in the construction of hyperbolic deep neuralnetworks, as well as the generalization of the leading deep approaches to the Hyperbolic space. It also presents current applicationsaround various machine learning tasks on several publicly available datasets, together with insightful observations and identifying openquestions and promising future directions.

1 INTRODUCTION

Hyperbolic deep neural networks target data with hierarchical structure, addressing limitations of Euclidean representation learning. This survey reviews their foundations, neural components, applications, and future directions.

  • Many real-world domains contain tree-like or latent hierarchical structures, including social networks, language, and biological relationships.
  • Hyperbolic deep neural networks potentially offer better generalization, fewer parameters and dimensions, lower distortion, and improved interpretability.
  • The survey presents context on hyperbolic geometry, reviews algorithms and applications, and identifies open questions and future directions.
  • Hyperbolic models are already used for hierarchical representation in academia, recommender systems at millions of users, and drug hierarchy learning.

2 HYPERBOLIC GEOMETRY

Hyperbolic geometry replaces Euclid’s parallel postulate and is formalized through manifolds, metrics, curvature, and equivalent geometric models. Its core maps and geometric notions support computation on these spaces.

  • Hyperbolic geometry has constant negative sectional curvature and satisfies Euclid’s postulates except the parallel postulate.
  • Replacing the parallel postulate with its negation yields non-Euclidean geometries, including hyperbolic geometry.
  • A manifold is locally approximated by Euclidean space, while a Riemannian metric supplies local notions of angle, length, area, and volume.
  • Exponential maps move tangent vectors onto a manifold, logarithmic maps return manifold points to tangent spaces, and parallel transport carries vectors between tangent spaces.
  • Negative curvature makes volumes grow faster than in Euclidean space, and the Poincaré disc is log3-hyperbolic whereas Euclidean space is not δ-hyperbolic.

THE HYPERBOLIC SPACE

Building neural networks in hyperbolic space requires principled generalizations of Euclidean operations, but manifold computation and closed-form geometry create substantial challenges. Tangent-space and gyrovector approaches provide alternative constructions.

  • Hyperbolic neural-network construction is difficult because Riemannian operations required by neural architectures are harder to develop than their Euclidean counterparts.
  • Manifold implementations are less efficient, while geodesics, exponential maps, and distance functions may lack closed-form expressions.
  • A common strategy maps hyperbolic data to tangent space, performs Euclidean operations, and maps the results back.
  • Tangent-space approximations can negatively affect learning, motivating operations built directly from hyperbolic geometry.
  • Gyrovector spaces generalize Euclidean vector spaces through Möbius transformations, including addition, scalar multiplication, vector multiplication, and exponential and logarithmic maps.

3.2 Mean in the Hyperbolic Space

Computing means in hyperbolic space is central to pooling, statistics, and graph aggregation, but Euclidean averaging may leave the manifold. The literature therefore develops differentiable, tangent-space, and closed-form alternatives.

  • Directly averaging hyperbolic inputs can produce a result outside the manifold, so mean computation requires geometry-aware methods.
  • Tangential aggregation projects hyperbolic features to tangent space and uses attention weights to compute graph-neighborhood means.
  • The Einstein midpoint provides a concise hyperbolic mean in Klein coordinates, with Lorentz factors and projections connecting equivalent models.
  • The gyromidpoint is a closed-form Poincaré-model average defined for weighted samples in gyrovector spaces.
  • A differentiable Fréchet-mean method applies to Lorentz and Poincaré models and is more efficient than RGD and Karcher Flow in experiments.

3.3 Concatenation and Split Operations

Hyperbolic concatenation and split operations can be implemented through tangent-space mappings or Möbius transformations. Beta variants address the reduced representational power of coordinate splitting, while Möbius-based concatenation can be computationally costly and unbalanced.

  • Motivation: Concatenation and split are used for feature fusion, GCN filters, and computing correlations in attention mechanisms.These operations therefore support several standard neural-network constructions beyond simple representation rearrangement.
  • Tangent-space operations: Tangent-space splitting maps a hyperbolic embedding to the origin's tangent space, partitions its coordinates, and can map the parts back with the exponential map.Concatenation is performed inversely by joining tangent-space representations before mapping them to the hyperbolic space.
  • Beta operations: Merely splitting coordinates lowers output gyrovector norms and limits representational power.Beta-split and beta-concatenation introduce a scalar coefficient to address this limitation.
  • Möbius-based concatenation: Möbius-based concatenation projects hyperbolic representations into a common space using linear projection functions defined through Möbius transformations.The method introduces one projection function for each input representation before producing the concatenated result.
  • Möbius-based concatenation: Repeated Möbius additions and multiplications incur heavy computational cost and create unbalanced priority among input sub-gyrovec­tors.This is a stated drawback relative to the preceding beta methods.

3.4 Convolutional Neural Network Operations

Hyperbolic CNN operations remain relatively underexplored, with existing approaches using tangent-space approximations or Poincaré-manifold constructions. A reported construction concatenates receptive-field features with beta-concatenation and applies a Poincaré fully connected layer.

  • Motivation: Hyperbolic CNN research is limited despite CNNs being fundamental to many machine-learning and computer-vision tasks.Hyperbolic image embedding applies hyperbolic geometry to image classification and person re-identification.
  • Tangent-space approach: Tangent-space CNN generalization may collapse to a vanilla Euclidean CNN when multiple tangent-space CNNs are stacked.The text also questions whether the approximation preserves hyperbolic advantages.
  • Poincaré construction: A Poincaré convolutional layer beta-concatenates receptive-field representations at each feature pixel and transfers the resulting vector with a Poincaré fully connected layer.For a C-channel input tensor and filter size K, each receptive field is concatenated into a vector in the Poincaré ball.
  • Open issues: The described hyperbolic CNN remains the only version identified in the cited work and could be improved in convolutional strategy and computational efficiency.This marks both a narrow literature base and an explicit development target.

3.5 Recurrent Neural Network Operations

Hyperbolic RNNs generalize recurrent updates by replacing Euclidean operations with Möbius operations in gyrovector space, and the same strategy extends to GRUs. Existing work is limited to costly Poincaré-model operations.

  • Euclidean RNNs: RNNs support temporal behavior by using previous outputs as inputs and are commonly used for sequence-learning tasks.The recurrent update tracks a hidden state at the next step from the current hidden state and input.
  • Hyperbolic RNNs: Hyperbolic RNNs replace Euclidean addition and multiplication with Möbius operations in gyrovector space.The generalized update uses the current hidden state and input to compute the next hidden state.
  • GRU extension: The same gyrovector-space strategy is extended to gated recurrent units.The cited work applies the idea to GRU architecture rather than introducing a separate recurrent construction.
  • Limitations and directions: Existing hyperbolic recurrent approaches are limited to the Poincaré model and use operations that are costly compared with their Euclidean counterparts.Future work is identified in developing more efficient operations and extending methods to models such as the Lorentz model.

3.6 Activation function

Hyperbolic activation functions can preserve the manifold directly or apply ordinary functions through the tangent space. Existing approaches also vary by curvature handling, and one method removes activations because manifold operations are already nonlinear.

  • Manifold-preserving activations: Manifold-preserving nonlinearities can be applied on a manifold after the exponential map, but the appropriate activation differs across manifolds.The paper presents this as one route for introducing nonlinearity while respecting manifold structure.
  • Möbius activations: A Möbius version of a function performs the function through the tangent space at the origin and can serve as an activation when input and output dimensions match.The same projection idea is also used for general functions between Euclidean spaces and hyperbolic representations.
  • Curvature-aware activations: A graph-convolutional activation variant accounts for different curvatures across layers by defining logarithmic and exponential maps at layer-specific manifold origins.Its distinguishing feature is curvature handling rather than a different overall tangent-space principle.
  • Activation-free design: One approach removes activation functions because operations on the manifold itself are considered nonlinear enough to make them unnecessary.This is presented as an alternative design choice rather than a universal rule.

3.7 Batch Normalization

Hyperbolic batch normalization extends normalization to Riemannian settings, including a differentiable Fréchet-mean formulation. The survey identifies this area as underdeveloped, with unclear necessity on bounded manifolds.

  • Batch normalization re-centers and re-scales layer activations to limit internal covariate shift, speed training, and improve stability.
  • A Riemannian extension replaces the Euclidean mean with a differentiable Fréchet mean.
  • Hyperbolic batch normalization has been addressed in only one reported work, while hyperbolic group and instance normalization remain unreported.
  • Whether normalization methods are needed on bounded manifolds such as the Poincaré model remains unclear.

3.8 Classifiers and Multiclass Logistic Regression

The survey reviews hyperbolic multiclass logistic regression and support vector classifiers, including geodesic decision boundaries, parameterizations, and optimization strategies. These constructions adapt Euclidean classification objectives to hyperbolic models such as the Lorentz and Poincaré formulations.

  • Multiclass Logistic Regression: Hyperbolic multiclass logistic regression predicts class probabilities using signed distances from hyperbolic hyperplanes.
  • Multiclass Logistic Regression: Replacing Euclidean addition with Möbius addition yields a hyperbolic formulation of multiclass logistic regression.
  • Multiclass Logistic Regression: A scalar bias parameter reduces each class’s parameterization from 2n parameters to n+1 while representing the same discriminative hyperplane.
  • Support Vector Classifiers: Hyperbolic SVMs use geodesic decision boundaries in the Lorentz model and optimize maximum-margin objectives analogous to Euclidean SVMs.
  • Support Vector Classifiers: The reviewed SVM work extends the construction to nonlinear kernel methods and adversarial-example training.

3.9 Fully-Connected Layers

Hyperbolic fully-connected layers generalize Euclidean linear transformations through matrix-vector operations and manifold-aware bias translations. The survey also describes a Poincaré formulation designed to preserve hyperbolic hyperplane structure.

  • Fully-Connected Layers: Fully-connected layers connect every input to each activation unit and generalize Euclidean transforms to hyperbolic space.
  • Fully-Connected Layers: One construction performs matrix-vector multiplication and translates biases using Möbius translation through tangent spaces.
  • Fully-Connected Layers: Möbius translation can produce surfaces that are no longer hyperbolic hyperplanes, with contour shape determined by row-vector norms and bias.
  • Fully-Connected Layers: A Poincaré fully-connected layer uses a nonlinear transformation v_k(x) and parameters z_k and r_k to define the linear transformation.

3.10 Optimization

Hyperbolic optimization adapts stochastic gradient methods to parameters on Riemannian manifolds, where coordinates, sparsity, and transport create distinctive challenges. The survey covers intrinsic Riemannian optimization, Poincaré and Lorentz updates, and software support for adaptive methods.

  • Optimization Challenges: Standard Adagrad, Adadelta, Adam, and AMSGrad are designed for Euclidean parameters and do not directly optimize hyperbolic spaces.
  • Riemannian Optimization: Riemannian SGD uses intrinsic gradients and exponential-map updates without depending on a particular embedding or local coordinates.
  • Model-Specific Updates: Poincaré optimization approximates the exponential map with retraction and projects embeddings back inside the unit-norm model boundary.
  • Optimization Challenges: Non-Euclidean adaptivity is difficult because manifolds lack intrinsic coordinates, making coordinate-wise sparsity and updates ambiguous.
  • Optimization Challenges: Parallel transporting a canonical tangent-space coordinate system can introduce curvature- and path-dependent rotation that breaks gradient sparsity.
  • Software Support: Geoopt provides a PyTorch manifold interface supporting Riemannian SGD and advanced adaptive optimization algorithms.

4 NEURAL NETWORKS IN HYPERBOLIC SPACE

Hyperbolic neural networks extend representation learning and common neural components to geometries suited to hierarchical data. The surveyed approaches cover embeddings, clustering, attention, graph networks, normalizing flows, and variational auto-encoders, while addressing computational and numerical challenges.

  • 4.1 Hyperbolic Embeddings: Poincaré embeddings model latent hierarchies and can outperform Euclidean embeddings in representation capacity and generalization.The surveyed work identifies Poincaré embeddings as an early approach explicitly designed for latent hierarchical structures.
  • 4.1 Hyperbolic Embeddings: Hyperbolic embedding methods include tree constructions, hyperbolic multidimensional scaling, and Poincaré adaptations of GloVe for unsupervised word representations.Tree embeddings can avoid optimization, h-MDS recovers points from hyperbolic distances, and Poincaré GloVe replaces Euclidean distance while absorbing squared norms into biases.
  • 4.1 Hyperbolic Embeddings: Hyperbolic models address numerical instability by using tiling-based representations whose representation, distance, and gradient errors remain bounded independently of distance from the origin.The instability arises from floating-point computation and ill-conditioned Riemannian metrics as points move outward.
  • 4.2 Hyperbolic Cluster Learning: Hyperbolic hierarchical clustering represents depth through vector norms, placing roots near the origin and leaves near the Poincaré-ball boundary.The child-parent dissimilarity function encourages the hierarchical norm ordering, while the original cost function is computationally expensive for large-scale applications.
  • 4.3 Hyperbolic Attention Network: The survey reviews hyperbolic attention and graph networks, including a Hyperbolic Transformer, curvature-generalized κ-GCN, and graph aggregation operations on Riemannian manifolds.The Hyperbolic Transformer is reported as superior to the Euclidean Transformer under the given comparison, while κ-GCN provides a uniform GCN across zero, negative, and positive curvature.
  • 4.5 Hyperbolic Normalizing Flows: Hyperbolic normalizing flows and variational auto-encoders adapt probabilistic modeling to hierarchical manifolds through tangent-bundle coupling transforms, reparameterizable sampling, and geometry-aware decoders.The survey also covers pseudo-hyperbolic Gaussian distributions, maximum-entropy and wrapped normal distributions, and restricted-normal approaches such as S-VAE.
  • 4.6 Hyperbolic Variational Auto-Encoders: A key limitation is that von Mises–Fisher distributions have one scalar covariance parameter, while Riemannian Normal sampling can be computationally expensive.Wrapped Normal distributions are described as computationally efficient to sample.

5 APPLICATIONS AND PERFORMANCE

Across NLP, graph, knowledge-graph, and computer-vision tasks, hyperbolic models often show advantages in hierarchical settings, especially with restricted dimensions, but benefits can disappear as dimensions increase or when labels depend only on local neighborhoods.

  • NLP with Hyperbolic Embeddings: Hyperbolic MLRs outperform Euclidean versions in sentence entailment, with the model in achieving comparable performance using half the parameters.The model also shows more stable training and narrower confidence intervals.
  • NLP with Hyperbolic Embeddings: Hyperbolic Transformers improve over Euclidean Transformers, with larger gains under restricted capacity; Poincaré CNN gains disappear as embedding dimensions increase.On WMT’17 English-German, Poincaré CNNs improve over Euclidean models in fewer dimensions, but the advantage disappears at higher dimensions.
  • NLP with Hyperbolic Embeddings: Hyperbolic methods benefit text classification, question answering, word embeddings, and taxonomy embedding, often achieving strong performance with fewer dimensions or parameters.HyperText with 50 dimensions outperforms FastText with 300, while HyperQA is described as fast and parameter efficient; unsupervised hypernymy embeddings nearly outperform supervised Euclidean counterparts.
  • Hyperbolic for Graph applications: Graph experiments report gains for hyperbolic embeddings and HGCN, particularly on low-δ-hyperbolic data and dimensions below 20, while performance can reverse at dimensions above 256.HGNN performs better on the Lorentz model than the Poincaré model in most cases, but its advantage disappears as dimension increases.
  • Knowledge Graph completion: Hyperbolic recommendation and knowledge-graph models show strong reported results, including scaling to millions of users and nearly doubling Euclidean translational-model MRR.The HyperKG comparison is not directly state-of-the-art because it excludes several advanced competing techniques.
  • Hyperbolic for Computer Vision: Computer-vision studies report superiority on person re-identification and few-shot classification, while variational models gain most at low latent dimensions and require further evaluation at larger scale.The hyperbolic VAE advantage decreases as latent dimension increases; on MNIST reconstruction, the advantage at latent dimension six disappears when the dimension doubles.

6 DISCUSSION AND OPEN PROBLEMS

The discussion identifies unresolved choices in hyperbolic model selection, limited computer-vision coverage, inconsistent empirical evaluation, and opportunities for automated architecture design.

  • 6.1 Which hyperbolic models?: Lorentz models offer better numerical stability in some work, but their unboundedness complicates integration with modern neural networks and model choice remains unresolved.The literature is concentrated on Poincaré models, while tiled Lorentz approaches seek greater stability.
  • 6.2 Hyperbolic Computer Vision: Hyperbolic convolutional networks remain underexplored for image and video tasks, partly because conventional CNNs already perform strongly in computer vision.Existing hyperbolic networks are mainly designed for NLP, graph, and other tree-structured datasets.
  • 6.3 Consistency and Stability: Hyperbolic embeddings improve some problems and datasets but not others, while Euclidean models can be superior at larger embedding dimensions.The paper highlights uncertainty about when hyperbolic geometry provides compactness alone versus meaningful performance gains.
  • 6.3 Consistency and Stability: Comparisons across hyperbolic methods are difficult because studies use different tasks and evaluation protocols, motivating standard benchmarks.The inconsistency affects comparisons even when methods share the same backbone.
  • 6.4 Future Directions: Future work could combine Riemannian neural networks with automated machine-learning methods to design neural modules rather than only search projection dimensions.Existing NAS work searches for the best projection dimension in the Poincaré model.
Loading 2101.04562v3…