PaperlayerView exact revision on arXiv

Source-linked AI summary

Hyperbolic Neural Networks

Octavian-Eugen Ganea, Gary Bécigneul, Thomas Hofmann

arXiv:1805.09112v2cs.LGstat.ML

TL;DR

Euclidean geometry is not always suitable for complex, hierarchical data, while corresponding hyperbolic neural-network layers have been lacking. The paper combines Möbius gyrovector spaces with Poincaré-model Riemannian geometry to derive hyperbolic versions of core neural architectures. These models outperform or match Euclidean counterparts on textual entailment and noisy-prefix recognition, especially when sequential data has implicit hierarchical structure.

  • Problem

    Many complex datasets have non-Euclidean or tree-like structure, but deep-learning tools for such domains lack principled generalizations of basic operations and standard neural architectures.

  • Method

    The paper combines Möbius gyrovector-space formalism with Poincaré-model Riemannian geometry to generalize MLR, FFNNs, RNNs, and GRUs to hyperbolic space.

  • Results

    Hyperbolic models outperform or match corresponding Euclidean architectures on textual entailment and noisy-prefix recognition, with larger improvements when the data structure is more tree-like.

  • Takeaways & Limitations

    Hyperbolic neural layers support sequential-data embedding and classification in settings with implicit hierarchical structure.

  • Takeaways & Limitations

    Hyperbolic neural networks have a highly non-convex optimization landscape, making initialization important because runs can converge to poor local minima.

Abstract

from arXiv · show

Hyperbolic spaces have recently gained momentum in the context of machine learning due to their high capacity and tree-likeliness properties. However, the representational power of hyperbolic geometry is not yet on par with Euclidean geometry, mostly because of the absence of corresponding hyperbolic neural network layers. This makes it hard to use hyperbolic embeddings in downstream tasks. Here, we bridge this gap in a principled manner by combining the formalism of Möbius gyrovector spaces with the Riemannian geometry of the Poincaré model of hyperbolic spaces. As a result, we derive hyperbolic versions of important deep learning tools: multinomial logistic regression, feed-forward and recurrent neural networks such as gated recurrent units. This allows to embed sequential data and perform classification in the hyperbolic space. Empirically, we show that, even if hyperbolic optimization tools are limited, hyperbolic sentence embeddings either outperform or are on par with their Euclidean variants on textual entailment and noisy-prefix recognition tasks.

1 Introduction

Euclidean embeddings are convenient and effective, but many complex datasets have non-Euclidean structure that hyperbolic geometry can represent more meaningfully. The paper develops hyperbolic neural-network tools to address the missing operations needed for deep learning in these spaces.

  • Euclidean embeddings provide vector operations and support neural-network inputs across tasks including sentiment detection, translation, entailment, and link prediction.
  • Tree-structured data can require low-distortion hyperbolic representations because hyperbolic distance growth matches the exponential growth of nodes with tree depth.
  • Non-Euclidean deep learning is limited by the lack of principled generalizations for basic operations and closed-form geometric objects.
  • Hyperbolic representations have outperformed Euclidean embeddings on hierarchical, taxonomic, and entailment data, but suitable deep-learning tools remain needed.
  • The paper generalizes basic operations, MLR, FFNNs, RNNs, and GRUs to the Poincaré model using gyrovector spaces and Riemannian geometry, then compares them with Euclidean variants.

2 The Geometry of the Poincaré Ball

The paper uses the Poincaré ball as a conformal hyperbolic model and develops its operations through Möbius gyrovector algebra and Riemannian geometry. These connections provide geodesics, exponential and logarithmic maps, and matrix-vector transformations needed for hyperbolic neural networks.

  • 2.1 Basics of Riemannian geometry: A Riemannian manifold equips a manifold with smoothly varying tangent-space inner products that induce global distances through shortest paths.
  • 2.2 The Poincaré ball: The Poincaré ball represents hyperbolic space as an open Euclidean ball with a conformal Riemannian metric, preserving angles relative to Euclidean geometry.
  • 2.3 Gyrovector spaces: Gyrovector spaces provide a non-associative algebraic framework for hyperbolic operations analogous to vector spaces in Euclidean geometry.
  • 2.3 Gyrovector spaces: Möbius operations recover Euclidean addition, scalar multiplication, and geometry as curvature c approaches zero.
  • 2.4 Connecting Gyrovector spaces and Riemannian geometry of the Poincaré ball: The paper connects Möbius operations with geodesics, exponential maps, logarithmic maps, and parallel transport in the Poincaré ball.
  • 2.4 Connecting Gyrovector spaces and Riemannian geometry of the Poincaré ball: These identities yield a natural matrix-vector multiplication and support shared parameters across tangent spaces, including biases and hyperbolic MLR parameters.

3 Hyperbolic Neural Networks

The section generalizes key neural-network operations and architectures to the Poincaré ball, including hyperbolic MLR, feed-forward layers, and recurrent networks. These constructions recover Euclidean operations as curvature c approaches zero while enabling hyperbolic classification and sequence modeling.

  • Hyperbolic feed-forward layers: Hyperbolic feed-forward layers combine Möbius matrix-vector multiplication, bias translation, concatenation, and pointwise nonlinearities.Bias translations between otherwise Euclidean-coordinate operations distinguish the hyperbolic model from its Euclidean counterpart.
  • Hyperbolic multiclass logistic regression: Hyperbolic MLR reformulates class probabilities using distances to Poincaré hyperplanes and recovers Euclidean softmax as c approaches zero.Each class uses a margin hyperplane parameterized by a point and tangent-space direction.
  • Hyperbolic feed-forward layers: The Möbius version transports Euclidean maps into the Poincaré ball while preserving composition, directions, and the Euclidean limit.For linear maps, this yields Möbius matrix-vector multiplication with associativity and preserved rotations.
  • Hyperbolic feed-forward layers: Concatenated hyperbolic inputs are combined through Möbius addition, and mixed hyperbolic-real features can be incorporated with a learnable hyperbolic bias.The construction reduces to ordinary matrix multiplication and vector addition when c approaches zero.
  • Hyperbolic RNN: A naive hyperbolic RNN replaces Euclidean matrix products and additions with Möbius operations, while Euclidean inputs can be mapped into the ball using the exponential map.The resulting recurrence applies a pointwise Möbius nonlinearity to the transformed hidden state and input.
  • Hyperbolic RNN: The GRU adaptation replaces pointwise gate scaling and interpolation with Möbius operations, and it recovers the usual GRU when c approaches zero.When the update gate is 0 or 1, the hidden state becomes the previous state or the intermediate state, respectively.

4 Experiments

The experiments evaluate hyperbolic and Euclidean neural architectures on textual entailment, noisy-prefix recognition, and WordNet subtree classification. Hyperbolic models are especially advantageous for tree-like sequential structure, while hyperbolic MLR improves subtree classification.

  • Models and setup: Hyperbolic sentence models use paired hyperbolic RNNs or GRUs, followed by a feed-forward network and multinomial logistic regression.The two sentence embeddings and their squared distance are passed through the classifier, with cross-entropy loss.
  • Sequential tasks: All GRU models outperform their RNN variants, and hyperbolic models gain most over Euclidean variants when the data structure is more tree-like.For PREFIX-10%, hyperbolic reduces error by factors of 3.35 versus Euclidean RNN and 1.5 versus Euclidean GRU.
  • Sequential tasks: On SNLI, hyperbolic methods perform similarly to their Euclidean variants, while hyperbolic and Euclidean MLR are on par with hyperbolic sentence embeddings.The authors state that further empirical investigation is needed for hyperbolic MLR in this setting.
  • Optimization behavior: Hyperbolic accuracy improves as sentence-embedding norms approach 1, after the model first adjusts the angular layout.The reported behavior is specific to the hyperbolic setting and reflects the ball border for c = 1.
  • Subtree classification: Hyperbolic MLR outperforms Euclidean variants in almost all WordNet subtree settings, sometimes by a large margin.Table 2 compares hyperbolic MLR with direct Euclidean MLR and Euclidean MLR after mapping embeddings to the tangent space at 0, across four subtrees and dimensions 2, 3, 5, and 10.

5 Conclusion

The paper generalizes major Euclidean deep-learning tools to constant-negative-curvature spaces and reports competitive performance on sequential data with implicit hierarchical structure. The authors identify hyperbolic non-convexity and optimization as directions for future work.

  • Contribution: MLR, FFNNs, RNNs, and GRUs are generalized to constant-negative-curvature spaces by combining Riemannian geometry with gyrovector spaces.The paper presents these generalizations as principled extensions of classic Euclidean deep-learning tools.
  • Empirical conclusion: Hyperbolic architectures outperform or match corresponding Euclidean architectures on sequential data with implicit hierarchical structure.This is the paper's stated empirical conclusion across the evaluated sequential settings.
  • Future directions: Future work includes understanding the hyperbolic non-convexity spectrum and developing other non-Euclidean deep-learning methods.The paper also reports that its data and TensorFlow code are publicly available.

A Hyperbolic Trigonometry

This appendix introduces hyperbolic angle terminology and states the hyperbolic law of sines. It also notes that the law of cosines can be adapted to hyperbolic space.

  • Angles: The angle ∠A is defined as the angle between the geodesics from A to B and from A to C.The paper also calls ∠A a gyroangle in the cited gyrovector-space literature.
  • Hyperbolic trigonometry: The hyperbolic law of sines is stated for points A, B, and C with angles and geodesic-segment lengths defined analogously at each vertex.The notation includes ∠B and the hyperbolic length of segment BA, denoted ˜c.
  • Hyperbolic trigonometry: The hyperbolic law of cosines can also be adapted to hyperbolic space.

B Proof of Theorem 4

The proof establishes the parallel transport isometry from the origin to a point in the Poincaré ball. It uses the geodesic from 0 to x, constructs a parallel vector field, and verifies parallelism through the Levi-Civita connection.

  • Conclusion: The resulting map is an isometry from the tangent space at 0 to the tangent space at x.The theorem concerns parallel transport of v ∈ T0Dn_c to TxDn_c.
  • Geodesic construction: The geodesic from 0 to x is parameterized as γ(t) = x ⊗c t for t ∈ [0, 1].
  • Parallel transport: A unique parallel vector field X along γ is chosen with initial value X(0) = v.This field transports v from the tangent space at 0 along the geodesic.
  • Proof mechanism: The proof verifies that X is parallel using the covariant derivative associated with the Levi-Civita connection.The connection is computed from its Christoffel symbols in local coordinates.

C Proof of Eq. (22)

The proof proceeds in two steps, using gyrospace cancellation to establish the target relation. Intermediate orthogonality conditions are used to complete the argument.

  • The proof is explicitly organized into two steps.
  • Gyrospace left-cancellation is combined with the preceding relation to derive the desired result.
  • The argument introduces a point c satisfying an orthogonality condition and derives the corresponding relation using Eq. (12).

D Proof of Theorem 5

The proof establishes existence and uniqueness of orthogonal projections in the Poincaré ball, then shows that these projections minimize distance to geodesics and support the theorem’s closed form.

  • Geodesic representation: Geodesics in the Poincaré ball are represented in gyrovector form, with endpoints at infinity on the ball boundary.The closed form of the geodesic is given by Eq. (10), and its angle relations are expressed through Möbius operations and Euclidean inner products.
  • Distance minimization: The orthogonal projection minimizes the distance between a point and the geodesic.The proof uses the hyperbolic sine law and the fact that a right-triangle hypotenuse is strictly longer than either other side.
  • Theorem conclusion: The proof constructs a point on a Poincaré hyperplane attaining the infimum distance and identifies the same distance as in the theorem statement.The construction combines the projection lemmas, hyperbolic sine-law relations, and the closed-form expressions in Eqs. (61), (62), (63), and (67).
  • Theorem conclusion: The remaining optimization reduces to maximizing a cosine angle over a Poincaré hyperplane, equivalent to a Euclidean orthogonal-projection problem.The resulting existence statement completes the proof after combining the preceding equations.

E Derivation of the Hyperbolic GRU Update-gate

The hyperbolic GRU update gate is derived by requiring invariance to smooth time-warpings and translating the resulting positive time-scaling into Möbius operations.

  • Time-warping motivation: The derivation starts from a naive RNN and imposes invariance of the neural-network class to smooth increasing time-warpings.A time-warping replaces t with α(t), whose derivative determines the learned temporal scaling.
  • Euclidean gate derivation: Because the time-warping derivative is positive, it is parameterized with a sigmoid to recover an update- or forget-gate equation.The resulting scalar gate is z(t) = σ(W_zh(t) + U_zx(t)).
  • Hyperbolic adaptation: Discretizing the hyperbolic dynamics yields a Möbius-scaled update expression involving the current hidden state, input, weights, and activation.The intermediate and final forms are given in Eqs. (75), (83), and (84).
  • Hyperbolic adaptation: The gyro-chain-rule extends the time-warping argument to hyperbolic hidden states in the Poincaré ball.The derivation uses a gyro-derivative, Möbius scalar associativity, and the hyperbolic RNN equation.
  • Final hyperbolic GRU: The proposed hyperbolic GRU replaces scalar pointwise scaling with Möbius matrix scaling diag(z_t) ⊗_c ·.The scalar update gate z_t is defined with a sigmoid, and the matrix scaling leads to Eq. (33).

F More Experimental Investigations

Additional experiments examine how hyperbolic sentence norms relate to accuracy on the PREFIX-30% task across hyperbolic and Euclidean architectures.

  • Experimental setup: Hyperbolic experiments compare accuracy and first-premise sentence norms across runs of hyperbolic GRU and hyperbolic RNN architectures.The plotted architectures use hyperbolic FFNNs and hyperbolic or mixed hyperbolic/Euclidean MLRs.
  • Observed behavior: Hyperbolic accuracy is often higher when sentence embeddings approach the boundary, and faster norm growth is associated with reaching a good local minimum.These observations are reported for both hyperbolic RNNs and GRUs and referenced against Figures 3 and 5.
  • Observed behavior: Test accuracy often begins increasing when sentence-embedding norms increase, whereas hyperbolic norms can remain near zero for several epochs before growing.The authors suggest this may reflect initial adjustment of angular layout near the origin before exploiting hyperbolic geometry.
  • Experimental setup: Euclidean experiments provide the corresponding PREFIX-30% accuracy and sentence-norm plots for Euclidean GRU, FFNN, and MLR layers.The horizontal axis measures millions of training examples processed.
Loading 1805.09112v2…