Source-linked AI summary
Hyperbolic Attention Networks
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, Nando de Freitas
TL;DR
Neural networks need representations suited to hierarchical and power-law data, while prior hyperbolic approaches mainly placed geometry on shallow-network parameters. The paper instead applies hyperbolic geometry to deep-network activations by reformulating soft attention over hyperboloid and Klein models, improving performance across graph, visual question answering, and translation tasks while keeping representations compact.
Problem
Neural network representations need suitable geometry for hierarchical and power-law data, while earlier approaches imposed hyperbolic geometry mainly on parameters of shallow networks.
Method
The method imposes hyperbolic geometry on deep-network activations by defining hyperbolic matching and aggregation operations for attention using hyperboloid and Klein models.
Results
The approach improves performance across link prediction, shortest path prediction, visual question answering, and English-to-German machine translation, with gains particularly prominent in relatively small models.
Takeaways & Limitations
Hyperbolic attention provides compact neural representations that better match complex data in limited model capacity.
Takeaways & Limitations
Activations must be mapped into appropriate hyperbolic manifolds, and some midpoint computations rely on points represented in Klein coordinates.
Abstract
from arXiv · showhide
We introduce hyperbolic attention networks to endow neural networks with enough capacity to match the complexity of data with hierarchical and power-law structure. A few recent approaches have successfully demonstrated the benefits of imposing hyperbolic geometry on the parameters of shallow networks. We extend this line of work by imposing hyperbolic geometry on the activations of neural networks. This allows us to exploit hyperbolic geometry to reason about embeddings produced by deep networks. We achieve this by re-expressing the ubiquitous mechanism of soft attention in terms of operations defined for hyperboloid and Klein models. Our method shows improvements in terms of generalization on neural machine translation, learning on graphs and visual question answering tasks while keeping the neural representations compact.
1 Introduction
Hierarchical and scale-free data exhibit exponential structural complexity that Euclidean representations cannot compactly capture. Hyperbolic attention networks impose hyperbolic geometry on deep-network activations and improve generalization across several tasks while keeping representations compact.
- Motivation: Real-world data often exhibits hierarchy, clustering, and power-law structure across natural and human-made systems.Examples include physical and biological phenomena, metabolic-mass relationships, social networks, and word frequencies.
- Motivation: Tree-like and scale-free structures expand exponentially with distance, motivating a geometry whose volume also grows exponentially.In n-ary trees, node counts grow as n^r; hyperbolic disc length and area likewise grow exponentially in radius.
- Motivation: Euclidean space expands only polynomially, so it cannot compactly represent hierarchical data with exponential complexity.In two dimensions, Euclidean length grows as 2πr and area as πr^2, whereas hyperbolic space provides exponential volume growth.
- Contribution: The paper proposes hyperbolic attention networks that place hyperbolic geometry on activations rather than only on parameters of shallow networks.This enables reasoning about embeddings produced by deep networks through hyperbolic operations for attention.
- Contribution: Hyperbolic attention improves generalization on neural machine translation, graph learning, and visual question answering while keeping neural representations compact.The approach is presented as a way to increase network capacity while matching the complexity of hierarchical and power-law data.
2 Models of hyperbolic space
Hyperbolic space can be represented through multiple isomorphic coordinate models with different computational affordances. The paper primarily uses the unbounded hyperboloid model and the Klein model for efficient hyperbolic aggregation.
- Model overview: Hyperbolic space cannot be isometrically embedded into Euclidean space, motivating models that represent it through Euclidean coordinate subsets with hyperbolic metrics.The paper mentions the Poincaré ball and other models as alternative realizations.
- Model overview: Different hyperbolic models are isomorphic but provide different coordinate systems and computational affordances.The paper uses model changes to select convenient operations without changing the underlying geometry.
- Model choices: The paper uses the hyperboloid model for projection into hyperbolic space and the Klein model because it supports an efficient hyperbolic aggregation operation.The choice of model is computational rather than geometric, since the models are isomorphic.
- Hyperboloid model: The hyperboloid model represents n-dimensional hyperbolic space as a manifold in (n + 1)-dimensional Minkowski space.It is the only commonly used unbounded model and is therefore convenient for projecting network activations into hyperbolic space.
- Klein model: The Klein model is the subset K^n = {x ∈ R^n | ∥x∥ < 1} of Euclidean space.Points can be projected between the hyperboloid and Klein models, and Klein distances can be inherited from hyperboloid distances.
3 Attention as a building block for relational reasoning
Attention is presented as an attentive read that matches queries to keys and aggregates values, providing a common primitive for relational reasoning. This formulation covers graph message passing, Relation Networks, and Transformers, and motivates replacing the read operation with a hyperbolic version.
- Attentive read: An attentive read matches each query with keys, assigns attention weights, and computes a normalized weighted average of values.The matching and aggregation stages can be specified through the scoring function, values, and normalization constant.
- Attentive read: Performing an attentive read for each graph element corresponds to message passing, with attention weights serving as edge weights.Each node aggregates messages from neighboring nodes along weighted edges.
- Relation Networks: Relation Networks can be expressed as attentive reads over object pairs, with pairwise transformations producing the values and learnable attention weights representing directed reasoning edges.The basic formulation uses α(oi, oj) = 1, vij = g(oi, oj, c), and Z = 1, while an augmented version makes α learnable.
- Scaled dot-product attention: Transformer scaled dot-product attention performs parallel attentive reads over input vectors using queries, keys, and values.The operation is described as all-to-all message passing, with one attentive read for each input element.
- The path forward: The framework sets up hyperbolic versions of Relation Networks and Transformers by replacing their attentive read operation with a hyperbolic one.The paper has introduced hyperboloid and Klein models before redefining the attentive read on hyperbolic points.
4 Hyperbolic attention networks
Hyperbolic attention networks map neural activations onto hyperbolic space and redefine attention matching and aggregation there. The construction preserves hyperbolic scaling while using model-specific operations for distances and weighted midpoints.
- Activation mapping: The method interprets network activations as points in hyperbolic space, enabling attention operations to use the manifold's metric structure.Activations are mapped onto an appropriate manifold before hyperbolic matching and aggregation are applied.
- Activation mapping: Activations in pseudo-polar coordinates are projected onto the hyperboloid as π((d, r)) = (sinh(r)d, cosh(r)).The projection uses a normalized direction vector d and radius r, increasing scale exponentially.
- Activation mapping: The projection preserves exponential growth in volume as the radius increases linearly, maintaining the scaling behavior needed for hierarchical structure.Without the exponential scaling factor, the effective hyperbolic distance from the origin would grow only logarithmically.
- Hyperbolic attention: Hyperbolic matching assigns attention using the hyperbolic distance between query and key points, with learned or manually set β and c parameters.The function can be exponential with softmax normalization or sigmoid-based.
- Hyperbolic attention: Hyperbolic aggregation uses the Einstein midpoint, an efficient weighted midpoint operation computed in Klein coordinates.Different hyperbolic models can be converted to and from Klein coordinates for midpoint computation, and the operation retains translation and rotation invariances expected of weighted averages.
- Experiments: Figure 3 evaluates Recursive Transformer models on shortest-path and link-prediction tasks across graphs of varying sizes, alongside radius distributions.The figure includes chance performance for shortest-path prediction and radius histograms for models trained on graphs with 100 and 400 nodes.
5 Experiments
The experiments evaluate hyperbolic attention in feedforward networks, Transformers, and Relation Networks across graph, visual reasoning, and translation tasks. Hyperbolic attention improves performance, especially for relatively small-capacity models.
- 5.1 Modeling scale-free graphs: Hyperbolic models outperform Euclidean models on link prediction for scale-free graphs of 1000 and 1200 nodes at equal capacity.The experiments use Recursive Transformers with tied self-attention weights across depth.
- 5.1 Modeling scale-free graphs: Hyperbolic models also improve shortest path length prediction, while learned activation scales increase for larger graphs.The task predicts path lengths up to 25 as an unbalanced classification problem.
- 5.2 Sort-of-CLEVR: Around 20 percentage points of improvement over the standard Relation Network occurs on low-capacity Sort-of-CLEVR models using hyperbolic attention.The relational MLP uses only two units, and models may use Euclidean or Einstein midpoint aggregation.
- 5.3 CLEVR: On CLEVR, hyperbolic attention with sigmoid consistently outperforms the other tested models across relational-network capacity levels.The experiments vary the capacity of the relational component and report test accuracy.
- 5.4 Neural machine translation: 28.45 BLEU is achieved by the best hyperbolic-attention Transformer on WMT14 English–German, versus 28.4 BLEU reported for the original big model.Improvements are more significant when model capacity is restricted, particularly with sigmoid attention weights.
6 Conclusion
The paper presents hyperbolic attention as a way to impose hyperbolic inductive biases on deep-network activations. Across graph, visual question answering, and translation tasks, performance improves, particularly in relatively small models.
- 6 Conclusion: Hyperbolic attention applies hyperbolic geometry to both attention-weight computation and aggregation over values.The mechanism is implemented in Relation Networks and Transformers.
- 6 Conclusion: Improved performance is reported for link prediction, shortest path prediction, two visual question answering datasets, and English-to-German translation.The gains are particularly prominent in relatively small models.
- 6 Conclusion: The results support the hypothesis that hyperbolic geometry induces more compact representations.This conclusion is tied to the stronger gains observed in relatively small models.
A.1 More on models of hyperbolic space
Figure 5 relates the paper’s coordinate representations to the hyperboloid, Klein, and Poincaré models of hyperbolic space.
- A.1 More on models of hyperbolic space: The left panel relates pseudo-polar coordinates in R^n to the hyperboloid in R^(n+1).The right panel shows projections among the hyperboloid, Klein, and Poincaré models.
- A.1 More on models of hyperbolic space: The right panel depicts projections relating the hyperboloid, Klein, and Poincaré representations.These are alternative representations of points used in the paper.
A.2 Scale-free graph generation
The scale-free graph experiments use the algorithm of von Looz et al. to generate graphs with specified parameters.
- A.2 Scale-free graph generation: The experiments set α to 0.95 and edge_radius_R_factor to 0.35 when generating scale-free graphs.Graph generation follows the algorithm described by von Looz et al.
A.3 Scale-free graph curriculum
Training on scale-free graph tasks uses a curriculum that gradually expands the angular slice used to extract connected graph components.
- Curriculum training is essential for the scale-free graph tasks, specifically LP and SPLP.
- The method extracts connected components by cutting the generated graph on the hyperbolic disk into angular slices.
- The slice begins at 30 degrees and gradually increases in angular size according to the number of curriculum lessons.
A.4 Travelling salesman problem (TSP)
For the travelling salesman problem, the authors train a DQN-like agent with the hyperbolic recursive transformer and compare coordinate variants in the model.
- The TSP graphs are generated following the procedure introduced in reference.
- The authors train an off-policy DQN-like agent with the hyperbolic recursive transformer on the TSP task.
- An ablation compares hyperbolic networks with and without implicit coordinates, with better performance when implicit polar coordinates are used.
- Figure 8 compares a hyperbolic recursive transformer with and without spherical coordinates on the travelling salesman problem.