PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully Hyperbolic Neural Networks

Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

arXiv:2105.14686v3cs.CLcs.LG

TL;DR

Existing hyperbolic networks perform most operations in Euclidean tangent spaces, limiting their fully hyperbolic character and creating stability concerns. This paper instead formalizes neural operations directly in the Lorentz model through boosts and rotations, and reports better or comparable results across representative tasks. The framework also outperforms Euclidean baselines with fewer parameters.

  • Problem

    Existing hyperbolic networks formalize most operations in Euclidean tangent spaces, and logarithmic and exponential map compositions weaken model stability.

  • Method

    The paper formalizes neural-network operations directly in the Lorentz model using relaxed Lorentz transformations, including boosts and rotations.

  • Results

    The fully hyperbolic networks outperform Euclidean baselines with fewer parameters and achieve better or comparable results than existing tangent-space hyperbolic networks.

  • Takeaways & Limitations

    Lorentz-model boosts and rotations provide the basis for fully hyperbolic neural networks across shallow and deep representative tasks.

  • Takeaways & Limitations

    The framework selects the Lorentz model as its cornerstone based on numerical stability and calculation simplicity of its maps and distance function.

Abstract

from arXiv · show

Hyperbolic neural networks have shown great potential for modeling complex data. However, existing hyperbolic networks are not completely hyperbolic, as they encode features in a hyperbolic space yet formalize most of their operations in the tangent space (a Euclidean subspace) at the origin of the hyperbolic space. This hybrid method greatly limits the modeling ability of networks. In this paper, we propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model by adapting the Lorentz transformations (including boost and rotation) to formalize essential operations of neural networks. Moreover, we also prove that linear transformation in tangent spaces used by existing hyperbolic networks is a relaxation of the Lorentz rotation and does not include the boost, implicitly limiting the capabilities of existing hyperbolic networks. The experimental results on four NLP tasks show that our method has better performance for building both shallow and deep networks. Our code will be released to facilitate follow-up research.

1 Introduction

Existing hyperbolic neural networks often perform most neural operations in Euclidean tangent spaces rather than directly in hyperbolic space. The paper proposes Lorentz-model operations that support fully hyperbolic networks and reports better or comparable results across representative tasks.

  • Motivation: Existing hyperbolic networks adapt Euclidean operations in tangent spaces because matrix-vector multiplication is difficult to define directly in hyperbolic spaces.Features are mapped between hyperbolic and tangent spaces using logarithmic and exponential maps.
  • Motivation: Logarithmic and exponential map compositions are complicated and can range to infinity, weakening model stability.
  • Approach: The proposed framework formalizes neural operations directly in the Lorentz model using relaxed Lorentz transformations, including boosts and rotations.The framework includes operations such as linear and attention layers.
  • Motivation: Tangent-space linear transformations correspond to relaxed Lorentz rotations and omit Lorentz boosts, limiting existing networks' modeling capabilities.
  • Results: Fully hyperbolic networks outperform Euclidean baselines with fewer parameters and are faster, more stable, and better or comparable to tangent-space hyperbolic networks.The evaluation covers knowledge graph embeddings, network embeddings, fine-grained entity typing, machine translation, and dependency tree probing.

2 Preliminaries

The paper uses the Lorentz model, an isometrically equivalent hyperbolic representation with a time axis and spatial axes, as its framework cornerstone. Lorentz transformations combine boosts and rotations, which directly transform points in the model.

  • Hyperbolic geometry: Hyperbolic geometry is a non-Euclidean geometry with constant negative curvature K.
  • Lorentz model: The Lorentz model is selected because its exponential and logarithm maps and distance function provide numerical stability and calculation simplicity.
  • Lorentz model: The Lorentz model represents points on the upper sheet of a hyperboloid in an (n + 1)-dimensional Minkowski space.
  • Lorentz model: The model distinguishes the 0-th coordinate as a time axis and the remaining n coordinates as spatial axes.
  • Lorentz transformations: Lorentz transformations decompose into a boost and a rotation, and both are linear transformations directly defined in the Lorentz model.The framework builds fully hyperbolic neural networks from these two transformation types.

3 Fully Hyperbolic Neural Networks

The framework builds neural-network operations directly in the Lorentz model, using a learned hyperbolic linear layer that can represent both Lorentz boosts and rotations. It extends this basis to attention, residual, and position-encoding layers while relating conventional tangent-space layers to restricted Lorentz transformations.

  • Fully Hyperbolic Linear Layer: The proposed Lorentz linear layer learns a matrix constrained to map every valid input in the Lorentz model to a valid output.The reformulation uses a matrix function f_x(M) to satisfy the Lorentz-space constraint during optimization.
  • Fully Hyperbolic Linear Layer: The set of transformations generated by the layer contains both Lorentz rotation and boost matrices.This establishes that the layer can cover the two transformation types used in the Lorentz model.
  • Relation to Existing Hyperbolic Layers: Conventional tangent-space hyperbolic linear layers correspond to special Lorentz rotations without boost operations.They map points to the tangent space, apply a Euclidean matrix transformation, and map the result back; the paper characterizes this as a restricted transformation.
  • Relation to Existing Hyperbolic Layers: The proposed linear layer avoids logarithmic and exponential maps, aiming to improve expressiveness, efficiency, and stability.The paper identifies compositions of hyperbolic and inverse-hyperbolic functions in existing layers as complicated and potentially destabilizing.
  • Fully Hyperbolic Attention Layer: Lorentz attention aggregates values through a Lorentz centroid whose weighted squared distance to the input point set is minimized.The centroid is chosen because it has a closed-form, fast computation, whereas the Fréchet mean requires iterative computation and the Einstein midpoint requires model transformations.
  • General Formula: The general formula relaxes full hyperbolicity in implementation while guaranteeing that its inputs and outputs remain in the Lorentz model.The paper explicitly marks this construction as an implementation relaxation rather than a fully hyperbolic operation.
  • Residual and Position Encoding Layers: The framework implements residual-like operations by using a computational block’s input as the bias of its final Lorentz linear layer.This compensates for the absence of well-defined vector addition in the Lorentz model; the same bias mechanism supports Lorentz position encoding.

4 Experiments

The experiments evaluate HyboNet across knowledge graph completion, machine translation, and dependency tree probing, comparing fully hyperbolic models with Euclidean and partially hyperbolic baselines. HyboNet generally achieves stronger results and convergence, including better structured-information representations.

  • Experimental scope: Experiments cover knowledge graph completion, machine translation, and dependency tree probing, with additional network embedding and entity typing results reported in the appendix.The study includes both shallow and deep neural networks.
  • Knowledge graph completion: HyboNet uses Lorentz embeddings and linear layers for knowledge graph completion on WN18RR and FB15k-237, evaluated with MRR and H@K.The model selects the best validation MRR and reports test performance.
  • Knowledge graph completion: When dimensionality reaches 500, HyboNet significantly outperforms all baselines on MRR, H@3, and H@1, while converging better and reaching a higher ceiling than other hyperbolic networks.The comparison attributes this advantage to the Lorentz linear layer rather than the conventional tangent-space linear layer.
  • Machine translation: On low-dimensional IWSLT’14 and WMT’14 translation, both HyboNet and HATT outperform the Euclidean Transformer, while HyboNet significantly improves over both.HATT performs only attention-weight calculation and aggregation in hyperbolic space, whereas HyboNet performs all operations there.
  • Machine translation: With Transformer-base model size on WMT’14, HyboNet outperforms Transformer and HATT and approaches the much larger Transformerbig.BLEU scores are reported on the test set after selecting the lowest-validation-perplexity model.
  • Dependency tree probing: Dependency probing finds that HyboNet’s intermediate representations reconstruct syntax trees better than other baselines, indicating stronger syntax-structure capture.The probing compares Transformer, HATT, and HyboNet representations on the IWSLT’14 English corpus.

5 Related Work

Prior work adapted diverse neural architectures to hyperbolic spaces, while this paper proposes a fully hyperbolic Lorentz-transformation framework and reframes tangent-space linear transformation as a relaxation of Lorentz rotation.

  • Hyperbolic neural models have been extended to attention, graph, prototypical, and capsule networks for downstream tasks.
  • The paper proposes a fully hyperbolic framework based on Lorentz transformations to address capability limits from tangent-space hybrid architectures.
  • It interprets the hyperbolic linear transformation used in previous work as a relaxation of Lorentz rotation.

6 Conclusion and Future Work

The paper introduces a Lorentz-transformation-based fully hyperbolic framework and reports faster speed, better convergence, and higher performance than other hyperbolic networks, while identifying remaining scope and efficiency challenges.

  • The framework uses Lorentz transformations to overcome capability limits attributed to tangent-space hybrid architectures.
  • Across representative NLP tasks, HYBONET reports faster speed, better convergence, and higher performance than other hyperbolic networks.
  • Future Work: Future work includes applying fully hyperbolic models to computer vision.
  • Limitations: HYBONET remains slower than Euclidean networks and cannot use half-precision training because of floating-point error.

A.1 Graph Neural Networks

HYBONET applies Lorentz-based layers to graph convolutional networks and generally performs well on link prediction and node classification, with especially strong results on highly hyperbolic datasets and improved stability.

  • Setup: HYBONET uses a Lorentz linear layer for feature transformation and neighboring-node centroids for aggregation.
  • Results: The evaluation reports ROC AUC for link prediction and F1 scores for node classification across four network embedding datasets.
  • Results: HYBONET outperforms other baselines on highly hyperbolic datasets and remains competitive on less hyperbolic datasets.
  • Results: HYBONET remains stable and is faster than LGCN, which generated NaN and failed training in 89 of 128 grid-search experiments.

A.2 Fine-grained Entity Typing

The entity-typing model combines mention and context representations in a Lorentz Transformer architecture and achieves strong macro-F1 performance with fewer parameters than several Euclidean and hyperbolic baselines.

  • Task: Fine-grained entity typing predicts multiple labels from coarse, fine, and ultra-fine inventories containing thousands of types.
  • Setup: The model combines a mention representation with distance-based attention over a Lorentz Transformer encoder.
  • Setup: Type labels are predicted when their sigmoid probability exceeds 0.5.
  • Results: HyboNet outperforms LabelGCN and MultiTask on Total with fewer parameters, while matching large Euclidean models on fine and ultra-fine results with significantly fewer parameters.
  • Results: HyboNet outperforms the HY xLarge model on coarse and fine results while using only slightly more parameters than HY base and fewer than HY Large.

B Data Preprocessing Methods

The experiments use task-specific preprocessing for knowledge graph completion, machine translation, network embeddings, and entity typing. Dataset statistics and evaluation tables document the experimental inputs and reported metrics.

  • Knowledge Graph Completion: Knowledge graph completion augments WN18RR and FB15k-237 with reciprocal relations for every triplet.Each (h, r, t) is paired with (t, r−1, h).
  • Preprocessing Conventions: The knowledge graph completion and entity-typing preprocessing follows methods attributed to Balazevic et al. (2019a) and López and Strube (2020), respectively.Entity-typing contexts are trimmed to 25 words and mentions to 5 words while preserving the mention near the context center.
  • Machine Translation: Machine translation preprocessing uses OpenNMT for WMT’14 and FairSeq for IWSLT’14, limiting source and target sentences to 100 words.The source and target vocabularies are not shared.
  • Network Embeddings: Network-embedding experiments use Disease, Air-port, Pubmed, and Cora datasets preprocessed by Chami et al. (2019).The datasets are published in the cited authors’ code repository.
  • Fine-Grained Entity Typing: The dataset for fine-grained entity typing contains 6,000 crowdsourced samples and 6M distantly supervised training samples.

C Experiment Details

The experiments implement Lorentz-model networks with 32-bit arithmetic and task-specific settings. The details cover Lorentz-layer construction, initialization, hyperparameter selection, and probing procedures across the evaluated tasks.

  • General Settings: All experiments use 32-bit floating point numbers, PyTorch, and Lorentz curvature K = −1.
  • Lorentz Linear Layer: Equation (9) constructs a Lorentz linear-layer output by computing its 0-th dimension, scaling the remaining vector, and concatenating the components.The 0-th coordinate uses a sigmoid term with scale λ, bias b, and ε; the spatial vector is scaled to satisfy the Lorentz-model norm.
  • Lorentz Linear Layer: Residual and position-embedding addition also use Equation (9).
  • Knowledge Graph Completion: Knowledge graph completion experiments use reciprocal-relation data, initialization settings, Max Norm constraints, and searches over batch size, margin, MaxNorm, λ, and learning rate.The explored values are specified for each dataset, with the last n embedding dimensions bounded by Max Norm.
  • Dependency Tree Probing: The probing setup maps Transformer representations into Lorentz space and compares Euclidean and Lorentz projections across layers using tree-distance and word-depth reconstruction objectives.The Lorentz Transformer discards the exponential map, probes every layer, and reports the best layer.
  • Fine-Grained Entity Typing: Entity-typing models initialize fixed word embeddings by isometrically projecting pretrained Poincaré GloVe embeddings into the Lorentz model, then encode mentions and contexts with Lorentz components.Mention representations use position encoding and centroid aggregation, while context representations use distance-based attention over a Lorentz Transformer encoder.
Loading 2105.14686v3…