PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the Fully Hyperbolic Vision Transformer in Polar Coordinates

Ahmad Bdeir, Niels Landwehr

arXiv:2609.32899v1cs.CV

TL;DR

Ambient-coordinate Lorentz operations become numerically unstable at large radii, prompting radius constraints that limit use of hyperbolic space. Polar-ViT reformulates transformer operations in polar coordinates and adds hyperbolic layers, positional encodings, and residuals. It outperforms Euclidean and hyperbolic baselines on vision benchmarks and transfers from ImageNet-1k to fine-grained classification.

  • Problem

    Ambient-coordinate Lorentz operations become numerically unstable at large radii, leading models to constrain embedding radii and limiting use of hyperbolic space.

    Source: §1 · pp. 1–2

  • Method

    Polar-ViT uses polar-form hyperbolic operations, separately maps direction and radius, adds horospherical positional encodings, and uses Lorentz-boost residuals.

    Source: §1 · p. 2, §4.4 · p. 8

  • Results

    Polar-ViT outperforms Euclidean and hyperbolic baselines on standard vision benchmarks and ImageNet-1k, and its ImageNet-1k checkpoints transfer to fine-grained classification.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Polar operations lessen the need for normalization, clipping, and rescaling, allowing more effective use of hyperbolic space.

    Source: §1 · p. 2

    Further work is needed on finetuning and on sensitivity to the pretraining recipe and hyperparameters.

    Source: §6 · p. 9

Abstract

from arXiv · show

Hyperbolic space can embed intrinsic hierarchies in data with low distortion due to the exponential growth of volume with distance from the origin. However, current Lorentz transformer blocks are formulated in ambient coordinates, where numerical errors increase at large radii due to instability in the Lorentzian inner product, leading many models to limit the radius to avoid this issue. This prevents us from utilizing the regions of hyperbolic space that motivate the geometry. To address this, we revisit the components of the transformer block in polar coordinates, where hyperbolic operations such as distance calculation and attention centroids can be computed without the numerical cancellation in their ambient-coordinate formulations. Specifically, we propose a polar fully connected layer that separately maps an embedding's direction and radius, allowing the radius of a feature to be learned rather than determined by the norm of a linear map. We additionally introduce horospherical shifts as relative positional encodings whose query-key distances grow logarithmically with the token gap. Finally, we reformulate the residual connection as average radius Lorentz boosts. Combining these components, we develop a fully hyperbolic transformer that substantially improves performance over Euclidean and hyperbolic baselines on standard vision tasks. We further evaluate our model on ImageNet and demonstrate its ability to generalize to other datasets using pre-trained weights, similar to Euclidean counterparts.

1 INTRODUCTION

Ambient-coordinate instability at large radii leads existing Lorentz models to constrain embeddings, while Polar-ViT reformulates transformer operations in polar coordinates. Its components include a polar fully connected layer, horospherical positional encodings, and intrinsic residual and normalization alternatives.

1 INTRODUCTION

  • Existing radius constraints mitigate Lorentz instability but limit access to the volume that motivates hyperbolic geometry; tangent-space aggregation has also been proposed to avoid the centroid operation.The Lorentz failure can silently return incorrect points, degrading or diverging training without an identifiable component-level cause.Source: §1 · pp. 1–2
  • Euclidean components for positional encodings, residuals, and normalization can introduce further instability when recomputing the time coordinate from space values.The time component grows exponentially with radius, making its recomputation subject to cancellation errors.Source: §1 · p. 2
  • Polar-ViT uses polar operations to keep weighted centroids and distances accurate beyond ambient limits, reducing the need for radius-constraining normalization, clipping, and rescaling.It represents embeddings by geodesic radius and unit direction.Source: §1 · p. 2
  • In single precision, the ambient weighted centroid fails beyond radius 8.3, while its polar form remains accurate to about radius 17.3.The instability also affects distances and weighted centroids in ambient coordinates.Source: §1 · pp. 1–2
  • The polar fully connected layer maps direction and radius separately, while horospherical shifts provide relative positional encodings that improve over Euclidean encodings alone.The paper also proposes intrinsic alternatives to residual connections and Layer Normalization.Source: §1 · p. 2
  • Polar-ViT releases Tiny and Small ImageNet-1k checkpoints that transfer effectively to fine-tuning tasks, lowering adoption costs for hyperbolic models.Source: §1 · p. 2

2 RELATED WORK

Related work develops hyperbolic neural networks and transformers, but ambient-coordinate operations constrain usable radii. Prior approaches to numerical stability include increasing precision, bounding radii, or changing parameterization; prior positional encodings include Euclidean and Lorentz transformations.

Hyperbolic Neural Networks

  • Lorentz networks extended hyperbolic models to vision and language, but ambient-coordinate storage limits the radii at which Lorentz operations remain well-defined.Source: §2 · p. 2

Hyperbolic Transformers

  • Hyperbolic transformers use Lorentz attention and weighted centroids, while vision models include Poincaré-ball HVT and Lorentz LViT; tangent-space aggregation is another response to centroid precision issues.LViT was the first fully hyperbolic Lorentz ViT trained on ImageNet-1k, but its transfer tests used HypLoRA.Source: §2 · pp. 2–3

Numerical Stability of Hyperbolic Representations

  • Prior stability methods increase precision, bound the radius, or change parameterization; Polar-ViT applies polar coordinates to transformer operations including the linear layer, attention, and centroid.Source: §2 · p. 3

Positional Encodings

  • Relative positional encodings make attention depend on index gaps; prior Lorentz approaches use rotations or boosts, whereas horospherical shifts slide points along horospheres.Horospheres had been used for prototypes, decision boundaries, dimensionality reduction, and linear layers, but not positional encoding.Source: §2 · p. 3

3 THE LORENTZ MODEL

The Lorentz model represents hyperbolic points on a hyperboloid, while polar coordinates encode each point by radius and direction. Ambient distance and centroid calculations can lose precision through cancellation at large radii; polar formulations avoid that subtraction and remain accurate farther out.

3 THE LORENTZ MODEL

  • Polar coordinates represent a point by its geodesic radius r and unit direction u, stored to relative precision ε at every radius.The ambient coordinates instead grow exponentially with radius, increasing their rounding error.Source: §3 · p. 3

3.1 PRECISION LOSS IN AMBIENT COORDINATES

  • For nearby points, distance information resides in c = cosh(dL(x, y)/√k) − 1, which is approximately dL(x, y)^2/(2k); ambient error can overwhelm c.At equal radius a, the absolute rounding error is εe^(2a)/4, making the computed distance meaningless once it exceeds c.Source: §3.1 · p. 4
  • For k = 1, ambient distance error exceeds the distance at radius 8.3 in single precision, 18.4 in double precision, and 2.8 in bfloat16.The logarithmic map, parallel transport, and tangent inner product of Riemannian optimizers fail at the same radius.Source: §3.1 · p. 4
  • Ambient centroid normalization subtracts two large squared quantities, so rounding can corrupt the result, reverse a decay step, or produce NaN.At radius 8, single precision returns radius 8.14 instead of the expected 7.985; at radius 9, the subtraction returns zero.Source: §3.1 · pp. 4–5

3.2 POLAR OPERATIONS AND THEIR PRECISION

  • Polar distance uses radial and angular terms without subtraction, while the polar centroid decomposes its denominator into sums of non-negative terms.For the centroid, the radial term E shrinks outward and the angular term S measures directional spread, vanishing when directions coincide.Source: §3.2 · p. 5
  • Polar operations remain accurate to the stored-point limit: about radius 17.3 in single precision and 6.2 in bfloat16, versus 8.3 and 2.8 in ambient form.Source: §3.2 · p. 5

4 METHODOLOGY

Polar-ViT stores embeddings by geodesic radius and unit direction, then builds its transformer operations around polar formulations. Its components separately control feature radius, aggregate attention values, encode relative position, and update residuals.

4.1 POLAR FULLY CONNECTED LAYER

  • Polar-FC maps the output direction from a learned linear transformation while separately parameterizing an unbounded output radius.Its radius uses weighted contributions from output norm, input radius, input direction, and a bias; this avoids the slow growth and fixed bound of prior Lorentz layers.Source: §4.1 · p. 6

4.2 ATTENTION WITH POLAR CENTROID AGGREGATION

  • Attention scores use negative squared polar distances, and values are aggregated with a polar centroid in a fused kernel that avoids storing the N × N score matrix.The attention layers use Polar-FC for queries, keys, and values.Source: §4.2 · pp. 6–7

4.3 HOROSPHERICAL POSITIONAL ENCODING

  • Horospherical shifts encode relative position while making query-key distance grow logarithmically with token gap.A learned per-head rate sets the shift for each token position, allowing heads to learn how local their attention is.Source: §4.3 · p. 7

4.4 NORMALIZATION AND RESIDUALS

  • Normalization computes centroid and dispersion over each sample’s token axis, with polar scalar multiplication rescaling radius while leaving direction unchanged.This normalization has no running state and does not couple samples.Source: §4.4 · p. 7
  • Boost residuals split updates into parallel and perpendicular components, centering the radial step across each sample’s tokens so the residual preserves their mean radius.The perpendicular boost changes direction without changing radius; the combined update is not itself a single boost.Source: §4.4 · p. 8

5 EXPERIMENTS

Polar-ViT is evaluated on image classification from scratch, ImageNet-1k, fine-grained transfer, and medical segmentation. It outperforms hyperbolic baselines in the reported classification comparisons, while its transfer advantage over Euclidean ViT is narrower than that of a stronger DeiT checkpoint.

5 EXPERIMENTS

  • Polar U-Net matches Euclidean U-Net and beats hyperbolic baselines in mean Dice on almost all five medical segmentation datasets.The paper reports this as an evaluation of polar components beyond transformers.Source: §5 · p. 8

5.1 IMAGE CLASSIFICATION FROM SCRATCH

  • 4.3%: ViT-Tiny Polar-ViT exceeds HexFormer-Hybrid on both CIFAR-100 and Tiny-ImageNet, while ViT-Base exceeds it by 4.8% on CIFAR-100.On Tiny-ImageNet at Base size, Polar-ViT reaches 68.19 versus 53.01 for LViT.Source: §5.1 · p. 8, Table 1 · p. 9

5.2 IMAGENET-1K

  • 3.5% and 2.2%: Polar-ViT exceeds the Euclidean ViT on ImageNet-1k at Tiny and Small sizes, respectively.Each size-matched pair was trained for 90 epochs; accuracy is averaged over three seeds.Source: §5.2 · p. 8, Table 2 · p. 9

5.3 TRANSFER TO FINE-GRAINED CLASSIFICATION

  • 3.9% and 5.3%: Tiny Polar-ViT exceeds Euclidean ViT on CUB and Aircraft, but a DeiT-S-based Flat-ViT exceeds it by 3.1% and 0.6%, respectively.The Flat-ViT checkpoint was trained for 300 epochs and reaches 79.9%.Source: §5.3 · p. 9, Table 3 · p. 9

6 CONCLUSION

The conclusion presents polar formulations as a more stable way to compute Lorentz weighted centroids and distances, and uses them to build Polar-ViT for vision benchmarks and transfer.

6 CONCLUSION

  • Polar-ViT combines separate direction-radius mapping, horospherical positional encoding, and residual connections formulated as isometries at any radius.The paper reports that weighted centroids and distances fail in single precision and that Polar-ViT outperforms Euclidean and hyperbolic baselines on standard vision benchmarks and ImageNet-1k.Source: §6 · p. 9

A AMBIENT FAILURE SITES IN A LORENTZ TRANSFORMER

Ambient Lorentz arithmetic can fail in centroid and optimizer operations at large radii, while polar forms express these operations without the same large-term cancellations. The appendix derives polar formulations for tangent-space operations and bounds their precision.

A AMBIENT FAILURE SITES IN A LORENTZ TRANSFORMER

  • Beyond r≈9, manifold weight decay’s centroid step moves a parameter outward instead of toward the origin.Table 4 lists centroid computations and conditions under which ambient arithmetic computes them correctly.Source: A AMBIENT FAILURE SITES IN A LORENTZ… · p. 13, Table 4 · p. 13
  • When ambient centroid normalization fails, rounding noise can place the output at an incorrect radius, and its gradient can greatly exceed the true tangent gradient.For leaked weight δ = 10−5, the single-precision relative gradient error reaches 0.6 at r = 8.Source: A AMBIENT FAILURE SITES IN A LORENTZ… · p. 13

B.1 TANGENT VECTORS IN THE POLAR FRAME

  • In the polar tangent frame, the Lorentzian inner product becomes a sum of radial and perpendicular products, avoiding the ambient cancellation in the radial norm.The ambient norm of a radial unit vector is computed as 0.87 at a = 8 and 0 at a = 12 in single precision.Source: §B.1 · p. 14

B.2 GRADIENT PROJECTION

  • Polar gradient projection uses radial and perpendicular components without division or subtraction of large terms.Models built from these polar operations have g0 = 0, making the stated projection exact.Source: §B.2 · p. 14

B.3 EXPONENTIAL MAP

  • The exponential map updates the radius in log space and rewrites inward-step terms so large quantities have the same sign.The log-space update avoids forming ea; the inward-step rewrite addresses cancellation in the ambient expression.Source: §B.3 · p. 14

B.4 LOGARITHMIC MAP √

  • The logarithmic map gives a purely radial tangent for points sharing a direction and a perpendicular tangent for nearby points at equal radius.For equal-radius points with a small angle, the perpendicular component approaches k sinh a θ.Source: §B.4 · p. 15, §B.5 · p. 15

B.5 PARALLEL TRANSPORT

  • Parallel transport evaluates the point inner product with the polar distance formula and the tangent inner product with the polar-frame expression.The optimizer retraction applies the exponential map followed by this transport.Source: §B.5 · p. 15

B.6 GYROADDITION

  • Polar gyroaddition reduces centering error: at r = 8, its geodesic error is 4.0 · 10−4 versus 0.70 for the ambient form.The polar form’s error stays within a factor of two of the resolution limit across the reported radii.Source: §B.6 · p. 15

B.9 PROOF OF PROPOSITION 2

The appendix explains the precision bound, derives horospherical shifts and their relative-position behavior, and describes implementation choices for stable polar arithmetic. It also specifies initialization choices for the models.

B.9 PROOF OF PROPOSITION 2

  • The polar-operation error bound follows from sums of non-negative terms and well-conditioned transformations, with at most 12 roundings in the longest centroid chain.The largest formed quantity overflows single precision at a = 88.7, and measured error stays within a factor of two of the bound.Source: §B.9 · p. 16

C.1 GENERATOR AND CLOSED FORM

  • The horospherical shift preserves the Lorentzian inner product and slides each horosphere along itself.Its generator has a terminating exponential series because A3 = 0.Source: §C.1 · p. 16

C.2 RELATIVE PROPERTY

  • Query-key squared distance depends on the relative positional offset along each grid axis; combining two axes preserves this property only to first order in their rates.The two shifts do not commute.Source: §C.2 · p. 16

C.3 DISTANCE LAW

  • Horospherical distance grows logarithmically for large shifts, so attention scores decay quadratically in ln t and more slowly at long range than nearby.Points farther toward the ideal point are less sensitive to position because the same shift moves them less.Source: §C.3 · pp. 16–17

D.1 POLAR ARITHMETIC

  • The implementation computes angular terms as squared direction chords and centers their gradients on the mean key direction to preserve precision.It computes centroid radius in the log domain, avoiding intermediate overflow before cosh a itself overflows.Source: §D.1 · p. 17
  • Compiled polar attention in a 9-block ViT takes the same time and memory as the ambient form.Layer scales and attention-head radius gains are calibrated on the first training batch, and horospherical rates are excluded from weight decay.Source: §D.1 · p. 17, §D.2 · p. 17

D.3 TRAINING PROTOCOL

The training protocol follows HexFormer’s recipe for ViT-Tiny and ViT-Base models, with specified optimization, augmentation, and radius-parameter decay choices.

D.3 TRAINING PROTOCOL

  • ViT-Tiny uses 9 layers, width 192, MLP width 384, and 12 heads; ViT-Base uses 12 layers, width 768, MLP width 3072, and 12 heads.Both use patch size 4 and curvature −1; results are averaged over 10 Tiny seeds and 5 Base seeds.Source: §D.3 · p. 17
  • All models train for 100 epochs with batch size 128, augmentations, label-smoothed cross-entropy, and Riemannian AdamW.Tiny models use learning rate 3 · 10−3, with cosine scheduling and 10 warmup epochs.Source: §D.3 · p. 17
  • Scale parameters are not weight-decayed, while polar fully connected layer radius parameters are decayed toward their initial values.Source: §D.3 · p. 17

D.4 IMAGENET-1K TRAINING AND FINE-TUNING

This section specifies the CUB-200-2011 and FGVC-Aircraft data and the ImageNet models and fine-tuning protocol. It also identifies Table 5 as a component ablation, without supplying its outcomes here.

D.4 IMAGENET-1K TRAINING AND FINE-TUNING

  • CUB-200-2011 and FGVC-Aircraft have no validation split, so test accuracy is reported at the last epoch without checkpoint selection.CUB-200-2011 has 5,994 training and 5,794 test images; FGVC-Aircraft has 6,667 trainval and 3,333 test images.Source: §D.4 · p. 17
  • Table 5 covers ten-seed CIFAR-100, ViT-Tiny component ablations, with each row adding one change to the preceding row.Source: Table 5 · p. 18
  • ImageNet-1k comparisons use matched-depth, width, head-count, and patch-size Euclidean ViT and Polar-ViT models at Tiny and Small sizes.The models train for 90 epochs with Beyer et al.’s recipe; the Polar-ViT uses Riemannian AdamW.Source: §D.4 · p. 18
  • Fine-tuning loads pretrained weights except for a newly initialized ImageNet classifier, then uses the H-CAST recipe for 100 epochs.The ViT-Small Polar-ViT reaches effective batch size 256 via gradient accumulation; the other models use batch size 256 as 128 × 2.Source: §D.4 · p. 18

E.1 COMPONENT ABLATION

The ablations measure component gains, compare output-radius rules, and test fully connected layer convergence across target radii.

E.1 COMPONENT ABLATION

  • 2.3 points from polar form plus scaled gyro layer norm, and horospherical encoding adds 2.1 points.Boost residual adds 0.8 points; polar fully connected adds 0.8 over Lorentz fully connected, while polar boost residual keeps accuracy about the same, removes embedding drift, and reduces spread to a third. The same-axis Lorentz-boost positional alternative reaches 79.82 ± 0.13, 0.3 points below the horospherical shift.Source: §E.1 · p. 18

E.2 RADIUS RULE OF THE POLAR FULLY CONNECTED LAYER

  • Random-start radius rules are 3.2 to 4.4 points below anchored initialization, while the hard-bound rule diverges.Moving parameters from the anchor reduces accuracy by 0.4 points at σ = 0.1 and 1.1 points at σ = 0.2.Source: §E.2 · pp. 18–19, Table 6 · p. 19

E.3 EXPRESSIVITY TEST

  • Without clipping, Polar-FC reaches every target in both precisions in 70 to 650 iterations, fewer than any other layer beyond radius 3.Targets span radii 1 to 18; Chen et al.'s layer reaches the iteration limit at radius 6 in Float64 and radius 5 in Float32.Source: §E.3 · p. 19

E.4 NUMERICAL PRECISION

The numerical tests compare ambient and polar operations across radii, and the computational-cost test measures training-step time.

E.4 NUMERICAL PRECISION

  • At r = 8, ambient squared distance and centroid deviate from double precision; beyond it, the ambient centroid returns radius 9 for every input.Polar operations stay within a factor of two of the Proposition 2 resolution limit at every radius. In the low-weight regime at r ≥ 8, 88 to 100% of ambient attention rows are off by more than 0.5 in geodesic distance, while the polar weight-decay step matches double precision at every radius.Source: §E.4 · p. 19, Table 7 · p. 19

E.5 COMPUTATIONAL COST

  • The polar model takes 1.1 times as long as the ambient Lorentz model and 2.8 times as long as Euclidean ViT per training step.At this input size, the fused attention kernel reduces attention-core time from 4.5 to 3.0 ms; direct polar-layer exchange agrees with the ambient model to 10^-14 in double precision.Source: §E.5 · p. 20, Table 8 · p. 20

F POLAR LAYERS IN A CONVOLUTIONAL NETWORK

The polar layer forms a Lorentz U-Net by replacing its layers with polar forms, and the benchmark compares segmentation performance across five datasets.

F POLAR LAYERS IN A CONVOLUTIONAL NETWORK

  • Replacing convolution, batch normalization, and activation with polar forms adapts the Lorentz U-Net without changing its structure.Each convolution concatenates its receptive field in polar form and applies Polar-FC; the polar network uses no radius cap.Source: F POLAR LAYERS IN A CONVOLUTIONAL… · p. 21
  • The evaluation uses five datasets and splits, shared U-Net topology, and 50 training epochs for both Lorentz models.The datasets cover skin lesions, polyps, and breast ultrasound images; the models are trained with the same recipe.Source: F POLAR LAYERS IN A CONVOLUTIONAL… · p. 21
  • The polar U-Net exceeds the ambient Lorentz U-Net on all five datasets by 0.01 to 0.08 Dice, with 0.5 K fewer parameters.It exceeds the published Poincaré U-Net on four datasets and is 0.02 below it on BUSI; versus the Euclidean U-Net, it is higher on ISIC 2016 and SANET, equal on ISIC 2018 and Kvasir-SEG, and 0.04 lower on BUSI while using 0.44 times its parameters.Source: F POLAR LAYERS IN A CONVOLUTIONAL… · pp. 21–22, Table 9 · p. 22
  • The polar model is more stable than the ambient Lorentz model; some ambient-model training runs required gradient clipping and a lower learning rate.Source: F POLAR LAYERS IN A CONVOLUTIONAL… · p. 22

Related papers

Loading 2609.32899v1…