Source-linked AI summary
Hyperbolic Vision Transformers: Combining Improvements in Metric Learning
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, Ivan Oseledets
TL;DR
Metric learning needs distances that reflect semantic similarity, while conventional embedding spaces may not adequately represent the hierarchical structure of natural data. This paper combines vision transformers, hyperbolic embeddings, and pairwise cross-entropy loss, and reports improved image-retrieval performance across evaluated formulations and datasets. The authors conclude that the combined components are important for achieving the best performance, with Hyp-DINO combining self-supervised and metric learning under minimal supervision.
Problem
Metric learning seeks embedding distances that group semantically similar objects and separate dissimilar ones, while hyperbolic geometry is motivated by the hierarchical structure of natural data.
Method
The method projects vision-transformer embeddings into the Poincaré ball and optimizes them with pairwise cross-entropy loss using hyperbolic distances.
Results
The hyperbolic counterpart outperforms the Euclidean setting, and the joint vision-transformer, hyperbolic-embedding, and pairwise-cross-entropy configuration provides the best image-retrieval performance.
Takeaways & Limitations
The authors conclude that each proposed component is crucial for best performance and that self-supervised learning and metric learning complement each other in Hyp-DINO.
Takeaways & Limitations
The study considers only the vision domain and category-level retrieval, leaving multi-domain common semantic metrics as a possible development.
Abstract
from arXiv · showhide
Metric learning aims to learn a highly discriminative model encouraging the embeddings of similar classes to be close in the chosen metrics and pushed apart for dissimilar ones. The common recipe is to use an encoder to extract embeddings and a distance-based loss function to match the representations -- usually, the Euclidean distance is utilized. An emerging interest in learning hyperbolic data embeddings suggests that hyperbolic geometry can be beneficial for natural data. Following this line of work, we propose a new hyperbolic-based model for metric learning. At the core of our method is a vision transformer with output embeddings mapped to hyperbolic space. These embeddings are directly optimized using modified pairwise cross-entropy loss. We evaluate the proposed model with six different formulations on four datasets achieving the new state-of-the-art performance. The source code is available at https://github.com/htdt/hyp_metric.
1. Introduction
Metric learning seeks embedding distances that reflect semantic similarity across similar and dissimilar examples, supporting tasks such as retrieval, recognition, and re-identification. The paper combines vision transformers, hyperbolic embeddings, and pairwise cross-entropy loss to improve image retrieval.
- Motivation: Metric learning learns representations whose distances encode semantic similarity between similar and dissimilar objects.It supports content-based retrieval, near-duplicate detection, face recognition, person re-identification, and zero- or few-shot learning.
- Motivation: Vision transformers offer a competitive alternative to CNN-based encoders and may produce more general features for retrieval of unseen image classes.Their weaker inductive biases can require more training data for fair generalization.
- Motivation: Hyperbolic spaces provide exponential volume growth, supporting hierarchical data representation and low-dimensional embeddings without sacrificing representation power.The paper motivates hyperbolic geometry because natural data is often intrinsically hierarchical.
- Motivation: Pair-based losses operate directly on embeddings and capture fine-grained interactions using pair relationships rather than requiring class labels.This property supports pairwise cross-entropy loss in self-supervised learning.
- Contributions: The proposed method projects embeddings into the Poincaré ball and applies pairwise cross-entropy loss with hyperbolic distances.The method combines a vision transformer encoder, dimensionality reduction, and hyperbolic-space optimization.
- Contributions: The joint use of vision transformers, hyperbolic embeddings, and pairwise cross-entropy loss provides the best performance for image retrieval.The paper evaluates these components as a combined architecture for metric learning.
2. Method
The method combines a vision transformer encoder with hyperbolic embeddings and a pairwise cross-entropy loss, using the Poincaré ball to represent and compare samples. It also estimates data hyperbolicity, applies feature clipping for numerical stability, and evaluates several transformer configurations.
- Pairwise Cross-Entropy Loss: The proposed loss operates in hyperbolic space, bringing positive representations closer while pushing negatives away through hyperbolic distances.It combines the representational expressiveness of hyperbolic space with the simplicity and generality of cross-entropy.
- Hyperbolic Embeddings: Hyperbolic distance approaches the Euclidean distance as curvature c tends to zero, while exponential volume growth provides greater representation power for tree-like structures.The Poincaré ball has a curvature parameter c and expands distances near its boundary.
- Hyperbolic Embeddings: Embeddings are projected into the Poincaré ball and optimized directly with hyperbolic distances rather than being mapped back to Euclidean space.The training pipeline uses the encoder output, an exponential mapping, and hyperbolic representations denoted z_i.
- Pairwise Cross-Entropy Loss: Each batch samples N categories with two images per category, producing K = 2N samples and N positive pairs; the loss is evaluated in both pair directions.The distance can be either hyperbolic or cosine, with temperature τ controlling the pairwise cross-entropy loss.
- δ-Hyperbolicity: Relative δ-hyperbolicity is used to assess how closely image embeddings match hyperbolic structure and to select the Poincaré-ball radius for each dataset.Lower relative δ values indicate greater intrinsic data hyperbolicity, and the analysis covers CUB-200, Cars-196, SOP, and In-Shop embeddings from three encoders.
3. Experiments
The experiments compare six hyperbolic and spherical vision-transformer formulations with existing retrieval methods across four benchmark datasets, while also testing architectural and hyperparameter choices. Hyperbolic embeddings generally improve over spherical versions, with performance depending on pretraining, dataset, curvature, dimensionality, and batch size.
- 3. Experiments: The study compares six method versions with current state-of-the-art approaches on four category-level image-retrieval benchmarks.The six versions vary hyperbolic versus hyperspherical embeddings and ViT pretraining schemes, and use 128-dimensional head embeddings or encoder embeddings.
- 3.3. Results: 9% separates Hyp-ViT from the second-best IRTR method on CUB, while ImageNet-21k pretraining contributes substantially to the strongest CUB performance.Hyp-DINO and Hyp-DeiT show smaller improvements, whereas the unfine-tuned ViT-S baseline is already strong; the authors hypothesize that bird classes in pretraining help separation.
- 3.3. Results: Hyperbolic versions consistently improve performance over spherical versions, while Hyp-ViT and Hyp-DINO are similar on SOP and In-Shop but differ substantially on Cars-196.DeiT versions outperform CNN-based models but perform worse than ViT- and DINO-based encoders.
- 3.3. Results: Training embeddings on the Cars-196 set form label-based clusters near the Poincaré-disk boundary, whereas test embeddings partly intermingle toward the center.The visualization uses UMAP with the hyperboloid distance and preserves image neighborhood relations; the test structure is described as showing possible hierarchical relationships.
- 3.4. Impact of Hyperparameters: Curvature values in (0.01, 0.3) are robust, but larger values degrade performance; the default curvature is reduced to 0.1 for stability.The head degrades faster because hyperbolic distance is also used for evaluation, and smaller curvature values are associated with better stability.
- 3.4. Impact of Hyperparameters: Lower embedding dimensionality reduces recall, while the method remains robust for batch sizes ≥400 and needs neither distributed multi-GPU training nor a momentum network on the considered datasets.The experiments report reasonable representation power even at lower dimensions despite 3,985 test categories.
4. Related Work
Prior work spans hyperbolic visual embeddings, vision-transformer metric learning, self-supervised learning, and pairwise metric-learning losses. This paper extends pairwise cross-entropy from Euclidean to hyperbolic distances.
- Hyperbolic embeddings have been applied to visual few-shot and zero-shot learning, including hybrid and kernelized architectures.
- Vision transformers have shown beneficial properties for category-level and object retrieval, with prior methods combining DeiT-style architectures, contrastive loss, and additional regularization.
- DINO trains vision transformers self-supervised and performs well on k-NN classification and image retrieval, motivating its use for metric learning.
- Classic metric-learning losses include contrastive, triplet-margin, and lifted-structure formulations.
- Pairwise cross-entropy losses such as InfoNCE and NT-Xent typically use Euclidean distances, whereas this method extends the loss to hyperbolic space.
5. Conclusion
The paper combines hyperbolic pairwise cross-entropy, vision transformers, and pretraining schemes for metric learning, and empirically evaluates their contributions. It reports category-level retrieval results while identifying broader-domain evaluation as future work.
- The method combines pairwise cross-entropy with hyperbolic distance and vision transformers using several pretraining schemes.
- The authors empirically verified that each proposed component is crucial for the best performance by comparing method versions under equal setups.
- Hyp-DINO combines self-supervised learning and metric learning with minimal supervision and is described as a powerful metric.
- The evaluation covers only the vision domain and category-level retrieval task, leaving multi-domain semantic metrics as a possible development.
- Metric-learning applications such as face recognition and person re-identification may involve privacy risks and learned social biases.
A. Things we tried but they did not work
Preliminary experiments tested alternative backbones and losses before focusing on the transformer-based final method. Several tried variants were omitted because they performed similarly or worse.
- A hyperbolic ResNet-50 version outperformed the sphere-based version, but CNN comparisons were omitted because the transformer backbone performed clearly better.
- A MoCo-loss modification performed similarly to plain cross-entropy and was excluded from the final version.
- The ProxyNCA-loss variant performed worse and was not included in the final method.
- The initial experiments focused on self-supervised learning in hyperbolic space before the method shifted toward metric learning.
B. Datasets visualization
The visualization section shows learned hyperbolic embeddings and corresponding evaluation images for CUB-200-2011 and Cars-196. Embedding layouts are reduced to two dimensions, while image grids preserve neighborhood relations.
- Figures 1–4 visualize Hyp-DINO embeddings for CUB-200-2011 and Cars-196 on the Poincaré disk using UMAP with hyperboloid distance.
- Different colors in the embedding visualizations indicate different classes, and each point represents a sample.
- Figures 5 and 6 show the first 4000 evaluation-split images for CUB-200-2011 and Cars-196.
- The image layouts project the embedding arrangements onto a uniform 2D grid while preserving sample neighborhood relations.