Source-linked AI summary
Hyperbolic Hierarchical Clustering for Visual Representation Learning
Jianan Wei, Guikun Chen, Zhiyuan Weng, Chunchao Guo, Yujia Wang, Wenguan Wang
TL;DR
Mainstream vision token mixers are effective but opaque, while clustering-based backbones have not fully realized clustering's potential. The paper introduces ClusterMixer and HCFormer, combining hierarchical clustering with Euclidean patch-level and hyperbolic window-level processing. HCFormer reports strong results across classification, detection, and segmentation, including gains over comparable baselines and prior clustering-based methods.
Problem
Mainstream token mixers are opaque, and prior clustering-based backbones have not fully exploited clustering for visual representation learning.
Method
The paper introduces ClusterMixer and HCFormer, using hierarchical clustering with Euclidean patch-level and hyperbolic window-level similarities.
Results
HCFormer outperforms comparable baselines and prior clustering-based backbones across image classification, semantic segmentation, object detection, and instance segmentation.
Takeaways & Limitations
HCFormer combines built-in interpretability with performance across diverse vision tasks, supporting clustering as a competitive backbone design direction.
Abstract
from arXiv · showhide
We investigate the token mixer in vision backbones by revisiting clustering, one of the most classic approaches in machine learning. An effective token mixer is a fundamental component of modern vision backbones like vision Transformers, facilitating information exchange between image patches. Mainstream token mixers, which rely on convolution, attention, MLP, or their hybrids, primarily focus on navigating the trade-off between accuracy and computational cost. However, a significant drawback of these methods is their black-box nature; their encoding process is opaque and lacks interpretability. Diverging from these opaque designs, we introduce ClusterMixer, a transparent token mixer that is grounded in a clustering paradigm and interpretable by design. ClusterMixer explicitly formulates the token mixing process through a hierarchical clustering mechanism. To model the natural, tree-like relationships inherent in visual data, the clustering is performed in hyperbolic space, which is well-suited for embedding hierarchies with low distortion. Building on this innovation, we present HCFormer, a new backbone architecture that integrates ClusterMixer with a series of meticulously designed clustering strategies to ensure robust performance across tasks. Extensive experiments demonstrate that HCFormer consistently outperforms its counterparts across diverse tasks, including image classification, object detection, instance segmentation, and semantic segmentation. Considering its transparency and efficacy, we hope HCFormer can facilitate a paradigm shift toward interpretable backbones.
1 Introduction
HCFormer addresses the opacity and performance limitations of mainstream and prior clustering-based token mixers with an interpretable, efficient framework built on hierarchical clustering across Euclidean and hyperbolic geometries.
- Prior token mixers deliver incremental gains but remain opaque, while clustering-based backbones have not fully exploited clustering for visual representation learning.
- ClusterMixer initializes tokens and centers, assigns tokens to clusters, and mixes tokens using the resulting cluster structure.
- Hierarchical clustering partitions patches into non-overlapping windows, restricts local mixing, and adds window-level mixing for global context with minimal overhead.
- HCFormer combines Euclidean patch-level clustering with hyperbolic window-level clustering to represent both flat and tree-like relationships.
- HCFormer reduces clustering complexity from quadratic to linear as data points and cluster centers increase, supporting more cluster centers in dense prediction tasks.
- HCFormer exceeds ResNet-50 by 2.6% and Swin-Tiny by 1.1% in ImageNet-1K top-1 accuracy, while improving semantic-segmentation performance by 0.7%~2.8% mIoU.
2 Related Work
Related work establishes clustering as an interpretable and flexible vision tool, hyperbolic geometry as suitable for tree-like data, and clustering-based representation learning as underexplored. HCFormer responds with a clustering-based backbone aimed at strong performance across vision tasks.
- Clustering partitions unlabeled data into groups using similarity measures and has been used in vision preprocessing and downstream tasks.
- Clustering methods are increasingly used because they offer flexibility and interpretability for complex data and neural-network decision analysis.
- Hyperbolic geometry is effective for modeling tree-like structures and has been applied to graphs, images, videos, and representation learning.
- The potential of clustering for representation learning remains underexplored, motivating HCFormer's clustering-based vision-backbone framework.
- HCFormer is presented as achieving strong performance across diverse vision tasks while contributing to a shift toward clustering-based backbones.
3 Method
HCFormer uses ClusterMixer to mix visual tokens through soft clustering, hierarchical windows, and dual Euclidean–hyperbolic similarity. Local patch clustering reduces complexity, while window-level clustering supplies global context and hyperbolic geometry models abstract relationships.
- ClusterMixer: HCFormer replaces conventional token mixers with ClusterMixer, which estimates centers, assigns tokens softly, and mixes features through aggregation and propagation.Cluster centers are estimated by average pooling, while assignments use feature similarity and relative positional bias.
- ClusterMixer: Average pooling estimates cluster centers with complexity linear in sequence length and no learnable parameters.The pooling size is K, and the number of centers per row is M_w = W/K.
- ClusterMixer: Soft clustering lets tokens belong to multiple clusters, while relative positional bias helps establish relational dependencies.The assignment combines similarity scaled by learnable α with learnable bias b_i,j.
- Hierarchical clustering: Hierarchical clustering partitions patches into local windows and then clusters windows globally, transforming ClusterMixer’s quadratic complexity into linear complexity.Window-level clustering represents each window by its mean patch embedding and restores global contextual interaction.
- Hyperbolic hierarchical clustering: Hyperbolic similarity is computed after mapping Euclidean tokens and centers onto the Lorentz hyperboloid using an exponential map.The Lorentz model is selected for numerical stability and computational efficiency, and similarity is measured by Lorentzian distance.
- Dual geometries: HCFormer uses Euclidean similarity for fine-grained patch clustering and hyperbolic similarity for abstract window clustering.The two computations remain parallel and decoupled, with window features upsampled before concatenation with patch features.
4 Experiment
HCFormer is evaluated across image classification, semantic segmentation, object detection, instance segmentation, and ablation studies. It outperforms comparable backbones while combining performance with computational efficiency and interpretable clustering behavior.
- Downstream Tasks: HCFormer is evaluated for semantic segmentation on ADE20K with Semantic FPN and for COCO detection and instance segmentation with Mask R-CNN.The reported evaluation uses mIoU for semantic segmentation and APbox and APmask for object detection and instance segmentation.
- Object Detection and Instance Segmentation: At the 34M scale, HCFormer achieves 38.7% APbox and 36.0% APmask, exceeding CoC and FEC under identical network sizes.The corresponding comparisons are 38.7% vs. 37.2% vs. 37.9% APbox and 36.0% vs. 35.4% vs. 35.5% APmask.
- Efficiency: HCFormer-Medium achieves 37% higher throughput than CoC-Medium/49, measuring 14.0 versus 10.2, while maintaining competitive accuracy.HCFormer-Nano also exceeds FEC-Small by 0.4% on both APbox and APmask with fewer parameters.
- Ablation and Diagnostics: Ablations show that hierarchical clustering outperforms shifted windows for ClusterMixer, while hyperbolic curvature variation has minimal impact.The baseline without proposed components achieves 71.9%, and higher hyperbolic cluster counts peak at 16 clusters in the reported setup.
5 Conclusion
The paper presents ClusterMixer and HCFormer as interpretable, clustering-based alternatives for visual representation learning. HCFormer improves performance over existing cluster-based approaches while retaining the claimed interpretability benefits of clustering.
- ClusterMixer uses clustering for token aggregation, while HCFormer combines it with tailored training strategies in a universal vision framework.
- The framework improves interpretability over conventional convolution- and attention-based methods while achieving superior performance to existing cluster-based approaches.
- The authors expect HCFormer’s balance of interpretability and performance to benefit a wider range of visual tasks.