Source-linked AI summary
Hire-MLP: Vision MLP via Hierarchical Rearrangement
Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, Yunhe Wang
TL;DR
Vision MLPs such as MLP-Mixer have difficulty handling different input resolutions and capturing local spatial information. Hire-MLP uses hierarchical rearrangements to model local and global information, and reports competitive results across vision tasks, including 82.1% top-1 accuracy on ImageNet for Hire-MLP-Small.
Problem
MLP-Mixer’s flattened-token design is inflexible to input-resolution changes and rarely explores local information, limiting its suitability as a general vision backbone.
Source: §1 · p. 1Method
Hire-MLP replaces token mixing with channel-mixing MLPs and hierarchical inner-region and cross-region rearrangements to capture local and global information.
Source: §1 · p. 1Results
Hire-MLP improves performance across classification, detection, instance segmentation, and semantic segmentation; Hire-MLP-Small reaches 82.1% top-1 accuracy on ImageNet.
Source: §1 · p. 2Takeaways & Limitations
The authors present Hire-MLP as a versatile vision backbone with a better accuracy-latency trade-off than prior MLP-based and transformer-based architectures.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Previous vision MLPs such as MLP-Mixer and ResMLP accept linearly flattened image patches as input, making them inflexible for different input sizes and hard to capture spatial information. Such approach withholds MLPs from getting comparable performance with their transformer-based counterparts and prevents them from becoming a general backbone for computer vision. This paper presents Hire-MLP, a simple yet competitive vision MLP architecture via \textbf{Hi}erarchical \textbf{re}arrangement, which contains two levels of rearrangements. Specifically, the inner-region rearrangement is proposed to capture local information inside a spatial region, and the cross-region rearrangement is proposed to enable information communication between different regions and capture global context by circularly shifting all tokens along spatial directions. Extensive experiments demonstrate the effectiveness of Hire-MLP as a versatile backbone for various vision tasks. In particular, Hire-MLP achieves competitive results on image classification, object detection and semantic segmentation tasks, e.g., 83.8% top-1 accuracy on ImageNet, 51.7% box AP and 44.8% mask AP on COCO val2017, and 49.9% mIoU on ADE20K, surpassing previous transformer-based and MLP-based models with better trade-off for accuracy and throughput. Code is available at https://github.com/ggjy/Hire-Wave-MLP.pytorch.
1. Introduction
Hire-MLP addresses vision MLPs’ difficulty with flexible input resolutions and local information by using hierarchical rearrangements. Its experiments report competitive results across image classification, detection, and segmentation.
1. Introduction
- MLP-Mixer’s token-count-dependent fully connected layers hinder transfer to different input resolutions, while its limited local-information modeling leaves an important spatial cue underexplored.These limitations motivate an MLP architecture that can handle flexible resolutions while encoding both local and global information.Source: §1 · p. 1
- Hire-MLP replaces token mixing with channel-mixing MLPs and hierarchical rearrangements: inner-region rearrangement captures local information, while cross-region rearrangement communicates across regions for global context.The design is described as handling flexible input resolutions; its operations use reshape and padding.Source: §1 · pp. 1–2
- Hire-MLP uses a hierarchical architecture that generates pyramid features for downstream vision tasks.The architecture includes height- and width-direction branches for spatial information and a channel-mixing branch.Source: §1 · p. 2
- 82.1% top-1 accuracy on ImageNet for Hire-MLP-Small, with larger variants reaching 83.2% and 83.8%.The paper also reports 50.7% box AP and 44.2% mask AP on COCO val2017, and 46.1% single-scale mIoU on ADE20K, 1.6% above Swin-T.Source: §1 · p. 2
2. Related Work
The related-work section traces CNN and vision-transformer development before contrasting MLP-based approaches. It positions Hire-MLP’s hierarchical rearrangement as a way to model local and global spatial information with a favorable performance-efficiency trade-off.
CNN-based Models
- CNN designs progressed from LeNet and AlexNet to architectures emphasizing feature fusion, deep networks, and efficiency.Examples include VGGNet, GoogLeNet, ResNet, MobileNet, and ShuffleNet.Source: §2 · pp. 2–3
Transformer-based Models
- Vision transformers apply transformer architectures to image patches, while later designs address data efficiency, pyramid features, local information, and attention cost.The section notes that self-attention remains computationally expensive and relatively slow on GPUs.Source: §2 · p. 3
MLP-based Models
- MLP-Mixer uses token- and channel-mixing MLPs, while CycleMLP, AS-MLP, and S2-MLP explore alternative spatial operations.Hire-MLP is distinguished by hierarchical rearrangement of tokens within and across local regions to capture local and global information.Source: §2 · p. 3
3. Method
Hire-MLP blocks combine a spatial hire module with channel mixing, using hierarchical rearrangements to capture local and global information. A four-stage pyramid architecture supports downstream vision tasks.
3.1. Hire-MLP Block
- Each Hire-MLP block applies a hire module and then a channel MLP with residual connections, producing intermediate feature Y and output feature Z.The block is stacked iteratively to construct the full architecture.Source: §3.1 · pp. 3–4
3.2. Hierarchical Rearrangement Module
- Inner-region rearrangement mixes tokens within fixed-size regions for local information; cross-region shifting changes region membership to exchange information and aggregate global context.Cross-region rearrangement is inserted before inner-region rearrangement every two blocks, and restoration preserves tokens’ relative positions; fixed region size supports variable-length inputs with linear computational complexity.Source: §3.2 · p. 4
3.3. Overall Architecture
- The four-stage pyramid reduces feature resolution from H/4 × W/4 to H/32 × W/32 while increasing output dimensions, producing features for classification, detection, and segmentation.The architecture applies patch embedding and Hire-MLP blocks, with token counts reduced and channels doubled as it deepens.Source: §3.3 · p. 5
4. Experiments
Experiments evaluate Hire-MLP on ImageNet classification, COCO detection and instance segmentation, and ADE20K semantic segmentation, alongside ablations of its rearrangement module. The reported results compare it with CNN-, transformer-, and MLP-based backbones across these tasks.
4.1. Image Classification on ImageNet
- 83.8% top-1 accuracy at 13.1G FLOPs makes Hire-MLP competitive on ImageNet; the Small model reaches 82.1% with 4.2G FLOPs and outperforms other MLP-based models.The paper also reports better performance than AS-MLP and CycleMLP by 0.5–0.8, and faster inference than transformer-based comparisons.Source: Table 1 · p. 5, §4.1 · p. 6
4.2. Ablation Study
- Region-size ablations indicate that larger regions are needed in lower layers, but further enlargement slightly reduces performance.The authors conjecture that increasing region size may lose information in the bottleneck structure.Source: §4.2 · pp. 6–7
- Removing cross-region shifting eliminates communication between regions and performs poorly; the shifted strategy also outperforms channel shuffle while preserving relative token positions.The step-size ablation identifies the no-shift configuration as lacking global information.Source: Table 3 · p. 6, Table 6 · p. 6, §4.2 · p. 7
- Circular padding is the most suitable tested padding mode for inner-region rearrangement.Padding is needed because the 7×7 stage-4 feature map for 224×224 inputs is not divisible by the region dimensions.Source: Table 4 · p. 6, §4.2 · p. 7
- The inner-region rearrangement is the most important component for local information; removing cross-region rearrangement drops top-1 accuracy to 81.18%, while removing the third branch costs 0.7%.Cross-region restoration contributes about 0.3% top-1 accuracy.Source: Table 5 · p. 6, §4.2 · p. 7
- A two-layer FC bottleneck offers a better accuracy–compute trade-off than one layer, while adding more layers brings no further benefit.The authors attribute improvements to hierarchical rearrangement rather than increased FC-layer count.Source: Table 7 · p. 6, §4.2 · p. 8
4.3. Object Detection on COCO
- Hire-MLP backbones consistently surpass ResNet, PVT, and CycleMLP under similar FLOPs in RetinaNet and Mask R-CNN; Cascade Mask R-CNN exceeds AS-MLP by 0.6–0.7 in both box AP and mask AP with fewer FLOPs.For RetinaNet, Hire-MLP gains +5.8∼7.1 AP over ResNets and +0.3∼2.2 AP over CycleMLPs, with slightly larger model size and FLOPs.Source: Table 8 · p. 7, Table 9 · p. 7, §4.3 · p. 8
4.4. Semantic Segmentation on ADE20K
- On ADE20K, Hire-MLP outperforms CycleMLP and PVT on Semantic FPN (44.3 vs. 42.4) with similar FLOPs and higher FPS, and achieves better multi-scale mIoU than Swin Transformer on UperNet.On UperNet single-scale mIoU, Hire-MLP is +1.6 higher than Swin-T.Source: §4.4 · p. 8, Table 10 · p. 8
5. Conclusion
Hire-MLP hierarchically rearranges tokens to aggregate local and global spatial information. The resulting architecture is effective and efficient, with reported performance improvements across vision tasks.
5. Conclusion
- Hire-MLP uses hierarchical token rearrangements to aggregate both local and global spatial information.The architecture is reported to achieve significant performance improvements across vision tasks.Source: §5 · pp. 8–9
- Inner-region rearrangement lets tokens within each region communicate and mix channels to extract local information.Source: §5 · p. 9
- Cross-region token shifting exchanges information between regions while preserving relative position.Source: §5 · p. 9