Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

Qibin Hou; Zihang Jiang; Li Yuan; Ming-Ming Cheng; Shuicheng Yan; Jiashi Feng

Source-linked AI summary

Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, Jiashi Feng

arXiv:2106.12368v1Jun 23, 2021cs.CV

TL;DR

MLP-like vision models can be efficient but often require large-scale data and lose positional information when spatial dimensions are flattened. Vision Permutator preserves three-dimensional features and separately encodes height and width, achieving strong ImageNet results without extra training data while retaining input-shape limitations.

Problem
Existing MLP-like models can lag without large-scale training data and lose positional information by flattening spatial dimensions.
Method
Vision Permutator retains three-dimensional token representations and separately encodes height, width, and channel information with Permutator blocks.
Results
81.5% top-1 accuracy is achieved on ImageNet without extra training data, and the 88M model reaches 83.2% accuracy.
Takeaways & Limitations
Separate height and width encoding improves performance over MLP-like models that treat the two spatial dimensions as one.
Takeaways & Limitations
Fixed fully connected layer shapes prevent arbitrary-sized input images and limit downstream use with varying input sizes.

Abstract

from arXiv · show

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies along one spatial direction and meanwhile preserve precise positional information along the other direction. The resulting position-sensitive outputs are then aggregated in a mutually complementing manner to form expressive representations of the objects of interest. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. Code is available at https://github.com/Andrew-Qibin/VisionPermutator.

1 INTRODUCTION

Vision Permutator targets data-efficient MLP-based visual recognition by preserving two-dimensional positional information while encoding spatial features separately along height and width. Using ImageNet-1k alone, it improves classification performance across model scales.

1 INTRODUCTION: MLP-like networks can be efficient, but their ImageNet classification performance still lags without large-scale training data.The passage contrasts training efficiency with dependence on datasets such as ImageNet-22K and JFT-300M.
1 INTRODUCTION: Vision Permutator preserves the original spatial dimensions and separately encodes features along height and width instead of flattening them together.This design addresses positional-information loss in models operating on flattened tokens×channels representations.
1 INTRODUCTION: The architecture tokenizes image patches, processes height×width×channels embeddings through Permutators, and uses global average pooling before classification.Its Permutator blocks perform spatial and channel feature encoding.
1 INTRODUCTION: 81.5% top-1 accuracy is achieved on ImageNet without extra training data, while 55M and 88M models reach 82.7% and 83.2%, respectively.These results are reported for the small-sized model and larger Vision Permutator variants.

2 RELATED WORK

Prior visual-recognition models use convolution, self-attention, or flattened MLP operations to encode spatial information. Vision Permutator instead operates on three-dimensional features and encodes height and width separately.

2 RELATED WORK: Image-classification networks are broadly grouped into CNNs, vision transformers, and MLP-based models.The related-work discussion uses these three categories to situate Vision Permutator.
2 RELATED WORK: CNNs primarily use spatial convolutions and pooling, while later variants add skip connections, bottlenecks, and parallel specialized-filter paths.These developments aim to support deeper networks and richer feature processing.
2 RELATED WORK: Vision transformers use self-attention to capture global information rather than relying on CNN-style local convolutions.The passage identifies self-attention as the defining spatial-encoding mechanism of ViTs.
2 RELATED WORK: Recent pure MLP-like models flatten height, width, and channel representations into a two-dimensional tokens×channels table.Vision Permutator differs by retaining three-dimensional features and encoding height and width separately.

3 VISION PERMUTATOR

Vision Permutator combines patch embeddings with Permutator blocks that separately process height, width, and channel information. Its variants differ in model scale, patch size, stages, and fine-level token encoding, while Weighted Permute-MLP recalibrates branch contributions.

3 VISION PERMUTATOR: The network splits 224×224 images into 14×14 or 7×7 patches, maps them to tokens, applies Permutators, averages spatially, and predicts classes with a fully connected layer.This describes the end-to-end Vision Permutator pipeline.
3.1 PERMUTATOR: Each Permutator block contains Permute-MLP for spatial encoding and Channel-MLP for channel mixing, alongside normalization and skip connections.Channel-MLP uses two fully connected layers with GELU between them.
3.1 PERMUTATOR: Permute-MLP accepts H×W×C representations and uses three branches to encode height, width, and channel information before fusion.The branch outputs are combined and passed through a fully connected layer.
3.1 PERMUTATOR: The height and width branches split channels into segments, permute spatial and channel dimensions, and apply dimension-specific linear projections.The implementation uses H, W, C, and S to organize the branch computations.
3.1 PERMUTATOR: Weighted Permute-MLP replaces simple element-wise branch addition with branch-importance recalibration using split attention.The weighted version is used by default in the Permutator.
3.2 VARIOUS CONFIGURATIONS OF VISION PERMUTATOR: Vision Permutator configurations vary by Small, Medium, and Large model size, patch size, number of stages, and added fine-level token encoding.The /7 variants use two stages and additional Permutators for fine-level representations.

4 EXPERIMENTS

Experiments evaluate Vision Permutator on ImageNet-1k without extra training data, comparing it with MLP-like, CNN, and transformer models and analyzing design choices through ablations. Results show strong accuracy under similar model-size constraints, with benefits from height-and-width encoding, fine-level tokens, scaling, and augmentation.

Main Results on ImageNet: 81.5% top-1 accuracy is achieved by ViP-Small/7 with 25M parameters on ImageNet without external data.This result is reported as better than most existing MLP-like models and comparable to gMLP-B with 73M parameters.
Main Results on ImageNet: 81.5% accuracy for ViP-Small/7 exceeds ResNeSt-50’s 81.1% and is reported as better than several comparable CNN- and transformer-based models.The paper also states that a gap remains relative to recent state-of-the-art models such as NFNet, LV-ViT, and CaiT.
Ablation Analysis: 80.5% top-1 accuracy for ViP-Small/14 exceeds ViP-Small/16’s 79.8% when using a smaller initial patch size and more input tokens.The table caption states that changing initial patch size does not greatly change speed efficiency.
Ablation Analysis: 81.5% top-1 accuracy for ViP-Small/7 improves on ViP-Small/14’s 80.5% while reducing learnable parameters, although efficiency decreases slightly.ViP-Small/7 adds four Permutators for fine-level representations with 32 × 32 tokens.
Ablation Analysis: Increasing model size through more layers and larger hidden dimensions consistently improves Vision Permutator performance, while all four tested augmentations contribute to performance.The ablations cover model scaling and Random Augmentation, CutOut, MixUp, and CutMix.

CONCLUSIONS AND FUTURE WORK

Vision Permutator separately encodes height and width information, improving performance over recent MLP-like models. However, its fixed spatial parameter shapes prevent arbitrary-sized inputs and complicate downstream applications.

Separately encoding height and width information largely improves performance compared to MLP-like models that combine both spatial dimensions.
Fixed fully connected parameter shapes make Vision Permutator unable to process input images with arbitrary shapes.This scaling problem also exists in other MLP-like models.
Future work targets arbitrary-shape image processing and downstream tasks including object detection and semantic segmentation.The authors also plan to continue developing MLP-like models because of their high parallelization efficacy.