PaperlayerView exact revision on arXiv

Source-linked AI summary

CycleMLP: A MLP-like Architecture for Dense Prediction

Shoufa Chen, Enze Xie, Chongjian Ge, Runjian Chen, Ding Liang, Ping Luo

arXiv:2107.10224v4cs.CV

TL;DR

Existing MLP-like vision models are limited by fixed input scales and quadratic costs, restricting their use for dense prediction. The paper introduces CycleMLP, a hierarchical architecture built around Cycle FC, which combines flexible resolutions with linear complexity. CycleMLP outperforms existing MLP-like models and achieves comparable or sometimes better performance than CNNs and Transformers across recognition and dense prediction tasks.

  • Problem

    Prior MLP-like models require fixed image scales and have quadratic computational and memory costs, limiting their suitability for dense prediction.

  • Method

    CycleMLP builds hierarchical vision architectures using Cycle FC, which enlarges the receptive field while preserving flexible input resolutions and linear computational complexity.

  • Results

    CycleMLP outperforms existing MLP-like models and is comparable to, or sometimes better than, CNNs and Transformers across recognition and dense prediction tasks.

  • Takeaways & Limitations

    The results indicate that an attention-free architecture can serve as a general vision backbone for recognition and dense prediction.

  • Takeaways & Limitations

    Weight interpolation may hurt performance when input scale varies.

Abstract

from arXiv · show

This paper presents a simple MLP-like architecture, CycleMLP, which is a versatile backbone for visual recognition and dense predictions. As compared to modern MLP architectures, e.g., MLP-Mixer, ResMLP, and gMLP, whose architectures are correlated to image size and thus are infeasible in object detection and segmentation, CycleMLP has two advantages compared to modern approaches. (1) It can cope with various image sizes. (2) It achieves linear computational complexity to image size by using local windows. In contrast, previous MLPs have $O(N^2)$ computations due to fully spatial connections. We build a family of models which surpass existing MLPs and even state-of-the-art Transformer-based models, e.g., Swin Transformer, while using fewer parameters and FLOPs. We expand the MLP-like models' applicability, making them a versatile backbone for dense prediction tasks. CycleMLP achieves competitive results on object detection, instance segmentation, and semantic segmentation. In particular, CycleMLP-Tiny outperforms Swin-Tiny by 1.3% mIoU on ADE20K dataset with fewer FLOPs. Moreover, CycleMLP also shows excellent zero-shot robustness on ImageNet-C dataset. Code is available at https://github.com/ShoufaChen/CycleMLP.

1 INTRODUCTION

MLP-based vision models face fixed-resolution requirements and quadratic costs that conflict with dense prediction. CycleMLP addresses these issues with a hierarchical architecture and Cycle FC, enabling flexible resolutions, linear complexity, and broad task coverage.

  • Motivation: Dense prediction requires multi-scale training and varying resolutions, but prior MLP structures typically require fixed image scales.These models also incur computational and memory costs quadratic in image size.
  • Method: CycleMLP uses a hierarchical architecture to generate pyramid features for recognition and dense prediction tasks.The architecture is designed to address variable input scales and dense-prediction requirements.
  • Method: Cycle FC combines Channel FC’s flexible-resolution, linear-complexity behavior with a larger receptive field for spatial context.It samples points cyclically along the channel dimension while retaining Channel FC’s parameter and FLOPs complexity.
  • Results: Extensive experiments show CycleMLP is comparable to, and sometimes better than, CNNs and Transformers on dense prediction tasks.The evaluated tasks include COCO detection and segmentation and ADE20K semantic segmentation.
  • Results: CycleMLP outperforms existing MLP-like models across a broad ImageNet accuracy–FLOPs range.Figure 2 compares models trained on ImageNet-1K without extra data.

2 METHOD

CycleMLP builds a hierarchical MLP-like visual backbone around Cycle FC, which aggregates spatial context across flexible image resolutions while retaining linear computational cost. Its architecture combines multi-scale Cycle FC blocks, channel MLPs, residual connections, and staged feature pyramids for recognition and dense prediction.

  • Cycle FC: Cycle FC replaces Spatial FC for token mixing, sampling channels at cyclical spatial offsets to enlarge the receptive field.The operator uses offsets along height and width while preserving the channel-projection structure.
  • Cycle FC: Cycle FC maintains Channel FC’s linear parameters and FLOPs with respect to spatial scale while supporting flexible input resolutions.Channel FC is resolution-agnostic but has limited spatial context; Cycle FC expands that context without changing the stated complexity.
  • Cycle FC and MHSA: Cycle FC imposes a sparse spatial-weighting inductive bias relative to MHSA, selecting weights associated with spatial shifts while retaining an enlarged receptive field.The paper relates the selected Cycle FC weights to reshaped MHSA weights and contrasts this sparse design with convolution and self-attention efficiency.
  • CycleMLP Block: Each CycleMLP block uses three parallel Cycle FCs with stepsizes 1 × 7, 7 × 1, and 1 × 1, followed by a channel-MLP, LayerNorm, and residual connections.The parallel design is motivated by convolution factorization and criss-cross attention.
  • Overall Architecture: CycleMLP forms four hierarchical stages that reduce token count and increase channel capacity at transitions, producing pyramid features for dense prediction.The final-stage feature shape is H/32 × W/32 × C4, and the staged structure is intended to serve as a general-purpose visual backbone.

3 EXPERIMENTS

Experiments show that CycleMLP provides a resolution-flexible, computationally efficient backbone with competitive or superior performance across classification, detection, segmentation, and robustness evaluations.

  • ImageNet-1K Classification: CycleMLP consistently outperforms existing MLP-like models in the accuracy-FLOPs tradeoff across a wide range of computational budgets.CycleMLP-B2 matches gMLP-B at 81.6% top-1 accuracy while using 3.9G versus 15.8G FLOPs.
  • ImageNet-1K Classification: 81.6% top-1 accuracy is achieved by CycleMLP-B2, matching gMLP-B while reducing FLOPs by more than 3×.CycleMLP-B2 uses 3.9G FLOPs, compared with 15.8G for gMLP-B.
  • Dense Prediction: CycleMLP enables dense-prediction experiments that previous MLP-like models generally avoid because they cannot handle variable input scales.The proposed Cycle FC addresses this issue while retaining linear computational complexity in image size.
  • ImageNet-1K Classification: 83.4% top-1 accuracy lets CycleMLP-B slightly exceed Swin-B’s 83.3% with similar parameters and FLOPs.GFNet has similar ImageNet classification performance, but its resolution-correlated architecture requires parameter interpolation when input scale changes.
  • Ablation Study: Removing any of CycleMLP’s three parallel branches reduces top-1 accuracy, especially when removing the 1×7 or 7×1 branch.The drop persists after matching parameter and FLOP counts with alternative branch configurations.
  • Object Detection and Instance Segmentation: CycleMLP-based RetinaNet surpasses ResNet, ResNeXt, and PVT under similar parameter constraints, while Mask R-CNN shows similar comparisons and slightly better performance than Swin.These results support CycleMLP as a general-purpose backbone for object detection and instance segmentation.
  • Semantic Segmentation: 0.9 mIoU is the advantage of CycleMLP-B2 over Swin-T on ADE20K, using slightly fewer parameters: 30.6M versus 31.9M.CycleMLP also significantly outperforms ResNet and PVT with similar parameter counts.
  • Robustness and Adaptability: CycleMLP has a larger effective receptive field than Swin and stronger robustness than Transformer and existing MLP models on ImageNet-C.Resolution-adaptability evaluations use models trained at 224×224 and tested at varied resolutions without fine-tuning.

4 CONCLUSION

CycleMLP is presented as a versatile attention-free vision backbone built on Cycle FC. Experiments support its use for image classification and dense prediction tasks.

  • Conclusion: CycleMLP is a versatile MLP-like architecture built upon Cycle FC, which supports variable input scales and can replace vanilla fully connected layers.The paper frames CycleMLP as a general vision backbone for recognition and dense prediction.
  • Conclusion: CycleMLP outperforms existing MLP-like models on ImageNet classification and achieves promising results on object detection, instance segmentation, and semantic segmentation.The reported dense-prediction tasks span detection and both instance- and semantic-level segmentation.
  • Conclusion: The results indicate that an attention-free architecture can serve as a general vision backbone.This conclusion is supported across classification and multiple dense-prediction tasks.

B COMPARISON OF MLP BLOCKS

Existing MLP blocks use spatial fully connected projections tied to sequence length, while CycleMLP replaces this with Cycle FC to support dynamic image scales.

  • MLP-Mixer, ResMLP, and gMLP transpose tokens, apply a spatial fully connected layer, then transpose back.
  • These spatial projections cannot handle variable image scales because their fully connected layers are configured by sequence length.
  • Cycle FC has a configuration independent of image scale and naturally handles dynamic input scales.
  • Multi-head self-attention reshapes features into T = H × W tokens and forms an attention matrix A ∈ R^{T×T} whose entries score token pairs.
  • Relative positional encoding re-parameterizes attention scores using token-to-token relative distances and learnable bias vectors.
  • Setting the query and key projections to zero and the positional key projection to identity reduces the attention score to a positional term.

D ARCHITECTURE VARIANTS

CycleMLP provides two model zoos, one following PVT and one following Swin, scaled through stage-wise architectural hyperparameters.

  • The model zoos are PVT-Style CycleMLP-B1 to -B5 and Swin-Style CycleMLP-T, -S, and -B.
  • Models are scaled by varying each stage’s stride, token channel dimension, block count, and expansion ratio.

E.1 IMAGENET CLASSIFICATION

The paper evaluates CycleMLP on ImageNet classification and also describes varying-complexity instantiations, while noting that further kernel optimization remains outside scope.

  • ImageNet-1K contains 1.2M training images and 50K validation images across 1,000 categories, with top-1 validation accuracy reported.
  • CycleMLP variants differ in expand ratio and number of repeated layers, following a design that increases channels while shrinking spatial resolution with depth.
  • Further kernel optimization for Cycle FC may improve speed but is beyond the scope of this work.
  • COCO detection and instance-segmentation experiments use RetinaNet and Mask R-CNN with ImageNet-pretrained backbones.

E.3 ADE20K SEMANTIC SEGMENTATION

CycleMLP is evaluated for semantic segmentation on ADE20K using a PVT-following training setup and validation-set backbone comparisons.

  • ADE20K covers 150 semantic categories with 20K training, 2K validation, and 3K testing images.
  • The experiments use the mmsegmentation codebase and train for 40K iterations on 8 Tesla V100 GPUs with 4 samples per GPU.
  • Table 12 reports semantic segmentation results for different backbones on the ADE20K validation set.

F SAMPLING STRATEGIES

CycleMLP’s cyclical sampling preserves recognition-relevant local and fine-grained information better than random and dilated alternatives, while dense sampling increases computational cost. Stepsize choices therefore balance receptive-field coverage against representation quality.

  • −1.3% Top-1 accuracy on ImageNet-1K followed random sampling across three independent trials.The authors hypothesize that random sampling disturbs object semantics, whereas cyclical sampling aggregates adjacent pixels.
  • +1.0% acc over random sampling but −0.5% acc below cyclical sampling characterized dilated sampling.Dilated sampling retains local aggregation but loses fine-grained information relative to cyclical sampling.
  • Dense sampling substantially increases parameters and FLOPs, requiring 100-epoch comparisons under identical learning configurations.The authors limited training duration because dense sampling made the models more expensive to train.
  • Figure 6 illustrates dilated CycleMLP with dilation=2 and stepsize=3.
  • Table 15 compares different stepsizes, including even stepsize and configurations such as 7×2 and 4×4.

G VISUALIZATION EXAMPLES

CycleMLP visualizations show how stepsize configurations organize sampled points into cyclical channel groups. Changing stepsize enlarges spatial coverage but can reduce channel-wise representation capacity.

  • G VISUALIZATION EXAMPLES: Visualizations cover the general 3×3 case, even stepsizes, and cases where the height or width stepsize equals 1.
  • G VISUALIZATION EXAMPLES: Stepsize changes preserve parameter count for fixed input and output channels, creating a trade-off between spatial and channel representation.
  • G VISUALIZATION EXAMPLES: 1×7 and 7×1 achieve the best performance, especially for semantic segmentation on ADE20K, among the tested stepsizes.Larger stepsizes expand the spatial receptive field but reduce the number of channel groups.
  • G VISUALIZATION EXAMPLES: Figure 7 groups sampled points by periods and marks their absolute feature-map coordinates, with dashed lines linking cyclical periods.
Loading 2107.10224v4…