PaperlayerView exact revision on arXiv

Source-linked AI summary

Are we ready for a new paradigm shift? A Survey on Visual Deep MLP

Ruiyang Liu, Yinghui Li, Linmi Tao, Dun Liang, Hai-Tao Zheng

arXiv:2111.04060v6cs.CV

TL;DR

The paper asks whether a pure MLP stack can become a new computer-vision paradigm, and surveys its connections to convolution and self-attention alongside MLP-like variants. It concludes that current data and computational capability are insufficient for pure vision MLPs to learn effectively, while noting that human intervention remains important.

  • Problem

    The paper asks whether a pure MLP stack can become a new paradigm for computer vision, prompted by recent interest in deep MLP models.

    Source: §1 · p. 2, §3.1 · p. 9

  • Method

    The survey examines Token-mixing MLP's relation to depthwise convolution and discusses block redesign prompted by its bottleneck.

    Source: §1 · p. 2, §4.3 · p. 24

  • Results

    The authors conclude that current data and computational capability are still insufficient for pure vision MLP models to learn effectively, and human intervention remains important.

    Source: §7 · p. 35

  • Takeaways & Limitations

    The receptive-field analysis suggests that the purported new paradigm is instead moving toward the old paradigm.

    Source: §4.3 · p. 24

    Token-mixing MLP faces high computational complexity and resolution sensitivity.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recently, the proposed deep MLP models have stirred up a lot of interest in the vision community. Historically, the availability of larger datasets combined with increased computing capacity leads to paradigm shifts. This review paper provides detailed discussions on whether MLP can be a new paradigm for computer vision. We compare the intrinsic connections and differences between convolution, self-attention mechanism, and Token-mixing MLP in detail. Advantages and limitations of Token-mixing MLP are provided, followed by careful analysis of recent MLP-like variants, from module design to network architecture, and their applications. In the GPU era, the locally and globally weighted summations are the current mainstreams, represented by the convolution and self-attention mechanism, as well as MLP. We suggest the further development of paradigm to be considered alongside the next-generation computing devices.

1. Introduction

The survey asks whether deep MLPs can become a new computer-vision paradigm and compares Token-mixing MLP with convolution and self-attention. It concludes that current MLP-like variants still lag those approaches, while suggesting future paradigm shifts should be considered alongside new computing hardware.

1. Introduction

  • The survey compares convolution, self-attention, and Token-mixing MLP, reviewing MLP-Mixer in detail.It frames this comparison around whether MLP can become a new paradigm for computer vision.Source: §1 · p. 2
  • Token-mixing MLP is an enlarged depthwise-convolution variant with weights shared between channels, but it faces high computational complexity and resolution sensitivity.The paper identifies these as challenges for the design.Source: §1 · p. 2
  • The analyzed MLP-like variants are moving toward CNN designs, yet their visual-task performance still lags CNN- and Transformer-based methods.The survey concludes that MLP is not currently a new paradigm that can push computer vision to new heights.Source: §1 · p. 2
  • The survey suggests considering future paradigm development alongside the next generation of computing devices.It situates this suggestion in a history where datasets and computing capacity accompany paradigm shifts.Source: §1 · p. 2

2. Preliminary

This preliminary section introduces MLP, CNN, and Transformer and their historical paradigms to support the later discussion of visual deep MLPs.

2. Preliminary

  • The section briefly introduces MLP, CNN, and Transformer, together with their histories and corresponding paradigms.These preliminaries provide context for the later discussion of visual deep MLPs.Source: §2 · p. 3

2.1. Multilayer Perceptron and Boltzmann Machine

  • MLPs use fully connected layers, but their neuron count depends on input size and their parameter cardinality and computational cost grow quadratically with image resolution.The section also notes that stacked restricted Boltzmann machines are more computationally expensive than MLPs, especially at inference.Source: §2.1 · p. 3

2.2. Convolution Neural Network

  • CNNs use convolutional and pooling layers, with shared sliding kernels making convolution more parameter-efficient than fully connected layers.The section describes CNN inductive biases including local connectivity and translation invariance as helpful for image feature extraction.Source: §2.2 · p. 4

2.3. Vision Transformer

  • Transformers use self-attention to weight value vectors across the entire space, enabling long-distance feature capture and global information.The section reports Transformer-based architectures outperforming CNNs on tasks including object detection and segmentation.Source: §2.3 · p. 6

3. Pioneering Model and New Paradigm

This section reviews MLP-Mixer and contemporaneous MLP models, examining Token-mixing MLPs in relation to convolution and self-attention and their potential as a new paradigm. It also discusses their computation across shallow and deeper layers and identifies overfitting, quadratic image-size complexity, and fixed-resolution inputs as bottlenecks.

3.1. Structure of Pioneering Model

  • MLP-Mixer [15] embeds image patches, then stacks Mixer layers that apply shared Token-mixing and Channel-mixing MLPs before average-pooling features for classification.Token mixing operates across patches within each channel; channel mixing operates across channels within each patch feature.Source: §3.1 · pp. 7–8
  • FeedForward [61] reverses MLP-Mixer’s mixing order, while ResMLP uses a single-layer Token-mixing MLP and affine transformations instead of layer normalization.ResMLP’s affine transformation is described as stabilizing training.Source: §3.1 · p. 9
  • Pioneering MLP models achieve performance comparable to CNNs and ViT for image classification.These results challenge assumptions about the necessity of convolutional and self-attention layers.Source: §3.1 · p. 9

3.2. Token-mixing MLP, a New Paradigm?

  • Token-mixing MLP provides global spatial aggregation with fixed, input-independent weights, unlike local convolution and input-dependent self-attention.It aggregates spatial information from a single channel and shares weights across channels.Source: §3.2 · pp. 9–10
  • Shallow fully connected layers in MLP-Mixer and ResMLP show local connectivity and convolution-like computation; deeper networks expand the effective receptive field and their weights become disorganized.The authors note that whether disorganization reflects insufficient training data or is desirable remains unclear.Source: §3.2 · pp. 11–12
  • Despite similar structures apart from normalization, MLP-Mixer and ResMLP learn vastly different weights, raising questions about generic visual features and interpretability.The paper states that MLP interpretability remains far behind.Source: §3.2 · p. 12

3.3. Bottlenecks

  • Without local-connectivity and self-attention inductive biases, Token-mixing MLPs are more flexible but risk overfitting, so large-scale data is needed to narrow classification-accuracy gaps with ViT and CNN.Source: §3.3 · p. 12
  • Token-mixing MLP complexity is quadratic in image size, making existing MLP-like models intractable at high resolution with current computing capability.Source: §3.3 · p. 12
  • Token-mixing MLPs are resolution-sensitive because fixed fully connected layer sizes prevent flexible inputs, making them non-transferable and impractical across differing training and evaluation resolutions.Source: §3.3 · p. 12
  • MLP-like variants modify Token-mixing MLPs to reduce computation, remove resolution sensitivity, or restore local receptive fields.Source: §3.3 · p. 12

4. Block of MLP Variants

Recent MLP-like variants respond to MLP-Mixer challenges chiefly through block redesign; the section compares their designs, properties, and receptive fields.

4. Block of MLP Variants

  • Figure 4 distinguishes resolution-sensitive RedOrange Token-mixing modules from resolution-insensitive Green modules.The figure’s lower part illustrates block designs, and variants are ordered by release month from left to right.Source: §4 · p. 13, Figure 4 · p. 13
  • Most variants retain tandem spatial- and channel-mixing MLPs, while gMLP is the exception.Their block redesigns primarily target Token-mixing MLP; most also reduce sensitivity to image resolution.Source: §4 · p. 13
  • The section examines redesigned blocks, compares their properties and receptive fields, and discusses the resulting findings.The authors report reproducing most MLP-like variants in Jittor and PyTorch.Source: §4 · p. 13

4.1. MLP Block Variants

This section surveys MLP-block variants that mix axial, channel, and spatial dimensions or incorporate other operators, describing their designs and mechanisms. It covers trade-offs and limitations involving spatial reach, receptive fields, resolution sensitivity, and computational complexity.

4.1.1. Axial and channel projection blocks

  • WaveMLP [76] treats tokens as waves with amplitude and phase, dynamically aggregating them along horizontal and vertical paths, while local windows limit long-range dependencies.Its local-window strategy is intended to address resolution sensitivity.Source: §4.1.1 · p. 15
  • MorphMLP [77] shifts from local detail modeling in early stages to long-range modeling later, increasing window size with depth; its authors report benefits for images and videos.It uses static aggregation, with local windows also addressing image-resolution sensitivity.Source: §4.1.1 · p. 16
  • ViP [71], Sparse MLP [73], RaftMLP [74], and DynaMixer [75] mix along spatial axes, reducing complexity while retaining some long-range dependence, but remain resolution-sensitive.WaveMLP [76] and MorphMLP [77] use local windows to address resolution sensitivity, but this sacrifices long-range dependencies.Source: §4.1.1 · p. 16

4.1.2. Channel-only projection blocks

  • AS-MLP [80] uses parallel horizontal and vertical shifts with channel projection, while CycleMLP uses deformable convolution for spatial integration and three branches fused by split attention.AS-MLP has two branches and CycleMLP three; AS-MLP also supports shifting strategies that yield a receptive field similar to dilated convolution.Source: §4.1.2 · pp. 16–18
  • ActiveMLP [85] dynamically estimates spatial offsets to find and fuse helpful contextual features, unlike the manually set offsets of AS-MLP and CycleMLP.The operation is described as equivalent to a 1 × 1 deformable convolution.Source: §4.1.2 · p. 18
  • HireMLP [86] communicates spatial information through inner-region and cross-region feature rearrangements before channel projection.Inner-region rearrangement expands the feature map along height or width, while cross-region rearrangement moves it cyclically.Source: §4.1.2 · p. 18
  • After shifting, channel projection aggregates features from different locations across channels, which the survey characterizes as an artificially designed deformable convolution limited to local extraction.The six models discussed before MS-MLP communicate only localized information through feature-map movement.Source: §4.1.2 · p. 18
  • MS-MLP [87] mixes neighboring and distant tokens across fine-to-coarse levels using shifting and depthwise convolutions of different sizes before channel alignment.The survey notes that its receptive field still has some vacancy.Source: §4.1.2 · p. 18

4.1.3. Spatial and channel projection blocks

  • gMLP [67] uses linear spatial projection with multiplicative gating, splitting channels into two parts so one gates the other, with channel projection around the gate.Its spatial gating output is an element-wise product; the block has no channel-mixing MLP within the gating unit.Source: §4.1.3 · p. 19
  • gMLP achieves good performance in both computer vision and natural language processing tasks.The passage supplies no benchmark values or comparisons.Source: §4.1.3 · p. 19
  • Lou et al. [88] extend Mixture-of-Experts to Token-mixing MLP in Sparse-MLP(MoE), reporting performance gains over the primitive MLP-Mixer.Carlos et al. [90] had applied MoE to Channel-mixing MLP in the Transformer block earlier.Source: §4.1.3 · p. 19
  • Yu et al. [91] constrain Token-mixing MLP weights to a circulant structure, reducing translation sensitivity while retaining global receptive fields, but the model remains resolution-sensitive.The constraint reduces parameters from O(H^2W^2) to O(HW), not computation; FFT is used to reduce FLOPs.Source: §4.1.3 · p. 19
  • RepMLPNet [92] learns parallel fully connected and convolutional layers during training, then merges them by structural re-parameterization; the resulting weights suggest stronger neighborhood focus.Its resolution sensitivity prevents it from serving as a general vision backbone.Source: §4.1.3 · p. 19

4.1.4. Others

  • ConvMLP [93] replaces Token-mixing MLP with a 3×3 depth-wise convolution and is treated as a pure CNN, while LIT [94] replaces early-stage self-attention with MLP.UniNet [95] searches combinations of convolution, self-attention, and MLP; the survey says these methods are beyond Vision MLP's scope.Source: §4.1.4 · p. 20

4.2. Receptive Field and Complexity Analysis

The section compares receptive-field patterns and computational complexity across MLP-like blocks, showing trade-offs between global interaction, resolution sensitivity, and lower-cost designs. It argues that local receptive fields resemble convolution, while restricted interactions and image resolution impose important constraints.

4.2.1. Receptive field

  • MLP-Mixer, gMLP, Sparse-MLP(MoE), CCS, and RepMLPNet retain global receptive fields, but that globality is defined over patches rather than pixels.The authors also classify receptive fields as global, cruciform, or local.Source: §4.2.1 · pp. 20–21
  • Local receptive fields in MLP-like variants can be realized with convolution, leaving them without an essential difference from CNNs.MS-MLP’s depth-wise convolution before shifting makes its receptive field most like a local window with spatial projection.Source: §4.2.1 · p. 21
  • Axial projection forms a cruciform receptive field by serially or parallelly decomposing full spatial projection along horizontal and vertical axes.Tokens at positions sharing neither the same row nor column cannot interact in a single projection.Source: §4.2.1 · p. 21
  • Global and cruciform projections require coverage of the full image dimensions, tying fully connected layer neurons to resolution and restricting image sizes.Local receptive fields are used to eliminate this resolution sensitivity, but abandon the global receptive field.Source: §4.2.1 · p. 22

4.2.2. Complexity analysis

  • Full spatial projection uses O(H^2W^2) parameters and O(H^2W^2C) FLOPs, both quadratic with image resolution.The authors say this makes large-resolution images difficult to handle with current computational power.Source: §4.2.2 · p. 22
  • Axial projection reduces spatial parameters from O(H^2W^2) to O(H^2 + W^2) and FLOPs from O(H^2W^2C) to O(HWC(H + W)).Three parallel branches have O(H^2 + W^2 + C^2) parameters and O(HWC(H + W + C)) FLOPs; a local window further lowers these to O(2L^2 + C^2) and O(HWC(2L + C)).Source: §4.2.2 · p. 24
  • Feature-map shifting with channel projection reduces computational complexity to O(C^2) parameters and O(HWC^2) FLOPs, while its weights are decoupled from image resolution.Source: §4.2.2 · p. 24
  • Lower-complexity networks can use more layers and channels when parameter counts remain comparable.DynaMixer’s dynamic estimation of the weight matrix leads to higher complexity.Source: §4.2.2 · p. 24

4.3. Discussion

The discussion describes MLP-like variants as responses to limits in current data and computational power, using designs that reduce complexity and address resolution sensitivity. Its receptive-field analysis suggests the new paradigm is moving toward the old one, while balancing long-distance dependence and resolution sensitivity remains a challenge.

4.3. Discussion

  • Recent variants reduce complexity and address resolution sensitivity through orthogonal decomposition, local windows, shifted feature maps, and other designed operations.The authors say current data and computational power remain insufficient for pure MLPs.Source: §4.3 · p. 24
  • Computational complexity has theoretical significance but is not the only determinant of inference time or final model efficiency.Source: §4.3 · p. 24
  • Receptive-field analysis suggests the new paradigm is moving toward the old paradigm, and the authors bluntly describe MLP development as heading back toward CNNs.The discussion calls for continued effort to balance long-distance dependence with image-resolution sensitivity.Source: §4.3 · pp. 24–25

5. Architecture of MLP Variants

MLP-like models use single-stage, two-stage, and pyramid stacking patterns, with architectures evolving toward pyramids as patch sizes shrink. Finer patches can improve recognition, but their computational cost and the fairness of architecture comparisons remain concerns.

5. Architecture of MLP Variants

  • Classification MLPs use single-stage, two-stage, and CNN-like pyramid structures, with multiple blocks stacked within each stage.Single-stage blocks keep feature-map size unchanged; pyramid structures reduce resolution across stages.Source: §5 · pp. 25–26, §5.1 · p. 26

5.1. From Single-stage to Pyramid

  • Coarse patch partitions constrain feature fineness and affect downstream detection and instance segmentation, especially for small targets.In pyramid structures, overlapping patch embedding—7 × 7 convolution with stride 4 instead of 4 × 4—provides better results.Source: §5.1 · pp. 26–27
  • Two-stage models merge patches after early layers, while most recently proposed MLP-like variants adopt pyramid structures.Vision Permutator begins with 7 × 7 patch slices and halves feature-map height and width while doubling channels at merging.Source: §5.1 · pp. 26–27
  • ImageNet-1k Top-1 accuracy rose from 80.6% to 81.5% with finer patch representations, and from 80.9% to 82.0% for S2MLPv2.DynaMixer also adopts a two-stage configuration.Source: §5.1 · pp. 26–27

5.2. Discussion

  • As patch size decreases, token counts increase; to control computation, interactions are limited to a small range, so preserving a global receptive field requires reducing feature-map size, yielding a pyramid structure.Whether a single-stage model would be worse than the pyramid model is currently unknown; its calculation cost is known to increase significantly.Source: §5.2 · p. 27

6. Applications of MLP Variants

This section surveys MLP-like variants across classification, detection and segmentation, low-level vision, video, and point-cloud tasks, including their performance and framework limitations in high-level vision. It also discusses block redesign, MLP-specific frameworks and training strategies, and point-cloud analysis with unordered visual data.

6. Applications of MLP Variants

  • Because MLP research is recent, the review emphasizes classification and detection or segmentation, giving only brief introductions to low-level vision, video, and point-cloud applications.The authors attribute the limited treatment of those latter areas to the small number of available works.Source: §6 · p. 28

6.1. Image Classification

  • Without extra training data, CNNs and Transformers exceed 87% Top-1 accuracy while MLP-like variants reach 84.3%, and extra data widens the gap.The review notes that architecture-specific training strategies may help, but no MLP-specific training mode is yet established.Source: §6.1 · p. 28
  • MLP-like models can be competitive with CNNs and Transformers under the same training strategy and data volume, while data and training strategies may yield larger gains than module redesign.The comparison is based on ImageNet-1K results without extra data and on broader representative model comparisons.Source: §6.1 · p. 28, Table 4 · p. 29, Table 5 · p. 30

6.2. Object Detection and Semantic Segmentation

  • MLP variants are used as backbones in traditional CNN-based frameworks for object detection and semantic segmentation, where their empirical performance remains weaker than that of the most advanced CNN approaches.Source: §6.2 · p. 31

6.3. Low-level Vision

  • MixerGAN applies MLP-Mixer to unpaired image translation and succeeds on generative objectives, while MAXIM uses spatially gated MLPs and achieves state-of-the-art results across several image-processing tasks.MAXIM's reported tasks include dehazing, deblurring, denoising, deraining, and enhancement.Source: §6.3 · pp. 32–33

6.4. Video Analysis

  • MorphMLP [77] achieves competitive performance with recent state-of-the-art methods on the Kinetics-400 dataset, demonstrating that an MLP-like backbone is also suitable for video recognition.Source: §6.4 · p. 33

6.5. Point Cloud

  • PointMLP [133] uses geometric affine normalization and residual MLPs for local points to achieve new state-of-the-art results, and MLPs and Transformers are appealing for unordered point-cloud data.Point clouds are sparse, unordered, and irregular; MLPs are permutation-invariant, matching this data characteristic.Source: §6.5 · p. 34

6.6. Discussion

  • The review identifies applications spanning classification, generation, detection, segmentation, and video analysis.These are among the diverse tasks listed in the discussion.Source: §6.6 · p. 34
  • Performance is linked to artificial redesign of the MLP block, while MLP-specific frameworks and training strategies may improve it further.Source: §6.6 · p. 34
  • Pure MLPs have demonstrated advantages in point-cloud analysis, encouraging their application to visual tasks with unordered data.Source: §6.6 · p. 34

7. Summary and Outlook

The review describes how deep MLPs approximate full connectivity through patch-level projections and surveys trade-offs in module and architecture design. It concludes that current data and computational capability are insufficient for pure vision MLPs to learn effectively, while outlining further research directions.

7. Summary and Outlook

  • Recent deep MLPs partition images into patches and alternate fully connected projections across space and channels, approximating full connectivity and a global receptive field at patch level.The projections decompose traditional fully connected projections, with spatial projection further split into horizontal and vertical directions.Source: §7 · p. 35
  • Module variants either reduce computational complexity at the cost of full spatial connectivity or address resolution sensitivity while eroding global receptive fields.The latter route adopts CNN-like improvements and can transfer pretrained models to downstream tasks.Source: §7 · p. 35
  • Pyramid stacking appears preferable for finer features, but its comparison with single-stage models is unfair because their initial patch sizes differ.Whether a 4 × 4 single-stage model performs worse than a pyramid remains unknown, and its calculation cost would increase significantly.Source: §7 · p. 35
  • Current data and computational capability are not enough for pure vision MLPs to learn effectively, and human intervention remains important.The authors use this conclusion to motivate future research directions.Source: §7 · p. 35
  • The review calls for vision-tailored designs that combine local and global receptive fields while maintaining resolution insensitivity.It notes that current MLP-like variants tend to make an either/or choice between short- and long-range dependencies.Source: §7 · p. 36
  • MLP-specific scaling and efficient designs remain open needs because added parameters have yielded limited gains and computation can hinder edge deployment.The review points to mobile platforms and resource-constrained devices as settings where efficient designs are lacking.Source: §7 · p. 36
  • Dedicated MLP pretraining and optimization remain open questions, including whether existing self-supervised methods and optimizers are suitable.The review asks whether methods effective for CNNs or Transformers transfer to MLPs and what optimizer is best for MLPs.Source: §7 · pp. 36–37
  • The review also identifies interpretability and paradigms beyond weighted sums as future directions as data and computing power improve.It links interpretability to analyzing learned filters and feature maps, and names Boltzmann machines as an example of another paradigm.Source: §7 · p. 37

Data and Code Availability

The authors reproduced most variants of MLP-like models in Jittor and PyTorch; code is available on GitHub, and the review generated no new data.

Data and Code Availability

  • The authors reproduced most MLP-like model variants in Jittor and PyTorch, made code available on GitHub, and generated no new data.The code repository is https://github.com/liuruiyang98/Jittor-MLP.Source: Data and Code Availability · p. 38
Loading 2111.04060v6…