Source-linked AI summary
ResMLP: Feedforward networks for image classification with data-efficient training
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, Hervé Jégou
TL;DR
ResMLP addresses whether image representations can remain competitive with fewer hard-coded architectural priors than convolutional or attention-based models. It uses residual MLP layers with linear patch interaction and cross-channel feed-forward processing, and achieves strong ImageNet trade-offs while extending competitively to self-supervised learning and machine translation.
Problem
Vision architectures have progressively reduced hard-coded image priors, motivating whether a purely MLP-based model can remain competitive for image classification and beyond.
Method
ResMLP alternates residual cross-patch linear layers with cross-channel feed-forward MLPs, uses affine transformations instead of normalization, and adapts the design to self-supervised learning and sequence-to-sequence translation.
Results
ResMLP achieves unexpectedly high ImageNet performance and competitive WMT translation results; self-supervised pre-training reaches 79.9% top-1 accuracy on ImageNet-val (+0.5%).
Takeaways & Limitations
Strong performance from a simple MLP architecture suggests that severe structural constraints in patch communication need not drastically limit accuracy when paired with modern training schemes.
Takeaways & Limitations
ImageNet-1k validation comparisons may be inconclusive because the validation set was used for hyper-parameter selection.
Abstract
from arXiv · showhide
We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. It is a simple residual network that alternates (i) a linear layer in which image patches interact, independently and identically across channels, and (ii) a two-layer feed-forward network in which channels interact independently per patch. When trained with a modern training strategy using heavy data-augmentation and optionally distillation, it attains surprisingly good accuracy/complexity trade-offs on ImageNet. We also train ResMLP models in a self-supervised setup, to further remove priors from employing a labelled dataset. Finally, by adapting our model to machine translation we achieve surprisingly good results. We share pre-trained models and our code based on the Timm library.
1 Introduction
ResMLP extends the move toward fewer image-specific architectural priors by using only MLP-based residual operations, while remaining competitive on image classification and machine translation.
- Architecture: ResMLP processes image patches with a cross-patch linear layer followed by a cross-channel MLP, using residual connections and affine transformations.The cross-patch operation is applied independently across channels, while the cross-channel operation is applied independently per patch.
- Architectural simplifications: Replacing self-attention with linear layers yields a simpler architecture that uses no positional embeddings, class token, or batch-statistics-based normalization.The authors report more stable training than ViTs under the same training scheme and speculate that linear patch interactions contribute to this stability.
- Interpretability: Visualized patch interactions resemble convolutions in lower layers and become longer-range in later layers.The linear patch-interaction structure makes these learned filters directly visualizable.
- Machine translation: A sequence-to-sequence ResMLP adapted for variable-length inputs is competitive with Transformers on challenging WMT machine-translation benchmarks.The adaptation retains cross-attention between encoder and decoder while replacing Transformer layers with ResMLP layers.
- Image classification: ResMLP achieves surprisingly good accuracy/complexity trade-offs on ImageNet-1k without normalization based on batch or channel statistics.The models also benefit significantly from distillation and remain compatible with augmentation-based self-supervised learning.
2 Method
ResMLP replaces key Vision Transformer components with simpler linear and affine operations organized in residual layers, with alternative pooling and sequence-to-sequence adaptations extending the design.
- Overall architecture: ResMLP embeds non-overlapping image patches, repeatedly applies residual cross-patch and cross-channel transformations, then average-pools patches for classification.The pooled representation is passed to a linear classifier trained with cross-entropy loss.
- Residual Multi-Perceptron Layer: Each residual layer applies a linear sublayer across patches followed by a feed-forward sublayer across channels, with skip connections around both.The architecture replaces LayerNorm with a learnable affine transformation that does not use channel-wise statistics.
- Layer transformations: The cross-patch matrix A exchanges information between patches, while matrices B and C implement the per-location cross-channel feed-forward sublayer.The feed-forward structure matches a Transformer sublayer with GELU replacing ReLU; B and C have dimensions 4d×d and d×4d.
- Differences with ViT: Compared with ViT, ResMLP removes self-attention, positional embeddings, the class token, and normalization based on batch statistics.Patch positions are implicitly encoded by the linear layer, and average pooling replaces the class token.
- Class-MLP: Class-MLP replaces average pooling with linear interactions over aggregated patches, improving performance at the cost of additional parameters and computation.The patch embeddings remain frozen during these pooling layers.
- Sequence-to-sequence ResMLP: For sequence-to-sequence tasks, ResMLP replaces encoder and decoder self-attention layers while retaining decoder cross-attention to encoder outputs.The linear sublayers are adapted to language modeling by constraining their matrices.
3 Experiments
Experiments evaluate ResMLP across supervised, self-supervised, distillation, transfer, and architectural analyses, showing strong but regularization-sensitive performance and distinctive sparse cross-patch interactions.
- Main results: Experiments compare ResMLP with convolutional and Transformer architectures using ImageNet accuracy alongside throughput, FLOPs, parameters, and peak memory.The comparisons use 224×224 inputs and comparable model sizes or computational settings.
- Main results: ResMLP’s accuracy remains strong, although its accuracy–FLOPs–throughput trade-off is weaker than those of convolutional networks and Transformers.The result suggests the layer’s structural constraints do not drastically limit performance with sufficient data and modern training schemes.
- Self-supervised setting: 79.9% top-1 accuracy on ImageNet-val (+0.5%) is achieved after fine-tuning a self-supervised ResMLP, substantially improving over supervised-only training.DINO pre-training produces results competitive with ConvNets in k-NN evaluation but below ViT.
- Knowledge distillation setting: ResMLP greatly benefits from distillation from a ConvNet, while the experiments continue to observe overfitting in feedforward networks on ImageNet.The authors identify additional regularization from distillation as a possible explanation for the improvement.
- Visualization and linear interaction: Cross-patch layers show convolution-like early patterns and are sparser than per-patch MLP matrices.The analysis measures components below 5% of each matrix’s maximum absolute value and motivates pruning, quantization, and convolutional variants.
4 Related work
ResMLP builds on fully connected, convolutional, and transformer architectures while reducing image-specific architectural priors. Its key structural change is replacing transformer self-attention with a linear layer for patch interactions.
- Fully-connected networks: Fully connected networks have been studied as alternatives to convolutional networks for several vision tasks, including natural-image datasets.Prior work also examined infinite-width networks and inverse scattering problems.
- Fully-connected networks: Tensorizing Network also seeks to remove architectural priors through more general tensor operations, but it compresses MLP layers of standard convolutional networks.This distinguishes its purpose from ResMLP's architecture design.
- Related components: A fully connected layer is equivalent to a convolution with a 1 × 1 receptive field, while other convolutional architectures combine small receptive fields such as 1 × 1 and 3 × 3.Examples include VGG, ResNext, and Xception.
- Transformers: ResMLP takes non-overlapping patches as input and uses MLP layers with the ViT structure, replacing self-attention with a linear patch interaction layer.This preserves a patch-based organization while removing self-attention.
- Transformers: The paper questions the importance of self-attention, paralleling Synthesizer's replacement of dot-product attention with a feed-forward network in natural language processing.Synthesizer uses data-dependent weights determined from queries, unlike ResMLP.
5 Conclusion
ResMLP combines linear patch communication with channel-wise feed-forward residual blocks and achieves unexpectedly high ImageNet performance when paired with modern transformer-era training. Its learned interactions include both convolution-like filters and sparse long-range connections.
- Conclusion: ResMLP achieves unexpectedly high ImageNet classification performance with residual blocks combining a one-hidden-layer feed-forward network and a linear patch interaction layer.The result depends on adopting a modern training strategy introduced for transformer-based architectures.
- Conclusion: The architecture's linear layers make learned patch-communication filters directly visualizable, revealing both convolution-like structure and sparse long-range interactions.Sparse long-range interactions appear as early as the second layer.
A Report on our exploration phase
The exploration phase progressively replaced attention and normalization components with simpler MLP-based alternatives. A square linear patch-communication layer and affine transformation produced favorable accuracy–complexity behavior, while distillation improved the baseline.
- Exploration setup: The CaiT-S24 exploration model used 24 layers, working dimension 384, 224×224 images, and N = 16 × 16 patches, reaching 82.7% supervised top-1 accuracy.These settings defined the exploration experiments.
- SA → MLP: Replacing self-attention layers while retaining other CaiT elements reduced ImageNet top-1 accuracy to 80.2% (-2.5%).The replacement alternates residual blocks operating along channel and patch dimensions.
- Class-attention → class-MLP: Replacing class-attention with class-MLP yielded 79.2% ImageNet-val top-1 accuracy, comparable to a ResNet-50 trained with a modern strategy.This attention-free network became the baseline for subsequent ablations.
- Distillation: 81.5% ImageNet-val top-1 accuracy was achieved by the same model with hard distillation.The chosen distillation method requires no parameter tuning relative to vanilla cross-entropy.
- Communication MLP: The explored communication MLP varied its latent size through expansion factors e in linear N × e × N — GELU — linear e × N × N.Average pooling was used before classification in this experiment.
- Communication layer: 79.5% performance followed adoption of a simple N ×N linear patch-communication layer after large expansion factors proved detrimental.The linear choice also enabled visualization of interactions between patches.
- Normalization: Affine transformation was retained instead of normalization based on batch or channel statistics.The choice was made for simplicity and independence from those statistics.
- Position encoding: Removing position encoding did not change results with an MLP or simple linear patch-communication layer.The linear layer and its bias implicitly encode patch identity and position.
B Analysis of interaction layers in 12-layer networks
The 12-layer analysis examines learned linear patch-interaction matrices and their sparsity. Supervised and distilled models show patterns similar to deeper networks, whereas the shorter self-supervised model is less sparse in patch communication.
- Interaction visualizations: The 12-layer supervised ResMLP shows linear patch-interaction layers similar to those observed in the 24-layer model.All 12 linear patch-interaction layers are visualized for the ImageNet-1k-trained model.
- Sparsity: Supervised 12-layer sparsity levels are similar to those in the supervised 24-layer model.The analysis reports sparsity measurements analogous to the deeper-model analysis.
- Sparsity: The distilled 12-layer model has overall sparsity similar to supervised 12- and 24-layer models.The comparison covers both linear and MLP layers.
- Sparsity: The self-supervised model is less sparse on the patch-communication linear layer than the supervised and distilled models.It was trained for 300 epochs, compared with 400 epochs for the other models.
C Model definition in Pytorch
ResMLP is implemented as a stack of residual blocks that mix information across patches and channels using separate linear and MLP components. The full model projects image patches, applies the blocks, averages patch representations, and classifies them.
- C Model definition in Pytorch: The implementation uses an affine transformation without a normalization layer, parameterized by learnable scale and bias vectors.The affine operation computes alpha * x + beta.
- C Model definition in Pytorch: The channel MLP expands each patch representation from dim to 4 * dim, applies GELU, and projects it back to dim.The MLP uses two linear layers with GELU between them.
- C Model definition in Pytorch: ResMLP blocks combine a linear layer between patches with an MLP that processes patches independently.The block is described as a linear operation on patches followed by channel-wise MLP processing.
- C Model definition in Pytorch: Each block applies affine transformations, two residual branches, and learnable LayerScale parameters before returning the updated representation.The patch-mixing and channel-MLP branches are added residually after LayerScale modulation.
- C Model definition in Pytorch: The full network projects the input image into patches, processes them through repeated ResMLP blocks, averages patches, and applies a linear classifier.The model stacks blocks according to depth, then uses average pooling before classification.
D Additional Ablations
The additional ablations examine training strategy, training duration, and pooling choices. They show that ResMLP benefits from modern training, requires longer schedules than DeiT to reach saturation, and gains substantially from class-MLP pooling, although the pooling gap narrows in deeper models.
- D Additional Ablations: The original ResNet-50 training recipe degrades ResMLP performance relative to the DeiT training strategy.The ablation removes components of DeiT training to recover the ResNet-50 recipe and observes lower performance.
- D Additional Ablations: ResMLP-S36 performance saturates after 800 training epochs, whereas DeiT saturates after 400 epochs.The comparison indicates that ResMLP needs more epochs to become optimal.
- D Additional Ablations: Class-MLP pooling performs much better than average pooling with only small changes in FLOPs and parameter count.The comparison covers different depths with and without distillation.
- D Additional Ablations: The performance gap between class-MLP and average pooling decreases as models become deeper.This trend is reported for the pooling comparison across deeper models.