PaperlayerView exact revision on arXiv

Source-linked AI summary

Patches Are All You Need?

Asher Trockman, J. Zico Kolter

arXiv:2201.09792v1cs.CVcs.AIcs.LG

TL;DR

The paper investigates whether Vision Transformers’ strong performance is partly attributable to patch representations rather than the Transformer architecture itself. It introduces ConvMixer, a simple convolutional patch-based model using standard convolutions, and reports competitive or superior results against several Transformer, MLP-Mixer, and classical vision baselines. The findings provide evidence that patch embeddings are an important component of newer vision architectures, while broader experiments remain necessary.

  • Problem

    The paper asks whether Vision Transformers’ strong performance results partly from patch-based input representations rather than solely from the Transformer architecture.

  • Method

    ConvMixer operates directly on patches, separates spatial and channel mixing, preserves equal resolution and size, and uses standard convolutions throughout.

  • Results

    ConvMixers outperform Vision Transformer and MLP-Mixer variants and are competitive with ResNets, DeiTs, and ResMLPs under the reported settings.

  • Takeaways & Limitations

    Patch embeddings and isotropic patch-based designs are supported as powerful components of vision models, motivating ConvMixer as a baseline for future patch-based architectures.

  • Takeaways & Limitations

    The model is not state-of-the-art, and more experiments are needed to disentangle patch embeddings from other architectural choices and test large-scale pretraining.

Abstract

from arXiv · show

Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/locuslab/convmixer.

1 Introduction

The paper asks whether Vision Transformers’ performance comes partly from patch-based input representations rather than solely from Transformer architecture. It introduces ConvMixer, a simple convolutional model that operates on patches and separates spatial from channel mixing.

  • ConvMixer directly operates on patches, maintains equal resolution and size across layers, and avoids successive downsampling.
  • The architecture consists of patch embeddings, repeated fully convolutional processing, global pooling, and a linear classifier.
  • ConvMixer separates spatial mixing from channel-wise mixing, using standard convolutions for both operations.
  • ConvMixer outperforms similar-parameter ResNets and some Vision Transformer and MLP-Mixer variants despite its extreme simplicity.

2 A Simple Model: ConvMixer

ConvMixer combines patch embeddings with repeated convolutional mixing blocks while preserving the spatial layout of patches. Its design uses depthwise convolutions for spatial mixing, pointwise convolutions for channel mixing, and unusually large kernels.

  • Patch embeddings are implemented as convolutions with kernel size and stride equal to the patch size, producing an h-dimensional representation.
  • Each ConvMixer block applies depthwise convolution for spatial mixing followed by 1 × 1 pointwise convolution for channel mixing.
  • Global pooling converts the final representation into a feature vector of size h for a softmax classifier.
  • The model is controlled by width h, depth d, patch size p, and depthwise kernel size k, and supports variable-sized inputs.
  • Large kernels provide a way to mix distant spatial locations while retaining convolution’s vision-oriented inductive bias and data efficiency.

3 Experiments

Experiments primarily evaluate ConvMixer on ImageNet-1k without pretraining or additional data. ConvMixer achieves strong accuracy across parameter budgets, while performance depends on kernel and patch size choices and inference is slower than competitors.

  • 81.4% top-1 accuracy is achieved by ConvMixer-1536/20 with 52M parameters, while ConvMixer-768/32 reaches 80.2% with 21M parameters.
  • Reducing the kernel size from k = 9 to k = 3 lowers ConvMixer-1536/20 accuracy by approximately 1%, while increasing patch size from 7 to 14 makes it around 4× faster but lowers accuracy to 78.9%.
  • ConvMixer-1536/20 outperforms ResNet-152 and ResMLP-B24 with substantially fewer parameters and is competitive with DeiT-B.
  • 82.2% top-1 accuracy is obtained for ConvMixer-1536/20 after adapting later-published regularization parameters, a gain of +0.8%.
  • On CIFAR-10, ConvMixers exceed 96% accuracy with as few as 0.7M parameters.

4 Related Work

Related work develops isotropic patch-based architectures, explores convolutional alternatives and hybrids, and identifies trade-offs between patch size, accuracy, and model complexity. ConvMixer focuses on a simpler convolutional realization of patch-based processing.

  • Isotropic vision models maintain equal size and shape throughout the network and commonly begin with patch embeddings.
  • MLP-Mixer, ResMLP, CycleMLP, gMLP, and vision permutator replace self-attention and MLP operations with alternative spatial or channel mixing mechanisms.
  • Isotropic MobileNets reported a patch-size and accuracy trade-off and achieved similar performance with a substantially more complex block than ConvMixer.
  • Convolutional stems and nearby-patch combinations can improve Vision Transformers, but they conflate patch effects with convolutional inductive biases such as locality.
  • Other work combines convolution and self-attention or adds downsampling to bridge convolutional and Transformer architectures.

5 Conclusion

ConvMixers provide evidence that patch-based, isotropic representations are powerful independently of attention, while remaining simple and competitive rather than state-of-the-art.

  • ConvMixers independently mix spatial and channel locations of patch embeddings using only standard convolutions.
  • Large kernel sizes provide a substantial performance boost, reflecting the large receptive fields of ViTs and MLP-Mixers.
  • ConvMixers outperform ViT and MLP-Mixer while remaining competitive with ResNets, DeiTs, and ResMLPs without accuracy or speed optimization.
  • Patch embeddings allow all downsampling to happen at once, reducing internal resolution and increasing effective receptive field size for mixing distant spatial information.
  • The model is not state-of-the-art, and the authors position its simple design as a baseline or template for future patch-based architectures.
  • Future work includes deeper models with larger patches, improved training and tuning, faster large-kernel depthwise convolution, and evaluation on segmentation and detection.

A Comparison to other models

Across comparisons, ConvMixers achieve strong accuracy for their parameter budgets and often exceed similarly sized baselines, but their throughput is generally lower and comparisons have important confounds.

  • Additional A1-inspired regularization settings improved ConvMixer performance, although training remained limited to 150 rather than 600 epochs.
  • 0.36% higher accuracy: ConvMixer-768/32 exceeds DeiT-S with 0.9M fewer parameters.
  • ConvMixers generally have lower throughput, with ConvMixer-512/16 measured as 4× slower than DeiT-Ti.
  • Comparisons are constrained by different patch sizes, unequal tuning and training durations, and limited compute for ConvMixer experiments.
  • 0.37% lower accuracy: ResMLP-B24/8 underperforms ConvMixer-1536/20 despite having over twice as many parameters.
  • ConvMixer is around 1.2-1.6% less accurate than Swin Transformer for similar parameter budgets and is 4-6× slower.
  • 0.94% top-1 accuracy is lost when ConvMixer-1536/20 reduces kernel size from k = 9 to k = 3, despite only 2.2M fewer parameters.

B Experiments on CIFAR-10

CIFAR-10 experiments examine residual connections and normalization while acknowledging that CIFAR-10 tuning may not transfer to larger datasets.

  • 95.88% accuracy is achieved by the baseline without one residual connection, while including it reduces accuracy to 94.78%.
  • 0.31% accuracy is lost when all residual connections are removed, but the authors caution that this may not generalize to deeper models or larger datasets.
  • Replacing BatchNorm with LayerNorm decreases performance by around 1% and slows convergence in the tested shallow model.
  • The study tunes width, depth, patch size, kernel size, and weight decay on CIFAR-10 before evaluating the architecture.
  • The authors acknowledge that CIFAR-10 tuning does not necessarily generalize to larger datasets.

B.1 Results

CIFAR-10 experiments show that ConvMixer performance depends strongly on kernel size and patch size, while scaling width or depth and tuning weight decay can improve accuracy.

  • CIFAR-10 performance: >96% accuracy is achieved with 887,000 parameters on CIFAR-10, while >91% is achieved with 100,000 parameters.The experiments used RandAug, mixup, CutMix, random erasing, gradient norm clipping, and timm augmentations.
  • Scaling ConvMixer: Increasing width or depth increases accuracy, but doubling depth incurs less compute than doubling width.Deeper networks take longer to converge, whereas wider networks converge faster.
  • Kernel size: 93.61% accuracy with kernel size 3 rises by 1.50% at size 5 and 0.61% at size 7, with later gains becoming marginal.The authors therefore use kernel sizes 7 or 9 in most later experiments.
  • Kernel size: 94.29% accuracy from a depth-10 model with k = 3 remains 1.5% below a similarly sized depth-8 model with k = 9.Tripling depth to 24 with k = 3 still produced lower accuracy than the large-kernel comparison.
  • Patch size: Increasing patch size reduces accuracy on CIFAR-10: p = 2 lowers it by about 0.80%, while p = 4 achieves only 92.61%.For p = 2, increasing kernel size beyond 5 provides only marginal benefit, with a 1.13% improvement from 3 to 5.
  • Weight decay: Tuning weight decay provides an additional 0.15% accuracy at no cost, so weight decay is used in larger ImageNet experiments.The larger-scale experiments use weight decay without tuning.

C Weight Visualizations

The visualizations show structured patch-embedding filters and a depth-dependent hierarchy in ConvMixer’s hidden convolutional kernels.

  • Patch embeddings: Patch embeddings contain Gabor-like filters, colorful globs, rough edge detectors, and some filters that resemble noise.The authors describe the learned representations as difficult to interpret conclusively.
  • Patch embeddings: Patch-embedding weights look similar when changing from patch size 14 to 7, with the smaller-patch version appearing downsampled.The authors note that this does not explain the observed accuracy drop for larger patches.
  • Hidden kernels: Successive ConvMixer layers show a feature hierarchy: early depthwise kernels are relatively small, while later layers use their full allowed size.A similar progression was observed in ResMLP despite its unconstrained matrix-multiplication layers.

D Implementation

The ConvMixer implementation is presented in compact PyTorch code, with residual blocks repeated through the network before global pooling and classification.

  • Implementation: Residual blocks add each block’s output back to its input through self.fn(x) + x.This residual wrapper is used in the repeated ConvMixer block sequence.
  • Implementation: The model is defined as ConvMixer(h, d, k, p, n), using h, d, p, and k for dimension, depth, patch size, and kernel size.The implementation caption explicitly maps these symbols to the model’s configuration parameters.
  • Implementation: The paper also provides a more readable PyTorch implementation and an even terser version under 280 characters.The concise implementation is presented as a compact expression of the ConvMixer model.
Loading 2201.09792v1…