PaperlayerView exact revision on arXiv

Source-linked AI summary

RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality

Xiaohan Ding, Honghao Chen, Xiangyu Zhang, Jungong Han, Guiguang Ding

arXiv:2112.11081v2cs.CVcs.AIcs.LG

TL;DR

FC layers model long-range dependencies but lack the local priors useful for image recognition. The paper injects locality into FC layers through re-parameterized convolutional branches, then builds the hierarchical RepMLPNet architecture. RepMLPNet achieves favorable accuracy-efficiency trade-offs and transfers to Cityscapes semantic segmentation, while retaining common MLP weaknesses such as overfitting and large model size.

  • Problem

    FC layers capture long-range dependencies but are weaker at capturing local image patterns, motivating an MLP that combines global capacity with locality and remains effective for visual recognition.

  • Method

    Locality Injection merges trained parallel convolutional kernels into FC kernels, forming RepMLP Blocks with Global, Channel, and Local Perceptrons within a hierarchical MLP architecture.

  • Results

    RepMLPNet achieves favorable accuracy-efficiency with 100 ImageNet training epochs compared with other MLPs trained for 300 or 400 epochs, and shows satisfactory Cityscapes segmentation transfer.

  • Takeaways & Limitations

    The results support Locality Injection as a general MLP methodology and establish RepMLPNet as an MLP-style backbone applicable to semantic segmentation.

  • Takeaways & Limitations

    RepMLPNet can overfit, may be too large for low-power devices, and shows no superiority over traditional CNNs on semantic segmentation.

Abstract

from arXiv · show

Compared to convolutional layers, fully-connected (FC) layers are better at modeling the long-range dependencies but worse at capturing the local patterns, hence usually less favored for image recognition. In this paper, we propose a methodology, Locality Injection, to incorporate local priors into an FC layer via merging the trained parameters of a parallel conv kernel into the FC kernel. Locality Injection can be viewed as a novel Structural Re-parameterization method since it equivalently converts the structures via transforming the parameters. Based on that, we propose a multi-layer-perceptron (MLP) block named RepMLP Block, which uses three FC layers to extract features, and a novel architecture named RepMLPNet. The hierarchical design distinguishes RepMLPNet from the other concurrently proposed vision MLPs. As it produces feature maps of different levels, it qualifies as a backbone model for downstream tasks like semantic segmentation. Our results reveal that 1) Locality Injection is a general methodology for MLP models; 2) RepMLPNet has favorable accuracy-efficiency trade-off compared to the other MLPs; 3) RepMLPNet is the first MLP that seamlessly transfer to Cityscapes semantic segmentation. The code and models are available at https://github.com/DingXiaoH/RepMLP.

1. Introduction

RepMLPNet addresses the tension between FC layers’ long-range modeling and convolutional layers’ locality by injecting local priors into FC layers and organizing RepMLP Blocks hierarchically. The resulting model targets small-data visual recognition, favorable accuracy-efficiency, and downstream semantic segmentation.

  • 1. Introduction: CNNs capture local patterns through neighborhood operations, whereas FC layers can directly model dependencies between any two input points but ignore image locality.The paper identifies locality as a useful inductive bias for visual recognition and seeks an MLP that is effective with ordinary training and small data.
  • 1. Introduction: Locality Injection trains parallel convolutional branches with an FC layer, then merges the convolutional parameters into the FC kernel for faster inference.This preserves local-pattern extraction during training while avoiding the reduced parallelism associated with separate convolutional inference branches.
  • 1. Introduction: The RepMLP Block combines Global, Channel, and Local Perceptrons to model global channel relations, channel-wise spatial transformations, and local patterns.The Local Perceptron uses small kernels, while the Channel Perceptron reshapes features for linear mapping and the Global Perceptron aggregates spatial information.
  • 1. Introduction: The architecture is intended to overcome the parameter-growth problem of applying FC layers directly to large lower-level feature maps.The paper notes that FC parameter counts scale with input and output sizes, making lower-level layers potentially much larger than higher-level ones.
  • 1. Introduction: RepMLPNet uses a hierarchical design whose feature-map sizes decrease while channel counts increase, producing multi-level semantic features for downstream frameworks.The share-set count is smaller in lower-level layers and larger in higher-level layers to balance model size and representational capacity.
  • 1. Introduction: RepMLPNet combines Locality Injection, RepMLP Blocks, and hierarchical architecture to achieve favorable accuracy-efficiency with 100 ImageNet training epochs versus 300 or 400 for other MLPs.The paper also reports performance improvements for ResMLP and satisfactory transfer to semantic segmentation.

2. Related Work

The paper places Locality Injection within structural re-parameterization and frames it as an extension of that idea from convolutional models to FC layers. It contrasts this bridge between convolution and FC with prior re-parameterization designs and concurrent vision MLPs.

  • 2. Related Work: Concurrent vision MLPs such as MLP-Mixer, ResMLP, and gMLP use FC-like operations to mix channel or spatial information, motivating RepMLPNet’s distinct locality-injection approach.MLP-Mixer, for example, alternates channel mixing and token mixing.
  • 2. Related Work: Structural re-parameterization converts training-time structures into inference-time structures by transforming parameters, as illustrated by RepVGG, ACB, DBB, ResRep, and RepLKNet.These methods add or transform branches and kernels during training while retaining simpler or more efficient inference structures.
  • 2. Related Work: Locality Injection generalizes structural re-parameterization beyond convolution by merging a trained convolutional kernel into an FC kernel.The method is described as a simple, platform-agnostic, differentiable bridge between convolution and FC layers.

3. Locality Injection via Re-parameterization

Locality Injection makes a convolutional branch equivalent to an FC operation by converting the convolution kernel into a reshaped Toeplitz matrix and adding it to the FC kernel. This preserves the parallel branch’s local-pattern modeling while enabling a single merged FC layer at inference.

  • The method uses feature-map flattening for FC computation, with W shaped as (ohw, chw), while convolution outputs retain spatial tensor structure.
  • An FC layer ignores image locality because each output point depends on every input point without positional information.
  • A parallel convolution can be merged into the FC by constructing an equivalent kernel W(F,p) with the same shape as the FC kernel.
  • The equivalent convolution kernel is obtained by applying the convolution to a reshaped identity matrix and then reshaping the result into an FC matrix.
  • Constructing the equivalent matrix is nontrivial because convolution platforms use different algorithms, memory layouts, padding implementations, and allocation strategies.

4. RepMLPNet

RepMLPNet combines RepMLP Blocks with a hierarchical four-stage architecture that separates global, channel-wise long-range, and local modeling. Its stage outputs provide multi-scale feature maps suitable for downstream frameworks, while set-sharing controls the otherwise large FC parameter cost.

  • 4.1. Components of RepMLP Block: A RepMLP Block combines Global, Channel, and Local Perceptrons to model coarse global dependencies, within-channel long-range dependencies, and local patterns.
  • 4.1. Components of RepMLP Block: Set-sharing FC divides channels into groups that share spatial-mapping parameters, reducing parameters from c(hw)^2 to s(hw)^2 without reducing computation versus depth-wise FC.
  • 4.1. Components of RepMLP Block: The Local Perceptron uses depth-wise convolutions, whose batch-normalized outputs are merged into the Channel Perceptron’s FC3 through Locality Injection.
  • 4.2. Hierarchical Architectural Design: Table 1 compares tiny, base, deep, and large configurations, including RepMLP-D256 as deeper and narrower than RepMLP-B256 at comparable FLOPs and parameter counts.
  • 4.2. Hierarchical Architectural Design: RepMLPNet arranges blocks in four stages, downsampling spatial dimensions between stages while doubling channels; an FFN-style block increases depth.
  • 4.2. Hierarchical Architectural Design: RepMLPNet uses a hierarchical design whose stage outputs can directly supply the four feature-map levels required by UperNet, unlike MLP-Mixer or ResMLP.

5. Experiments

RepMLPNet is evaluated under controlled ImageNet settings, with comparisons spanning MLPs, CNNs, EfficientNets, component ablations, Locality Injection, and Cityscapes segmentation. The experiments report favorable accuracy-efficiency trade-offs, show benefits from Locality Injection and architectural components, and demonstrate downstream segmentation transfer.

  • 5.1. ImageNet Classification: 0.5% higher accuracy at one-quarter the FLOPs: RepMLPNet-T256 outperforms MLP-Mixer under fair training settings.RepMLPNet-B256 also matches ResMLP-B24 without distillation using one-quarter the training epochs, 40% of its FLOPs, and fewer parameters.
  • 5.1. ImageNet Classification: 1.9× faster with four times the FLOPs: RepMLPNet-T224 outpaces EfficientNet-B1, while RepMLPNet-B256 is 3.6% more accurate than EfficientNet-B2 at comparable throughput.The authors attribute high GPU throughput to matrix multiplication, architectural simplicity, and parallelism.
  • 5.1. ImageNet Classification: Around 1% higher accuracy: adding the Global Perceptron improves performance with only 0.5M additional parameters and FLOPs.Increasing the number of share-sets further improves performance without extra FLOPs, at the cost of a larger model.
  • 5.2. Locality Injection Matters: 9.51% higher accuracy: adding convolutional branches during training substantially improves ResMLP on CIFAR-100 despite only 216 extra training-time parameters.Removing Locality Injection lowers RepMLP-T224 and RepMLP-D256 ImageNet accuracy by 2.15% and 2.30%, respectively; the method improves multiple MLP models and datasets.
  • 5.3. Semantic Segmentation: RepMLPNet provides a viable MLP backbone for Cityscapes segmentation: its hierarchical features support multi-level downstream frameworks, and replacing 3×3 with 5×5 downsampling further increases mIoU with negligible extra FLOPs.Patch-based processing restores feature maps, while regular convolutions communicate across patch borders; edge predictions are reported as good as other regions.

6. Limitations and Conclusions

RepMLPNet combines locality injection, RepMLP Blocks, and hierarchical design to achieve favorable accuracy-efficiency trade-offs, while retaining common MLP weaknesses and no observed superiority over traditional CNNs for segmentation.

  • Limitations: MLPs remain easy to overfit, may be too large for low-power devices, and RepMLPNet shows no superiority over traditional CNNs for semantic segmentation.

Appendix A: Visualizing Locality Injection

Visualized FC kernels show that Locality Injection adds stronger weighting around neighboring positions while preserving long-range dependencies outside the merged 3×3 convolutional region.

  • Visualization setup: The original kernel visualization provides the baseline pattern used to contrast locality before and after Locality Injection.
  • Visualization setup: The visualization samples the FC3 kernel from the tenth RepMLP Block in stage three of RepMLPNet-D256 and reshapes it before inspecting one input-channel and output-position slice.
  • Kernel visualization: After Locality Injection, the FC kernel assigns larger values around the marked output position, indicating greater emphasis on neighboring input positions.The original FC kernel lacks a locality pattern, with neighbors not receiving larger values than other positions.
  • Kernel visualization: The re-parameterized kernel retains long-range capacity because some positions outside the 3×3 convolutional region still have larger values than positions inside it.

Appendix B: Details of Semantic Segmentation

RepMLPNet supports semantic segmentation through hierarchical multi-level features, fixed-size patch processing, and inter-patch communication, with edge predictions reported as comparable to internal pixels.

  • Semantic segmentation design: RepMLPNet addresses semantic segmentation with hierarchical features, non-overlapping patches, and communication between patches.UperNet’s four feature levels fit the hierarchical architecture, while patch splitting accommodates RepMLP Block’s fixed input size.
  • Inter-patch communication: A 3×3 stride-2 depth-wise convolution, preceded by 1×1 channel expansion, replaces 2× embedding to enable communication across patch boundaries while reducing computational cost.
  • Patch processing: For 512×1024 inputs, 4× downsampling yields 128×256 features that are split into 2×4 non-overlapping 64×64 patches before processing.The outputs of four stages are reshaped back and fed into UperNet.
  • Segmentation behavior: Predictions at patch edges are observed to be as good as predictions on internal pixels in Cityscapes validation examples.The reported test resolution is 1024×2048, with the first RepMLP Block receiving 256×512 features split into 32 patches.
  • Scope: The segmentation transfer is described as a promising first attempt, although the paper reports no superiority over traditional CNNs.
Loading 2112.11081v2…