PaperlayerView exact revision on arXiv

Source-linked AI summary

RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition

Xiaohan Ding, Chunlong Xia, Xiangyu Zhang, Xiaojie Chu, Jungong Han, Guiguang Ding

arXiv:2105.01883v3cs.CVcs.AIcs.LG

TL;DR

RepMLP addresses the difficulty of using fully connected layers for images, which offer global and positional modeling but lack local priors. It adds convolutional branches during training and merges them into fully connected layers for inference, improving CNN-based results across several vision tasks with lower computational cost.

  • Problem

    Fully connected layers provide efficient global capacity and positional perception but lack the local prior that makes convolution effective for image recognition.

  • Method

    RepMLP adds parallel convolution and batch-normalization layers to fully connected layers during training, then merges them into a three-FC inference block.

  • Results

    RepMLP outperforms traditional ConvNets across ImageNet classification, face recognition, and semantic segmentation, while running faster than convolution at the same parameter count.

  • Takeaways & Limitations

    Combining fully connected global and positional modeling with convolutional locality improves neural networks across tasks with both translation invariance and aligned positional patterns.

  • Takeaways & Limitations

    RepMLP assumes spatial dimensions h, w > 7 for its stated local convolution branches and prioritizes inference throughput and accuracy over parameter count.

Abstract

from arXiv · show

We propose RepMLP, a multi-layer-perceptron-style neural network building block for image recognition, which is composed of a series of fully-connected (FC) layers. Compared to convolutional layers, FC layers are more efficient, better at modeling the long-range dependencies and positional patterns, but worse at capturing the local structures, hence usually less favored for image recognition. We propose a structural re-parameterization technique that adds local prior into an FC to make it powerful for image recognition. Specifically, we construct convolutional layers inside a RepMLP during training and merge them into the FC for inference. On CIFAR, a simple pure-MLP model shows performance very close to CNN. By inserting RepMLP in traditional CNN, we improve ResNets by 1.8% accuracy on ImageNet, 2.9% for face recognition, and 2.3% mIoU on Cityscapes with lower FLOPs. Our intriguing findings highlight that combining the global representational capacity and positional perception of FC with the local prior of convolution can improve the performance of neural network with faster speed on both the tasks with translation invariance (e.g., semantic segmentation) and those with aligned images and positional patterns (e.g., face recognition). The code and models are available at https://github.com/DingXiaoH/RepMLP.

2 MEGVII Technology 3 Computer Science Department, Aberystwyth University, SY23 3FL, UK

The work acknowledges institutional and scholarship support, with additional author-contribution notes.

  • The work was supported by Chinese national research programs and the Beijing Academy of Artificial Intelligence.The listed support includes the National Key Research and Development Program of China, the National Natural Science Foundation of China, and BAAI.
  • Xiaohan Ding received funding from the Baidu Scholarship Program 2019.
  • Xiaohan Ding conducted the work during an internship at MEGVII, while the paper identifies equal and corresponding authors.

1. Introduction

RepMLP combines the global and positional strengths of fully connected layers with convolutional local priors through structural re-parameterization. The resulting block targets efficient image recognition across classification, aligned face recognition, and translation-invariant segmentation.

  • Image locality motivates convolutional local priors, while long-range dependencies require global capacity beyond repeated local operations.The paper notes that self-attention can provide global capacity but lacks local prior, and ViT required 3×10^8 images in JFT-300M to converge.
  • Fully connected transformations provide positional perception and global capacity because position-specific parameters connect every output point to every input point.The paper presents this operation as efficient in actual speed and theoretical FLOPs, particularly when throughput and accuracy matter more than parameter count.
  • During training, parallel convolution and batch-normalization layers are merged into the fully connected layer once for inference, yielding a three-FC RepMLP block without inference-time conversion costs.
  • RepMLP is reported to outperform traditional ConvNets across ImageNet classification, face recognition with positional priors, and semantic segmentation with translation invariance.
  • RepMLP equips fully connected layers with convolutional local priors while retaining global capacity and positional perception for image recognition.

2. Related Work

Related work covers self-attention and simplified global-attention modules, alongside structural re-parameterization methods that convert training-time branches into simpler inference architectures.

  • 2.1. Designs for Global Capacity: Non-local Networks and GCNet model global capacity through self-attention or global attention pooling, feature transformation, and aggregation.
  • 2.1. Designs for Global Capacity: RepMLP uses only three fully connected layers without self-attention and is reported to improve ResNet-50 more than Non-local and GC modules.
  • 2.2. Structural Re-parameterization: Structural re-parameterization constructs convolution and batch-normalization branches during training, then merges their parameters into a fully connected layer for inference.
  • 2.2. Structural Re-parameterization: ACB strengthens square convolutions with horizontal and vertical branches, while RepVGG converts identity and 1 × 1 branches into an inference-time 3 × 3-convolution architecture.
  • 2.2. Structural Re-parameterization: RepMLP differs from these methods by making MLPs powerful image-recognition replacements for regular convolutions rather than strengthening convolutions themselves.The paper notes that its training-time convolutions could additionally use ACB, RepVGG, or other convolutional forms.

3. RepMLP

RepMLP combines global and positional modeling from fully connected layers with convolutional local priors, using partitions and structural re-parameterization to preserve efficient inference. Its training-time components are merged into three fully connected layers, and the block is adapted for ResNet bottlenecks.

  • 3. RepMLP: RepMLP contains Global, Partition, and Local Perceptrons, with a differentiable method for converting the training-time block into three FC layers for inference.The Local Perceptron uses convolutional branches, while the Partition Perceptron models partition-level dependencies.
  • 3.1. Formulation: A feature map is flattened, multiplied by an FC kernel, and reshaped back, giving every output point access to every input point but losing image locality.The resulting FC has global capacity and positional perception, while its parameter count grows as COH^2W^2.
  • 3.2. Components of RepMLP: Partitioning reduces the FC parameter requirement from COH^2W^2 to COh^2w^2, while Global Perceptron restores correlations among spatially separated partitions.Global Perceptron pools each partition, processes the pooled values with BN and a two-layer MLP, then adds the result to the partition map.
  • 3.3. A Simple, Platform-agnostic, Differentiable Algorithm for Merging Conv into FC: The equivalent FC kernel for a convolution is constructed by convolving the convolution kernel with a reshaped identity matrix, yielding a platform-agnostic and differentiable conversion.The converted kernel can be derived during training and used in objectives such as penalty-based pruning.
  • 3.4. Converting RepMLP into Three FC Layers: Batch normalization is fused into preceding convolutions and FC3, then converted convolution kernels and replicated biases are added to obtain one inference-time FC kernel and bias.The Global Perceptron’s BN is absorbed into FC1 because sequential matrix multiplications can be merged.
  • 3.5. RepMLP-ResNet: RepMLP Bottlenecks replace stride-1 ResNet bottlenecks after channel reduction and expansion, while the original stride-2 bottleneck is retained.The proposed block is applied in ResNet experiments, although the paper identifies convolution-to-FC re-parameterization and RepMLP’s three components as its core contributions.

4. Experiments

Experiments test RepMLP as a pure MLP and as a building block in ImageNet classification, face recognition, and semantic segmentation. Results show that local priors, global information, and positional modeling jointly improve accuracy while retaining efficiency.

  • 4.1. Pure MLP and Ablation Studies: 91.11% accuracy at 52.8M FLOPs: the pure MLP approaches CNN performance on CIFAR-10 but does not outperform the Wide ConvNet.The comparable-FLOPs pure MLP uses stage widths 16, 32, 64, while the Wide ConvNet uses 32, 64, 128.
  • 4.1. Pure MLP and Ablation Studies: 8.5% lower accuracy without Local Perceptron and 3.5% lower accuracy when FC3 is replaced by a larger-receptive-field conv show the importance of local priors and FC capacity.Replacing RepMLP directly with a 9 × 9 convolution performs worse because it lacks Global Perceptrons; structural re-parameterization also reduces inference cost after training.
  • 4.2. RepMLP-ResNet for ImageNet Classification: 10% faster with fewer parameters: RepMLP-Res50 using 8× channel reduction runs faster than ResNet-50 in the c4-only study.Increasing parameters through groupwise 1 × 1 convolution raises parameters by 59% but decreases speed by only 0.7%; subsequent experiments use r = 2 or 4 and g = 4 or 8.
  • 4.2. RepMLP-ResNet for ImageNet Classification: 0.94% higher accuracy: using RepMLP only in c4 adds 5M parameters, while c3+c4 provides the best trade-off and all-stage replacement lowers accuracy.The experiments use g = 8 and r = 2, 2, 4, 4 for c2, c3, c4, and c5, respectively.
  • 4.2. RepMLP-ResNet for ImageNet Classification: RepMLP-Res50 matches ResNet-101 accuracy with 50% of its FLOPs, 4M fewer parameters, and 50% faster speed at 224 × 224 input.At 320 × 320 input, RepMLP-Res50 outperforms traditional ConvNets in accuracy, speed, and FLOPs; it is also around 1% more accurate than Non-local and GC counterparts.
  • 4.2. RepMLP-ResNet for ImageNet Classification: 77.14% versus 77.19%: the RepMLP Light Block matches ResNet-50 accuracy with 30% lower FLOPs and 55% faster speed.The Light Block uses only 1 × 1 convolutions for 8× channel reduction and expansion around RepMLP.
  • 4.3. Face Recognition: 4.91% higher accuracy and 8% faster: RepMLP-FaceRes outperforms MobileFaceNet, although it uses 2.5× FLOPs.Face recognition uses MS1M-V2 for training and MegaFace for evaluation at 96 × 96 resolution.
  • 4.4. Semantic Segmentation: 2.21% higher mIoU: PSPNet with RepMLP-Res50-g4/8 outperforms the Res-50 backbone while using lower FLOPs and running faster.The RepMLP model has more parameters, and the comparison uses Cityscapes semantic segmentation.

5. Conclusion

RepMLP equips fully connected layers with local prior while retaining their global capacity and positional perception. The method is intended for accuracy- and throughput-focused applications rather than parameter minimization.

  • RepMLP combines the global capacity and positional perception of FC layers with local prior by re-parameterizing convolutions into them.The training-time convolution and batch-normalization branches are merged into the FC for inference.
  • RepMLP is designed for applications prioritizing inference throughput and accuracy over the number of parameters.

Appendix A: RepMLP-ResNet for High Speed

The RepMLP Light Block targets higher speed by replacing the standard bottleneck's 3 × 3 convolution with aggressive channel reduction and expansion around RepMLP. It nearly preserves accuracy while reducing computation and increasing speed.

  • 77.14% vs. 77.19% accuracy, with 30% lower FLOPs and 55% faster speed than the original ResNet-50.The Light Block uses no 3 × 3 convolution and applies 8× channel reduction/expansion through 1 × 1 convolutions.

Appendix B: Converting Groupwise Conv into FC

The groupwise conversion derives the FC equivalent by splitting the input into parallel groups and converting each group separately. This extends the convolution-to-FC re-parameterization to grouped convolutions.

  • Groupwise convolution-to-FC conversion first splits the input into g parallel groups, then converts every group independently.When g = 1, the implementation exactly matches the paper's Eq. 15.

Appendix C: Absorbing BN into FC1

This appendix provides implementation material for RepMLP's re-parameterization and batch-normalization absorption. The included algorithms cover groupwise conversion and the associated parameter updates.

  • Algorithm 2 absorbs BN into FC1 by scaling its kernel and updating its bias to preserve the transformation.The BN scale and bias are projected through the FC1 kernel before the BN is removed.
  • Algorithm 1 converts groupwise convolution into an equivalent FC kernel and bias using an identity input construction and grouped convolution.
Loading 2105.01883v3…