Source-linked AI summary
RepVGG: Making VGG-style ConvNets Great Again
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, Jian Sun
TL;DR
Existing ConvNets often trade simplicity and deployability against the performance of complicated multi-branch designs. RepVGG separates training from inference through structural re-parameterization, achieving a plain 3 × 3-convolution/ReLU body with over 80% ImageNet top-1 accuracy and favorable speed-accuracy trade-offs.
Problem
Complicated ConvNets can be difficult to implement, reduce memory utilization, and have actual speeds that are not accurately reflected by FLOPs.
Method
RepVGG trains with identity and 1 × 1 branches, then structurally re-parameterizes them into an inference-time stack of 3 × 3 convolutions and ReLUs.
Results
RepVGG reaches over 80% top-1 accuracy on ImageNet and shows a favorable speed-accuracy trade-off compared to state-of-the-art models.
Takeaways & Limitations
The resulting models are especially suitable for GPUs and specialized inference chips because their inference body uses a single operator type.
Takeaways & Limitations
RepVGG prioritizes GPU and specialized-hardware speed over parameter count and may be less favored than MobileNets and ShuffleNets for low-power devices.
Abstract
from arXiv · showhide
We present a simple but powerful architecture of convolutional neural network, which has a VGG-like inference-time body composed of nothing but a stack of 3x3 convolution and ReLU, while the training-time model has a multi-branch topology. Such decoupling of the training-time and inference-time architecture is realized by a structural re-parameterization technique so that the model is named RepVGG. On ImageNet, RepVGG reaches over 80% top-1 accuracy, which is the first time for a plain model, to the best of our knowledge. On NVIDIA 1080Ti GPU, RepVGG models run 83% faster than ResNet-50 or 101% faster than ResNet-101 with higher accuracy and show favorable accuracy-speed trade-off compared to the state-of-the-art models like EfficientNet and RegNet. The code and trained models are available at https://github.com/megvii-model/RepVGG.
1. Introduction
RepVGG addresses the practical gap between simple, deployable ConvNets and complicated architectures by training with branches but inferring with a plain VGG-like stack. Structural re-parameterization converts the trained multi-branch model into a single-operator inference body with a favorable speed-accuracy trade-off.
- Motivation: Complicated multi-branch and specialized ConvNets can be difficult to implement, reduce memory utilization, and run slower than their FLOPs suggest.Actual inference speed depends on factors such as branching, memory access, and device support.
- RepVGG: RepVGG uses a VGG-like feed-forward topology whose body contains only 3 × 3 convolutions and ReLUs.Its concrete architecture is specified without automatic search, manual refinement, or compound scaling.
- Structural re-parameterization: Structural re-parameterization decouples training and inference by converting parameters from a multi-branch topology into an equivalent plain architecture.The transformation combines the 3 × 3, 1 × 1, identity, and batch-normalization parameters into a single 3 × 3 kernel and bias.
- Structural re-parameterization: Training-time RepVGG uses identity and 1 × 1 branches, while these branches are removed after training through algebraic parameter transformation.The identity branch can be treated as a degraded 1 × 1 convolution and then embedded into a 3 × 3 convolution.
- Inference: An inference-time RepVGG has one operator type—3 × 3 convolution followed by ReLU—supporting fast execution on GPUs and specialized hardware.The plain topology is described as memory-economical and can permit hardware with many 3 × 3-ReLU units.
- Contributions: RepVGG is presented as having a favorable speed-accuracy trade-off and effectiveness in image classification and semantic segmentation.The contributions also emphasize efficiency and ease of implementation.
2. Related Work
Related work explored increasingly complicated ConvNets, architecture search, and branch-free training, but RepVGG targets a simpler practical model that retains competitive performance. Its re-parameterization differs from prior kernel-parameter methods by using a real training-time multi-branch dataflow that converts into a plain model.
- Prior architectures: Complicated architectures and architecture-search methods can increase implementation effort, reduce parallelism, and limit deployment on ordinary GPUs.Some large NAS-generated models are reported as not trainable on ordinary GPUs.
- Plain ConvNets: Earlier branch-free ConvNet work mainly targeted convergence of very deep models rather than better performance and practicality than complicated architectures.The paper distinguishes its goal from training extremely deep plain networks.
- RepVGG: RepVGG instead seeks reasonable depth, favorable accuracy-speed trade-offs, and implementation with common components and simple algebra.Its training and inference architectures are deliberately decoupled through re-parameterization.
- Re-parameterization methods: DiracNet encodes convolution kernels mathematically for optimization, whereas RepVGG trains an actual multi-branch model that is later converted into another structure.The paper reports DiracNet below comparable ResNets by 2.29% on CIFAR-100 and 0.62% on ImageNet.
- Re-parameterization methods: ACB, DO-Conv, and ExpandNet apply structural re-parameterization as component-level replacements, while RepVGG uses it to train plain ConvNets.The paper identifies this training role as critical to RepVGG.
- Efficient convolution: RepVGG favors 3 × 3 convolutions because libraries such as NVIDIA cuDNN and Intel MKL optimize them on GPUs and CPUs.The paper contrasts theoretical FLOPs with hardware-aware measures such as Winograd multiplications and actual speed.
3. Building RepVGG via Structural Re-param
RepVGG uses structural re-parameterization to combine training-time branches into a plain inference model, targeting speed, memory efficiency, and flexibility. Its architecture and conversion rely on common 3×3 convolutions, branch fusion, and stage-wise width and depth choices.
- Motivation: RepVGG targets fast, memory-economical, and flexible inference by favoring simple ConvNets and accounting for memory access cost and parallelism beyond FLOPs.Branch additions and concatenations can incur significant memory access cost, while plain topologies improve configurability and pruning flexibility.
- Training-time Multi-branch Architecture: Multi-branch training addresses the poor performance of plain ConvNets by using identity, 1×1, and 3×3 branches in an information flow y = x + g(x) + f(x).The identity branch is used only when dimensions match, while the branches are intended for training time.
- Re-param for Plain Inference-time Model: Structural re-parameterization converts each BN-equipped branch into a biased convolution, then fuses the resulting kernels and biases into one 3×3 convolution.The identity branch is treated as a 1×1 convolution with an identity kernel; 1×1 kernels are zero-padded and added at the 3×3 kernel’s center.
- Re-param for Plain Inference-time Model: The branch-fusion equivalence requires matching strides and a one-pixel-smaller padding configuration for the 1×1 layer than for the 3×3 layer.For a 3×3 layer with padding 1, the 1×1 layer uses padding 0.
- Architectural Specification: RepVGG arranges 3×3 layers into five stages, downsamples at each stage’s first layer, and uses global average pooling plus a fully connected classification head.The stage depths are 1, 2, 4, 14, and 1 layers; stage widths are scaled using multipliers a and b, with stage 1 width min(64, 64a).
4. Experiments
Experiments show that RepVGG delivers strong accuracy-speed trade-offs on ImageNet, benefits critically from structural re-parameterization, and generalizes to semantic segmentation while remaining practical on GPUs.
- ImageNet classification: RepVGG-A0/A1/A2 are 1.25%/33%, 0.29%/64%, and 0.17%/83% better than ResNet-18/34/50 in accuracy/speed, respectively.The comparisons use ImageNet models trained with the stated simple augmentation and speed measured on a 1080Ti GPU.
- ImageNet classification: RepVGG-B1g4 is 0.37%/101% better than ResNet-101, and RepVGG-B1g2 runs 2.66× as fast as ResNet-152 at the same accuracy.The groupwise variants trade some accuracy for additional acceleration, while remaining more parameter-efficient than the compared ResNets.
- ImageNet classification: RepVGG-A2 is 1.37%/59% better than EfficientNet-B0, while RepVGG-B1 is 0.39% more accurate and slightly faster than RegNetX-3.2GF.RepVGG models also exceed 80% accuracy after 200 epochs, according to the authors.
- Efficiency measurement: Wino MULs are a better GPU speed proxy than theoretical FLOPs in the reported comparisons, although actual speed remains the definitive measure.The authors note that hardware and computing libraries determine whether Winograd acceleration applies.
- Structural re-parameterization: Removing both training-time branches reduces RepVGG-B0 accuracy to 72.39%, whereas the full model reaches 75.14%, a 2.75% improvement.Adding only the 1×1 or identity branch yields 73.15% or 74.79%, respectively; structural re-parameterization also speeds the unconverted training-time models.
- Structural re-parameterization: Replacing ResNet-50 convolutions with RepVGG blocks improves accuracy by only 0.03%, indicating the method is targeted to training powerful plain ConvNets rather than generic over-parameterization.The ACB comparison likewise suggests that extra component parameters alone do not explain RepVGG’s performance.
- Semantic segmentation: RepVGG-B1g2/B2 backbones outperform ResNet-50/101 by 1.71%/1.01% mIoU with higher speed on Cityscapes segmentation.The fast B1g2 variant is 0.37 mIoU better than the ResNet-101 backbone and runs 62% faster.
- Limitations: RepVGG prioritizes GPU and specialized-hardware speed over parameter minimization and may be less suitable than MobileNets or ShuffleNets for low-power devices.The authors still report RepVGG as more parameter-efficient than ResNets.
5. Conclusion
RepVGG combines a simple stack of 3 × 3 convolutions and ReLUs with structural re-parameterization to support efficient inference. It reaches over 80% top-1 accuracy on ImageNet and offers a favorable speed-accuracy trade-off.
- RepVGG uses a stack of 3 × 3 convolutions and ReLUs, making it especially suitable for GPUs and specialized inference chips.
- Structural re-parameterization enables the simple inference architecture while using a different training-time architecture.
- Over 80% top-1 accuracy on ImageNet demonstrates RepVGG's reported effectiveness as a plain architecture.
- RepVGG shows a favorable speed-accuracy trade-off compared to state-of-the-art models.