PaperlayerView exact revision on arXiv

Source-linked AI summary

VanillaNet: the Power of Minimalism in Deep Learning

Hanting Chen, Yunhe Wang, Jianyuan Guo, Dacheng Tao

arXiv:2305.12972v2cs.CV

TL;DR

Complex neural networks create deployment challenges, motivating a simpler architecture that avoids excessive depth, shortcuts, and intricate operations. The paper introduces VanillaNet with deep training and series-based activation techniques, and reports performance on par with well-known deep neural networks and vision transformers on large-scale image classification datasets.

  • Problem

    Increasing architectural complexity creates deployment challenges, while simpler plain networks have been hindered by weak non-linearity and poorer performance.

  • Method

    VanillaNet uses a shallow architecture without shortcuts or attention, trained with progressive nonlinear-layer removal and a series-based activation function.

  • Results

    VanillaNet performs on par with well-known deep neural networks and vision transformers on large-scale image classification datasets.

  • Takeaways & Limitations

    The experiments highlight the potential of minimalism for high-performance neural networks without complex architectures.

  • Takeaways & Limitations

    VanillaNet's simple and relatively shallow architecture has weak non-linearity that limits performance, requiring additional techniques to address it.

Abstract

from arXiv · show

At the heart of foundation models is the philosophy of "more is different", exemplified by the astonishing success in computer vision and natural language processing. However, the challenges of optimization and inherent complexity of transformer models call for a paradigm shift towards simplicity. In this study, we introduce VanillaNet, a neural network architecture that embraces elegance in design. By avoiding high depth, shortcuts, and intricate operations like self-attention, VanillaNet is refreshingly concise yet remarkably powerful. Each layer is carefully crafted to be compact and straightforward, with nonlinear activation functions pruned after training to restore the original architecture. VanillaNet overcomes the challenges of inherent complexity, making it ideal for resource-constrained environments. Its easy-to-understand and highly simplified architecture opens new possibilities for efficient deployment. Extensive experimentation demonstrates that VanillaNet delivers performance on par with renowned deep neural networks and vision transformers, showcasing the power of minimalism in deep learning. This visionary journey of VanillaNet has significant potential to redefine the landscape and challenge the status quo of foundation model, setting a new path for elegant and effective model design. Pre-trained models and codes are available at https://github.com/huawei-noah/VanillaNet and https://gitee.com/mindspore/models/tree/master/research/cv/vanillanet.

1 Introduction

VanillaNet responds to deployment challenges from increasingly complex neural networks by pursuing a simpler architecture without excessive depth, shortcuts, or intricate operations. Its training techniques preserve streamlined inference while supporting performance comparable to contemporary networks.

  • Motivation: Increasing neural-network complexity creates deployment challenges, including off-chip memory traffic from shortcuts and sophisticated engineering requirements for intricate operations.ResNet shortcuts merge features from different layers, while AS-MLP and Swin Transformer operations may require rewritten CUDA code.
  • Motivation: Plain convolutional networks received less attention because gradient vanishing and weaker benchmark performance made them less competitive than deep networks and vision transformers.A 34-layer plain network was reported to perform worse than an 18-layer one, while AlexNet and VGGNet were outpaced by ResNets and ViT.
  • VanillaNet: VanillaNet emphasizes simplicity by avoiding excessive depth, shortcuts, and intricate operations such as self-attention.The architecture is designed to address inherent complexity and suit resource-limited environments.
  • VanillaNet: VanillaNet uses deep training to progressively eliminate nonlinear layers after training, enabling easy merging while preserving inference speed.The method begins with layers containing nonlinear activations and removes these layers as training proceeds.
  • Results: VanillaNet combines its simplified architecture and training techniques with a series-based activation function to improve efficiency and accuracy over contemporary elaborate networks.The series-based activation incorporates multiple learnable affine transformations.

2 A Vanilla Neural Architecture

VanillaNet follows a conventional stem, body, and classifier structure but uses very few layers and no shortcuts. Because this shallow design has weak non-linearity, the paper introduces techniques to address that limitation.

  • Architecture: Standard image-classification networks commonly use a stem, a multi-stage main body, and a fully connected classification layer.The main body typically contains four stages formed by stacking repeated blocks.
  • Architecture: Existing networks often require many complex layers, with ResNet using 34 or 50 shortcut-connected layers and base ViT using 62 layers.These examples are associated with achieving over 70% top-1 ImageNet accuracy for ResNet and with multi-layer self-attention computation for ViT.
  • Architecture: VanillaNet uses the conventional stem, main body, and fully connected layer but employs only one layer in each stage.Its design targets as few layers as possible because complex designs and large depths can limit inference speed despite modern parallel hardware.
  • Architecture: The six-layer example uses a stride-4 stem, pooling across stages, channel increases in early stages, 1 × 1 convolutions, activations, batch normalization, and a final classifier.The architecture has no shortcut, since the authors found little performance improvement from adding one.
  • Limitation: Although VanillaNet is simple and shallow, its weak non-linearity limits performance, motivating a series of techniques to address the problem.

3 Training of Vanilla Networks

VanillaNet uses deep training and series-informed activations to compensate for the limited nonlinearity of a simple, shallow architecture. Training-time complexity is then reduced for inference by progressively removing activations and merging convolutional operations.

  • 3.1 Deep Training Strategy: VanillaNet trains two convolutional layers with an activation function before merging them into one convolution for faster inference.The activation gradually becomes an identity mapping as training progresses.
  • 3.1 Deep Training Strategy: A′(x) combines a usual activation A(x) with an identity mapping, with λ controlling the modified activation’s nonlinearity.At convergence, A′(x) becomes x, leaving no activation between the two convolutions.
  • 3.1 Deep Training Strategy: Batch normalization and its preceding convolution are first converted into a single convolution before the two 1×1 convolutions are merged.The merged representation uses convolution weights and biases together with batch-normalization scale, shift, mean, and variance.
  • 3.2 Series Informed Activation Function: Series-informed activation concurrently stacks multiple scaled and shifted activations to enhance nonlinearity without serially increasing network depth.The coefficients a_i and b_i prevent simple accumulation across the stacked activations.
  • 3.2 Series Informed Activation Function: The activation can incorporate neighboring inputs to enrich its approximation ability and learn global information from an input feature map.When n = 0, the series-based activation degenerates to the plain activation A(x).
  • 3.2 Series Informed Activation Function: 84 is the reported ratio for the 4th stage of VanillaNet-B with Cout = 2048, k = 1, and n = 7, indicating substantially lower activation cost than convolution.The paper states that the proposed activation’s computation cost remains much lower than that of the convolutional layers.

4 Experiments

Experiments evaluate VanillaNet through ablations, feature visualization, and comparisons on ImageNet and COCO. The results show that series activation and deep training improve shallow networks, while the simplified architecture offers strong accuracy-speed trade-offs.

  • Ablation Study: Series activation raises ImageNet top-1 accuracy from 60.53% with ReLU to 74.53% at n = 1, with n = 3 selected as a balance between accuracy and latency.Increasing the number of series improves performance; n = 3 is used thereafter.
  • Ablation Study: Deep training improves VanillaNet-6 ImageNet top-1 accuracy from the 75.23% baseline to 76.36%.The experiment supports its usefulness for the shallow network.
  • Ablation Study: Applying deep training and series activation improves AlexNet by about 6%, while producing little gain for ResNet-50.The authors associate the difference with ResNet-50 already having sufficient non-linearity.
  • Ablation Study: Adding shortcuts provides little improvement to VanillaNet, which the authors attribute to weak non-linearity rather than insufficient identity mapping.Shortcuts may reduce effective non-linearity by skipping activation functions.
  • Feature Visualization: GradCam++ visualizations show larger active regions for 9-layer VanillaNet than for similarly performing ResNet-50, whose active regions are smaller.The figure compares correctly and incorrectly classified samples using red for high activation and blue for weak activation.
  • Comparison with SOTA Architectures: VanillaNet-9 reaches 79.87% ImageNet accuracy at 2.91 ms GPU inference, over 50% faster than similarly performing ResNet-50 and ConvNextV2-P.VanillaNet-13-1.5׆ reaches 83.11% top-1 accuracy; Figure 3 identifies depth as strongly related to batch-size-1 inference speed.
  • Comparison with SOTA Architectures: On COCO detection and segmentation, VanillaNet achieves similar performance to ConvNext and Swin backbones while attaining much higher FPS despite higher FLOPs and parameter counts.The result demonstrates the effectiveness of the vanilla architecture on these tasks.

5 Conclusion

The paper investigates whether high-performing networks can avoid shortcuts, high depth, and attention layers. VanillaNet achieves comparable performance to deep networks and vision transformers with a concise architecture.

  • VanillaNet investigates high-performance neural networks without shortcuts, high depth, or attention layers.
  • The proposed deep training strategy and series activation function enhance VanillaNet's non-linearity during training and testing.
  • VanillaNet performs on par with well-known deep neural networks and vision transformers on large-scale image classification datasets.
  • The results show that a concise architecture can achieve comparable performance to state-of-the-art deep networks and vision transformers.

A Network Architectures

VanillaNet provides 7–13-layer architectures built from convolutional layers followed by activation functions, with channel scaling and adaptive pooling variants.

  • VanillaNet architectures contain 7–13 layers, with each convolutional layer followed by an activation function.
  • VanillaNet-13-1.5× multiplies the number of channels by 1.5.
  • The VanillaNet-13-1.5׆ variant uses adaptive pooling in stages 2, 3, and 4.The corresponding feature shapes are 40×40, 20×20, and 10×10.

B Training Details

ImageNet training uses 300 epochs with cosine learning-rate decay, while COCO detection and segmentation use ImageNet-pretrained weights and specified AdamW settings.

  • 300 epochs and cosine learning-rate decay are used to train VanillaNets for ImageNet classification.
  • The λ in Equation 1 is linearly decayed from 1 to 0 between epochs 0 and 100.
  • VanillaNet-11 uses VanillaNet-10 pretrained weights for initialization, with the same approach applied to VanillaNet-12 and VanillaNet-13.
  • COCO detection and segmentation use ImageNet-pretrained weights, AdamW, batch size 32, learning rate 1.4e-4, and 0.8 layer-wise decay for VanillaNet-11.
Loading 2305.12972v2…