PaperlayerView exact revision on arXiv

Source-linked AI summary

Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks

Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo, Song Wen, Chul-Ho Lee, S. -H. Gary Chan

arXiv:2303.03667v3cs.CV

TL;DR

Reducing FLOPs does not necessarily reduce neural-network latency because low FLOPS, driven especially by frequent memory access, can limit effective speed. The paper introduces PConv and builds FasterNet around it, achieving faster execution across devices while maintaining accuracy. FasterNet-T0, for example, is substantially faster than MobileViT-XXS across GPU, CPU, and ARM processors, while FasterNet-L matches Swin-B-level accuracy with higher throughput and lower CPU compute time.

  • Problem

    Reducing FLOPs does not necessarily produce proportionally lower latency because many networks have inefficiently low FLOPS, particularly from frequent memory access.

  • Method

    The paper proposes PConv, which applies convolution to only part of the input channels, and builds the hardware-friendly FasterNet family upon it.

  • Results

    FasterNet achieves favorable speed-accuracy trade-offs across devices and vision tasks, including 2.8×, 3.3×, and 2.4× faster execution than MobileViT-XXS on GPU, CPU, and ARM, respectively.

  • Takeaways & Limitations

    Optimizing FLOPS alongside FLOPs can produce neural networks with lower latency and higher throughput without compromising reported accuracy.

  • Takeaways & Limitations

    PConv requires stride 1 to align convolved and untouched channels, although additional operations can still down-sample spatial resolution.

Abstract

from arXiv · show

To design fast neural networks, many works have been focusing on reducing the number of floating-point operations (FLOPs). We observe that such reduction in FLOPs, however, does not necessarily lead to a similar level of reduction in latency. This mainly stems from inefficiently low floating-point operations per second (FLOPS). To achieve faster networks, we revisit popular operators and demonstrate that such low FLOPS is mainly due to frequent memory access of the operators, especially the depthwise convolution. We hence propose a novel partial convolution (PConv) that extracts spatial features more efficiently, by cutting down redundant computation and memory access simultaneously. Building upon our PConv, we further propose FasterNet, a new family of neural networks, which attains substantially higher running speed than others on a wide range of devices, without compromising on accuracy for various vision tasks. For example, on ImageNet-1k, our tiny FasterNet-T0 is $2.8\times$, $3.3\times$, and $2.4\times$ faster than MobileViT-XXS on GPU, CPU, and ARM processors, respectively, while being $2.9\%$ more accurate. Our large FasterNet-L achieves impressive $83.5\%$ top-1 accuracy, on par with the emerging Swin-B, while having $36\%$ higher inference throughput on GPU, as well as saving $37\%$ compute time on CPU. Code is available at \url{https://github.com/JierunChen/FasterNet}.

1. Introduction

Reducing FLOPs does not necessarily reduce latency because low FLOPS, especially from frequent memory access, can limit effective computational speed. The paper proposes PConv and FasterNet to improve this efficiency while maintaining accuracy across devices and tasks.

  • Motivation: Low FLOPs can still yield slow neural networks because depthwise and group convolutions increase memory access and may cause fragmented computation.MobileNets, ShuffleNets, GhostNet, and MicroNet illustrate this trade-off.
  • Motivation: Latency is captured by FLOPs/FLOPS, where FLOPS measures effective computational speed; many existing networks have lower FLOPS than ResNet50.CPU comparisons show that reduced FLOPs alone does not ensure sufficiently low latency.
  • Method: PConv applies regular convolution to only some input channels while leaving the rest untouched, reducing redundant computation and memory access.It uses feature-map redundancy to obtain lower FLOPs than regular convolution and higher FLOPS than depthwise or group convolution.
  • Method: FasterNet builds on PConv as a simple, hardware-friendly network family designed for high speed across GPU, CPU, and ARM processors.The work evaluates PConv and FasterNet across classification, detection, and segmentation tasks.
  • Results: 2.8×, 3.3×, and 2.4× faster than MobileViT-XXS on GPU, CPU, and ARM, FasterNet-T0 was also 2.9% more accurate on ImageNet-1k.FasterNet-L reached 83.5% top-1 accuracy while providing 36% higher GPU throughput and saving 37% CPU compute time.
  • Takeaway: The paper emphasizes optimizing FLOPS alongside FLOPs to obtain genuinely faster neural networks.Its contributions include PConv, FasterNet, and extensive validation across devices and vision tasks.

2. Related Work

Prior efficient vision models reduce complexity through convolutional factorization, compact attention, or MLP-based operators. The paper focuses on DWConv because its speed limitations remain relevant across CNN and hybrid architectures.

  • CNN: CNN efficiency methods commonly use group convolution or depthwise separable convolution to reduce computational complexity.These approaches are central to many fast CNN architectures.
  • ViT, MLP, and variants: ViT-related work improves accuracy-latency trade-offs by simplifying attention, incorporating convolution, or replacing attention with MLP-based operators.Some MLP-based designs evolve toward CNN-like architectures.
  • ViT, MLP, and variants: The paper examines DWConv because attention-based mechanisms generally run slower than convolutional counterparts and DWConv remains popular in hybrid models.The authors also note that attention’s advantage over convolution is unclear or debatable.

3. Design of PConv and FasterNet

The paper introduces PConv, which applies regular convolution to only part of the channels while leaving the rest untouched, reducing redundant computation and memory access. FasterNet builds on PConv and PWConv with a simple hierarchical architecture designed for hardware-friendly, low-latency inference.

  • PConv: PConv applies regular convolution to only a subset of input channels and leaves the remaining channels untouched to reduce redundancy.The untouched channels remain available for a subsequent PWConv, allowing information to flow through all channels.
  • PConv: PConv uses fewer FLOPs and less memory access than regular convolution, with FLOPs reduced to 1/16 and memory access to 1/4 when r = 1/4.Its computational cost and memory access are reduced by processing only cp representative channels.
  • PConv followed by PWConv: A PConv followed by a PWConv forms an effective T-shaped receptive field that concentrates computation on the center position.The center position is most frequently salient among filters in a pre-trained ResNet18, supporting this computation pattern.
  • PConv followed by PWConv: Decomposing the T-shaped convolution into PConv and PWConv further saves FLOPs by exploiting inter-filter redundancy.The decomposition also permits implementation using regular convolution in two steps.
  • FasterNet architecture: FasterNet uses four hierarchical stages with FasterNet blocks, embedding or merging layers, and more blocks assigned to the final two stages.The final two stages consume less memory access and tend to have higher FLOPS, motivating the allocation of more computation there.
  • FasterNet architecture: Each FasterNet block places a PConv before two PWConv layers and uses normalization and activation only after the middle layer.This placement is intended to preserve feature diversity and achieve lower latency; variants differ in depth, width, and activation choice.

4. Experimental Results

Experiments evaluate PConv and FasterNet across processors and vision tasks, showing improved on-device efficiency, accuracy–latency trade-offs, and downstream performance.

  • PConv efficiency: PConv uses 1/16 the FLOPs of a regular Conv and achieves 10.5×, 6.2×, and 22.8× higher FLOPS than DWConv on GPU, CPU, and ARM, respectively.GConv and DWConv reduce FLOPs but suffer sharply lower FLOPS, while channel expansion can increase latency.
  • PConv effectiveness: PConv followed by PWConv achieves the lowest test loss when approximating regular Conv 3 × 3 feature transformations.The experiment uses feature maps from four stages of a pre-trained ResNet50 and splits each dataset into train, validation, and test subsets.
  • ImageNet-1k classification: 2.8×, 3.3×, and 2.4× faster than MobileViT-XXS on GPU, CPU, and ARM, FasterNet-T0 is also 2.9% more accurate on ImageNet-1k.FasterNet establishes the strongest accuracy–latency/throughput trade-offs among the examined classification models.
  • ImageNet-1k classification: 83.5% top-1 accuracy, FasterNet-L is comparable to Swin-B and delivers 36% higher GPU inference throughput plus 37% lower CPU compute time.The comparison also reports 28% higher GPU throughput and 15% lower CPU compute time relative to ConvNeXt-B.
  • Downstream tasks: 38% lower GPU latency than PVT-Large, from 152.2 ms to 93.8 ms, FasterNet-L also achieves +1.1 box AP and +0.4 mask AP under similar FLOPs.FasterNet-S exceeds ResNet50 by +1.9 box AP and +2.4 mask AP while maintaining similar latency.
  • Ablation study: A partial ratio of 1/4 provides higher accuracy, throughput, and lower latency at similar complexity, while normalization and activation choices vary by model size.BatchNorm is selected because it can be merged into adjacent convolutions for faster inference; GELU suits smaller variants, whereas ReLU suits larger ones.

5. Conclusion

The paper identifies low FLOPS, especially from frequent memory access in DWConv, and proposes PConv and FasterNet to improve speed–accuracy trade-offs across devices and tasks.

  • Conclusion: The paper attributes low FLOPS in established neural networks to frequent memory access, particularly in DWConv.PConv is presented as a simple operator designed to reduce redundant computation and memory access.
  • Conclusion: FasterNet achieves state-of-the-art speed–accuracy trade-offs across various devices and vision tasks.The architecture is built upon PConv and is described as simple, fast, and effective.
  • Conclusion: The authors hope PConv and FasterNet will inspire simple yet effective neural-network designs beyond academia.

Appendix

The appendix provides supplementary details for experiments, comparisons, implementations, limitations, and future work.

  • Appendix: The appendix covers experimental settings, full comparison plots, architectural configurations, PConv implementations, related-work comparisons, limitations, and future work.

A. ImageNet-1k experimental settings

ImageNet-1k training and evaluation settings support reproduction of the main comparison results, with stronger regularization and augmentation for larger FasterNet variants.

  • A. ImageNet-1k experimental settings: ImageNet-1k training and evaluation settings are provided for reproducing the main results in Table 3 and Figure 7.
  • A. ImageNet-1k experimental settings: Regularization and augmentation magnitudes increase with model size to alleviate overfitting and improve accuracy.The compared models also use advanced training techniques.

B. Downstream tasks experimental settings

The downstream-task experiments use FasterNet as a backbone for object detection and instance segmentation on COCO2017. Mask R-CNN provides the detector framework, with specified backbone and add-on-layer initialization.

  • COCO2017 is used for object detection and instance segmentation experiments.
  • FasterNet is equipped with the Mask R-CNN detector for these downstream tasks.
  • ImageNet-1k pre-trained weights initialize the backbone, while Xavier initialization is used for add-on layers.

C. Full comparison plots on ImageNet-1k

On ImageNet-1k, the full comparison shows FasterNet maintaining better accuracy-throughput and accuracy-latency trade-offs across GPU, CPU, and ARM processors. The comparison extends the main-paper analysis across a larger latency range.

  • FasterNet consistently achieves better accuracy-throughput trade-offs than other networks on ImageNet-1k.
  • FasterNet consistently achieves better accuracy-latency trade-offs than other networks on ImageNet-1k.
  • The full comparison covers GPU, CPU, and ARM processors across a larger range of latency than the main-paper figure.

E. More comparisons with related work

Additional comparisons position FasterNet and PConv against prior FLOPS-oriented methods, related architectures, and alternative inference paradigms. The paper emphasizes PConv’s distinct operator design and FasterNet’s favorable accuracy-throughput and accuracy-latency trade-offs.

  • Improving FLOPS: RepLKNet increases kernel size and TRT-ViT reorders architecture blocks to improve FLOPS, whereas this paper proposes a new PConv operator.
  • PConv vs. GConv: PConv modifies GConv by applying computation to one group while leaving other groups untouched.
  • PConv vs. GConv: PConv reduces excessive memory access and further reduces intra-filter redundancy beyond GConv’s inter-filter redundancy.
  • FasterNet vs. ConvNeXt: FasterNet differs from ConvNeXt by using PConv, appending PWConv to aggregate information across channels, and removing middle activation functions.
  • Other paradigms for efficient inference: Neural architecture search, pruning, and knowledge distillation are orthogonal to FasterNet’s efficient network design and could be applied for better performance.
  • Other partial/masked convolution works: Other partial or masked convolution methods target invalid patches, self-supervised learning, or image synthesis, while this PConv targets channels for efficient inference.

F. Limitations and future work

The paper identifies two technical limitations and corresponding future directions. PConv requires stride 1 for channel alignment, while FasterNet’s convolutional design may have a limited receptive field.

  • PConv must use stride 1 so its convolved and untouched channels retain aligned spatial resolution.Additional downsampling layers can still reduce spatial resolution in the architecture.
  • FasterNet’s convolutional operators may have a limited receptive field.
  • Future work could enlarge the receptive field and combine FasterNet with other operators to pursue higher accuracy.
Loading 2303.03667v3…