PaperlayerView exact revision on arXiv

Source-linked AI summary

HetConv: Heterogeneous Kernel-Based Convolutions for Deep CNNs

Pravendra Singh, Vinay Kumar Verma, Piyush Rai, Vinay P. Namboodiri

arXiv:1903.04120v2cs.CVcs.AIcs.LG

TL;DR

As CNNs grow deeper and more complex, computation rises, while designing efficient architectures and pruning models can be costly. HetConv replaces standard filters with heterogeneous kernels, reporting 3X to 8X FLOPs reduction while maintaining accuracy.

  • Problem

    Deeper, more complex CNNs require more computation, while finding low-FLOP filter combinations for new architectures takes substantial work and pruning can be costly.

    Source: §1 · p. 1

  • Method

    HetConv replaces standard convolution filters with heterogeneous-kernel filters that can be plugged into existing architectures.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    3X to 8X FLOPs reduction is reported without sacrificing accuracy, and HetConv models have fewer FLOPs with better accuracy than existing convolutions.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    HetConv reduces FLOPs without replacing a convolutional layer with multiple layers, so the authors report zero added latency.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present a novel deep learning architecture in which the convolution operation leverages heterogeneous kernels. The proposed HetConv (Heterogeneous Kernel-Based Convolution) reduces the computation (FLOPs) and the number of parameters as compared to standard convolution operation while still maintaining representational efficiency. To show the effectiveness of our proposed convolution, we present extensive experimental results on the standard convolutional neural network (CNN) architectures such as VGG \cite{vgg2014very} and ResNet \cite{resnet}. We find that after replacing the standard convolutional filters in these architectures with our proposed HetConv filters, we achieve 3X to 8X FLOPs based improvement in speed while still maintaining (and sometimes improving) the accuracy. We also compare our proposed convolutions with group/depth wise convolutions and show that it achieves more FLOPs reduction with significantly higher accuracy.

1. Introduction

HetConv uses heterogeneous kernels in existing CNN architectures to reduce FLOPs while maintaining accuracy, with reported 3X to 8X reductions and zero added latency. The approach is presented as an alternative to architecture search and model compression.

  • Designing a new architecture requires substantial work to find an effective low-FLOP filter combination, while pruning requires a pretrained model and costly training and fine-tuning.The introduction also characterizes pruning as costly and tricky.Source: §1 · p. 1
  • Efficient-convolution approaches may trade accuracy for lower FLOPs or keep FLOPs unchanged while modifying existing architectures.The authors position HetConv as reducing FLOPs without compromising accuracy.Source: §1 · pp. 1–2
  • HetConv replaces standard convolution filters with a new heterogeneous convolution operation that can be plugged into existing architectures.The stated goal is to reduce FLOPs without compromising accuracy, rather than design a new architecture.Source: §1 · p. 2
  • HetConv has the same latency as the original architecture, unlike approaches that replace a convolutional layer with multiple layers.The introduction describes the latency of those other approaches as greater than zero.Source: §1 · p. 2
  • 3X to 8X FLOPs reduction is reported without sacrificing accuracy in existing architectures including ResNet and VGG-16.The authors also report results significantly better than existing pruning approaches.Source: §1 · p. 2

2. Related Work

Related work seeks efficiency through convolutional filters and model compression. The paper distinguishes its heterogeneous-kernel approach from homogeneous filters and pruning-based compression.

2. Related Work

  • Deep, complex CNNs can require billions of FLOPs and high-end machines, motivating interest in efficient models.The related-work section divides efficiency research into two parts.Source: §2 · p. 2

2.1. Efficient Convolutional Filter

  • Depthwise, pointwise, and groupwise filters support efficient architectures, but HetConv combines different kernel sizes within a filter rather than using kernels of one type.The related-work discussion notes that MobileNet’s FLOPs reduction comes with an accuracy drop relative to state-of-the-art models.Source: §2.1 · pp. 2–3

2.2. Model Compression

  • Filter pruning ranks and removes filters from a pretrained model, then retrains to recover accuracy, but iterative pruning and training are costly and greater FLOP pruning can sharply reduce accuracy.Source: §2.2 · p. 3

3. Proposed Method

HetConv combines 3 × 3 and 1 × 1 kernels within a filter to reduce computation while retaining spatial correlation on some channels. Its design avoids the added sequential latency of depthwise- or groupwise-plus-pointwise replacements, with P controlling the trade-off between FLOPs and accuracy.

3. Proposed Method

  • HetConv assigns M/P kernels to 3 × 3 and the remaining kernels to 1 × 1, reducing computation while retaining spatial correlation on some channels.P=1 recovers the standard convolutional filter.Source: Figure 1 · p. 3, §3 · pp. 4–5

3.1. Comparision with DepthWise followed by PointWise Convolution

  • HetConv uses fewer FLOPs and zero latency than DWC+PWC in the P=M extreme case, and is reported to achieve significantly better accuracy than MobileNet.The paper identifies MobileNet as using DWC+PWC.Source: §3.1 · p. 5

3.2. Comparision with GroupWise followed by PointWise Convolution

  • HetConv uses fewer FLOPs and zero latency than GWC+PWC in the P=G comparison.Source: §3.2 · p. 5

3.3. Running Latency

  • HetConv reduces FLOPs without replacing a layer with sequential layers, preserving zero added latency.The paper contrasts this with efficient convolutions that add latency by using sequential layers.Source: Figure 2 · p. 4, §3.3 · pp. 5–6

3.4. Speedup over standard convolution for different values of P

  • Speedup over standard convolution increases with P, which can be adjusted to trade off FLOPs and accuracy.Decreasing P makes the resulting convolution closer to standard convolution.Source: §3.1 · p. 5, §3.4 · p. 6, Figure 5 · p. 6

4. Experiments and Results

Experiments replace standard filters in large- and small-scale CNN benchmarks with HetConv; the notation section describes model and convolution variants.

4. Experiments and Results

  • Experiments cover ImageNet with ResNet-34, ResNet-50, and VGG-16, and CIFAR-10 with VGG-16, ResNet-56, and MobileNet.Squeeze-and-Excitation uses reduction ratio 8 in all experiments.Source: §4 · p. 6

4.1. Notations

  • The notation distinguishes architectures with part value P, groupwise convolution with group size β followed by pointwise convolution, and depthwise convolution followed by pointwise convolution.It also defines PC as P equal to the number of input channels.Source: §4.1 · p. 6
  • The notation also includes an SE variant with reduction ratio 8.Source: §4.1 · p. 6

4.2. VGG-16 on CIFAR-10

On CIFAR-10 VGG-16, HetConv reduces FLOPs with little reported accuracy loss and compares favorably with compression, groupwise-pointwise, and depthwise-pointwise alternatives.

4.2. VGG-16 on CIFAR-10

  • The CIFAR-10 VGG-16 setup replaces 12 standard convolutional layers with HetConv while leaving the initial convolution unchanged.The experiment uses the same number of filters as before.Source: §4.2 · p. 6
  • As P increases, FLOPs decrease without significant accuracy loss; adding SE initially raises accuracy but later degrades performance through over-fitting.Source: §4.2 · p. 6

4.2.1 Comparison with groupwise followed by pointwise convolution

  • 93.93% accuracy for VGG-16 P4 exceeds GWC4 PWC’s 92.76%, with lesser FLOPs for HetConv.Table 1 reports VGG-16 CIFAR-10 results across setups.Source: §4.2.1 · p. 6, Table 1 · p. 7

4.2.2 Comparison with depthwise followed by pointwise convolution

  • 93.42% accuracy for VGG-16 P64 exceeds DWC PWC’s 91.27%, with comparable FLOPs; PC and PC SE also outperform DWC PWC.The compared DWC-PWC setup replaces standard convolutional layers with two layers.Source: §4.2.2 · pp. 6–7, Table 1 · p. 7

4.2.3 Comparison with FLOPs compression methods

  • ∼85% FLOPs reduction with no accuracy loss is reported for VGG-16 P32 and P64, while FLOPs-compression methods lose more than 1% accuracy.Table 2 covers VGG-16 on CIFAR-10 and compares HetConv with model-compression methods.Source: Table 2 · p. 7, §4.2.3 · p. 7

4.3. ResNet-56 on CIFAR-10

On CIFAR-10 ResNet-56, HetConv replaces all standard convolutional layers except the initial layer, and selected configurations compare favorably with model-compression methods.

4.3. ResNet-56 on CIFAR-10

  • The model has 126.01M total FLOPs, with each convolutional layer in its three 16-32-64 stages using 2.36M FLOPs.Source: §4.3 · p. 7
  • All standard convolutional layers except the initial layer are replaced by HetConv while keeping the filter count unchanged.Source: §4.3 · p. 7
  • ResNet-56 P2 and P4 report significantly better accuracy than state-of-art model-compression methods alongside higher FLOPs reduction.Adding SE did not perform as expected because of over-fitting.Source: §4.3 · p. 7

4.4. MobileNet on CIFAR-10

On CIFAR-10 MobileNet, HetConv replaces depthwise-pointwise layer pairs with single layers; reported configurations have better accuracy at almost similar FLOPs.

4.4. MobileNet on CIFAR-10

  • The experiment replaces all MobileNet convolutional layers except the initial layer with HetConv, retaining the prior filter count.Source: §4.4 · p. 7
  • MobileNet P32 and P32 SE report significantly better accuracy, close to 1%, than MobileNet at almost similar FLOPs.Table 4 reports MobileNet CIFAR-10 results across setups.Source: Table 4 · p. 8, §4.4 · p. 8
  • Replacing a depthwise-pointwise pair with one HetConv layer reduces latency from one to zero.Source: §4.4 · p. 8

4.5. VGG-16 on ImageNet

On ImageNet, VGG-16 P4 achieves greater FLOP compression than Channel-Pruning without loss in accuracy. The experiment replaces all convolutional layers except the initial layer while keeping the number of filters unchanged.

4.5. VGG-16 on ImageNet

  • 65.8% FLOP compression for VGG-16 P4 versus 50.0% for Channel-Pruning, with no loss in accuracy.Table 5 presents VGG-16 results on ImageNet and compares HetConv with pruning approaches.Source: Table 5 · p. 8, §4.5 · p. 8
  • All VGG-16 convolutional layers except the initial layer are replaced with HetConv layers, keeping the number of filters unchanged.Source: §4.5 · p. 8
  • The experiment evaluates VGG-16 on the large-scale ImageNet dataset.Source: §4.5 · p. 8

4.6. ResNet-34 on ImageNet

On ImageNet, ResNet-34 P4 achieves greater FLOP compression than NISP, with significantly better accuracy. The experiment replaces all convolutional layers except the initial layer with HetConv.

4.6. ResNet-34 on ImageNet

  • 64.48% FLOP compression for ResNet-34 P4 versus 43.76% for NISP, with significantly better accuracy.Table 6 presents ResNet-34 results on ImageNet across different setups.Source: Table 6 · p. 8, §4.6 · p. 8
  • All ResNet-34 convolutional layers except the initial layer are replaced with HetConv layers.Source: §4.6 · p. 8
  • The experiment evaluates ResNet-34 on the large-scale ImageNet dataset.Source: §4.6 · p. 8

4.7. ResNet-50 on ImageNet

On ImageNet, ResNet-50 P4 has no loss in accuracy, while FLOP-pruning approaches show a heavy top-1 accuracy drop. HetConv is trained from scratch, unlike pruning approaches that require a pretrained model and iterative pruning and fine-tuning.

4.7. ResNet-50 on ImageNet

  • ResNet-50 P4 has no loss in accuracy, while FLOP-pruning approaches show a heavy top-1 accuracy drop.Table 7 compares ResNet-50 results on ImageNet with FLOP-pruning approaches.Source: Table 7 · p. 8, §4.7 · p. 8
  • The ResNet-50 experiment replaces standard convolutions with HetConv and trains for 90 epochs, with batch size 256 and weight decay 1e-4.The initial learning rate is 0.1 and is multiplied by 0.1 every 30 epochs.Source: §4.7 · p. 8
  • HetConv is trained from scratch, while pruning approaches require a pretrained model and iterative pruning and fine-tuning.The paper describes the pruning process as very time-consuming.Source: §4.7 · p. 8

5. Conclusion

The conclusion presents HetConv as a heterogeneous-kernel convolution compared with standard, depthwise, groupwise, and pointwise convolutions across VGG-16, ResNet, and MobileNet. It reports lower FLOPs with better accuracy than existing convolutions and FLOPs-compression methods, and says the approach has zero latency because it does not increase the number of layers.

5. Conclusion

  • Across VGG-16, ResNet, and MobileNet, HetConv is reported as more efficient than existing convolutions, with lower FLOPs and better accuracy.The comparison includes depthwise, groupwise, pointwise, and standard convolutions.Source: §5 · p. 8
  • HetConv is reported to have zero latency because replacing a layer does not increase the number of layers.Source: §5 · p. 8
  • The conclusion reports that HetConv-based models produce better results than FLOPs-compression methods and proposes using the convolution to design more efficient architectures.Source: §5 · p. 8
Loading 1903.04120v2…