Recent Advances in Convolutional Neural Network Acceleration

Qianru Zhang; Meng Zhang; Tinghuan Chen; Zhifei Sun; Yuzhe Ma; Bei Yu

Source-linked AI summary

Recent Advances in Convolutional Neural Network Acceleration

Qianru Zhang, Meng Zhang, Tinghuan Chen, Zhifei Sun, Yuzhe Ma, Bei Yu

arXiv:1807.08596v1Jul 23, 2018cs.LGstat.ML

TL;DR

Deeper CNNs and higher-dimensional inputs make training and inference computationally expensive, motivating faster implementation. The paper reviews acceleration methods across structure, algorithm, and implementation levels, and concludes that these methods remain open to further exploration while evaluation lacks a generalized comparison system.

Problem
Deeper CNN architectures and higher-dimensional data make training and inference computationally expensive, limiting implementation speed.
Method
The paper reviews recent CNN acceleration research and organizes it into structure-level, algorithm-level, and implementation-level methods.
Results
The review identifies acceleration methods spanning CNN compression, algorithm optimization, and hardware-oriented improvement, with substantial exploration space remaining at each level.
Takeaways & Limitations
CNN acceleration is examined as an interconnected combination of structure design, training algorithms, and hardware implementation.
Takeaways & Limitations
Acceleration performance remains difficult to compare because methods use case-by-case datasets and different evaluation criteria, with no generalized evaluation system.

Abstract

from arXiv · show

In recent years, convolutional neural networks (CNNs) have shown great performance in various fields such as image classification, pattern recognition, and multi-media compression. Two of the feature properties, local connectivity and weight sharing, can reduce the number of parameters and increase processing speed during training and inference. However, as the dimension of data becomes higher and the CNN architecture becomes more complicated, the end-to-end approach or the combined manner of CNN is computationally intensive, which becomes limitation to CNN's further implementation. Therefore, it is necessary and urgent to implement CNN in a faster way. In this paper, we first summarize the acceleration methods that contribute to but not limited to CNN by reviewing a broad variety of research papers. We propose a taxonomy in terms of three levels, i.e.~structure level, algorithm level, and implementation level, for acceleration methods. We also analyze the acceleration methods in terms of CNN architecture compression, algorithm optimization, and hardware-based improvement. At last, we give a discussion on different perspectives of these acceleration and optimization methods within each level. The discussion shows that the methods in each level still have large exploration space. By incorporating such a wide range of disciplines, we expect to provide a comprehensive reference for researchers who are interested in CNN acceleration.

1. Introduction

CNNs deliver strong performance across applications, but deeper architectures and higher-dimensional data make training and inference computationally expensive. The paper reviews recent acceleration work across structure, algorithm, and implementation levels, emphasizing their interdependence.

CNN applications: CNNs have demonstrated strong performance in computer vision and other fields by processing array inputs with convolution over local fields.The introduction also mentions applications including speech recognition.
Motivation: Deeper CNN architectures and higher-resolution inputs increase computational cost, slowing training and inference and limiting implementation.The paper links this challenge to the need for more urgent acceleration and optimization research.
Existing accelerators: Recent hardware accelerators include Google’s second-generation TPU, NVIDIA’s NVDLA, and Intel’s Nervana Neural Network Processor.These systems target deep-learning computation, including convolutions, matrix multiplications, and data movement.
Paper scope: The paper reviews CNN acceleration methods at structure, algorithm, and implementation levels rather than focusing only on neural-network applications or historical development.Its review emphasizes recent research where hardware limitations make efficiency a priority.
Paper scope: The three acceleration levels are treated together because hardware and algorithms are described as interwoven and highly dependent.The paper presents this integration as a distinction from reviews that do not incorporate hardware into algorithm discussions.

2. Convolutional Neural Network

CNNs combine convolution, subsampling, fully connected, and output layers to extract and classify features. Their local connectivity, weight sharing, pooling, and depth support efficient and translation-aware representation, while activation choices affect training behavior.

CNN structure: Modern CNNs range from the 7-layer LeNet-5 to deeper models such as AlexNet, GoogLeNet, VGG-16, and ResNet.LeNet-5 uses convolutional, subsampling, convolutional, subsampling, convolutional, fully-connected, and output layers.
CNN structure: CNNs typically repeat convolutional and subsampling layers to increase depth, followed by fully connected layers that map extracted features to labels.The common structural features are weight sharing, local connection, pooling, and multiple layers.
Convolutional layer: Convolution applies kernels to small local input patches, adds a bias, and passes the resulting feature map through an activation function.The cited notation identifies the input, kernel, bias, convolution output, and activation function.
Activation functions: Sigmoid and Tanh saturate at extreme inputs, producing near-zero gradients that hinder backpropagation and make training convergence difficult.ReLU, Leaky ReLU, PReLU, RReLU, and ELU are identified as non-saturated alternatives.
CNN properties: Weight sharing reduces parameters and sensitivity to location, while local convolution preserves input topology useful for recognizing patterns.These properties are presented as core advantages of CNN structure.
Subsampling layer: Pooling reduces feature-map resolution, parameters, and computation, while also improving robustness to distortions and individual-neuron errors.Typical methods include maximum and average pooling, alongside alternatives such as stochastic and spatial-pyramid pooling.

3. Acceleration Method Taxonomy

The paper organizes CNN acceleration around the progression from structure design to training algorithms and hardware implementation. Its taxonomy covers redundancy reduction, algorithmic efficiency, and hardware-aware acceleration.

Taxonomy rationale: The taxonomy follows CNN development from designing the structure, through training, to implementing the network on hardware.The paper treats these levels as connected stages rather than isolated categories.
Structure level: Structure-level methods exploit redundancy in weights and representation bits through decomposition, pruning, block-circulant projection, knowledge distillation, and fixed-point representation.These methods target CNN compression before algorithm and hardware considerations.
Algorithm level: Algorithm-level acceleration addresses training convergence and convolution complexity using back-propagation improvements and efficient convolution methods such as FFT.The taxonomy identifies convergence speed with stability as a training-optimization criterion.
Implementation level: Implementation-level methods combine CNN characteristics with hardware-specific properties of platforms such as FPGA and GPU.The taxonomy frames hardware exploration as the final stage after structure and training choices.

4. Structure Level

Structure-level acceleration reduces redundancy in CNN weights and representations through decomposition, pruning, structured matrices, knowledge distillation, and low-bit representations. These methods target smaller models, lower computation, faster inference, and reduced storage, though approximation can trade accuracy for efficiency.

Structure Level: Structure-level acceleration addresses redundancy in both weights and their numerical representation.The survey frames structure-level methods around reducing network redundancy before training and hardware implementation.
Layer Decomposition: Low-rank decomposition factorizes weight matrices or kernels to reduce parameters and computation.Reported results include 2∼2.5× speedup with less than 1% classification-performance drop for a single convolutional layer, and 8.5× CPU speedup with 1% error increase using CP-decomposition.
Network Pruning: Pruning removes unimportant weights, connections, filters, channels, or feature maps to reduce model size and inference cost.Structured pruning methods address hardware-unfriendly unstructured sparsity; examples report 5.1× CPU and 3.1× GPU convolutional speedups.
Block-circulant Projection: Block-circulant projection represents matrices with vectors and reduces storage requirements while preserving efficient computation.The approach applies circulant structures to fully connected and convolutional layers, including non-square matrices.
Knowledge Distillation: Knowledge distillation transfers information from a large complex network or ensemble to a compact neural network for model compression.The compact model is trained using data labeled or generated by the complex model and can approximate its functions while reducing overfitting in some settings.
Redundancy in Representations: One-bit and other low-bit representations replace conventional arithmetic with binary or quantized operations to accelerate CNNs.XNOR-Nets reports 58× convolutional-operation speedup, while binary-kernel decomposition reduces per-image operations by 47.7%.

5. Algorithm Level

The paper reviews algorithm-level CNN acceleration through distributed and improved gradient-based training, hybrid optimization, and efficient convolution algorithms. These methods target convergence and convolution costs, while exposing trade-offs such as communication overhead and frequency-domain transformation cost.

Gradient Descent Optimization: Distributed gradient descent reduces hardware workload through replica-level data parallelism or computation partitioning across cores.Replica training averages gradients from different batches, whereas each replica can distribute neuron subsets across cores.
Gradient Descent Optimization: Back-propagation variants combine gradient descent with cuckoo search or ant colony algorithms to increase search speed, reduce computational cost, or improve convergence stability.The cited hybrid methods pursue different optimization benefits rather than a single uniform objective.
Gradient Descent Optimization: Gradient-based training research addresses vanishing or exploding gradients through modified error functions, learning rates, momentum, and partial gradients.Adaptive learning rates depend on current and past parameters, while momentum is especially useful for oscillating optimization landscapes.
Feed-forward Efficient Convolution: Efficient convolution methods include im2col, Winograd filtering, and FFT, which reformulate convolution to reduce computation or improve execution efficiency.Im2col linearizes inputs for vectorized computation; Winograd reduces multiplications by increasing additions, while FFT moves multiplication into the frequency domain.
Feed-forward Efficient Convolution: FFT-based CNN acceleration performs well for large feature maps but incurs time and memory costs from transformations, padding, and frequency-domain processing.Applying FFT therefore requires balancing computation savings against transformation overhead.

6. Implementation Level

Implementation-level acceleration exploits GPU, FPGA, ASIC, and emerging-device characteristics to improve CNN training or inference. The design space remains broad because architectures must balance computation, memory movement, flexibility, and bandwidth constraints.

GPU: GPU clusters accelerate networks with over one billion parameters through parallel execution and distributed stochastic gradient descent.Research also explores model-data parallelism, data parallelism, and inter-cluster communication.
FPGA: FPGA acceleration benefits from reconfiguration and hierarchical storage, enabling flexible scheduling, complex data access, and reduced energy consumption.Reported accelerators achieve 11× speedup on Virtex 6 and 17.42× faster execution than CPU on AlexNet.
ASIC: ASIC designs improve CNN efficiency through low-precision computation, larger local memory, data locality, and reduced movement between processing elements and memory.A 64-chip system reports 450× speedup over a GPU with 150× energy reduction, while Chain-NN reaches 806.4 GOPS peak throughput on AlexNet.
ASIC: Sparse-network ASICs use specialized indexing and reconfigurable dataflows to handle irregular sparsity and improve compute-resource utilization.Cambricon-X reaches 544 GOP/s in 6.38mm2, and MAERI reports 6.9× speedup at 50% sparsity.
Frontier New Devices: Emerging devices such as resistive processing units and memristor crossbars combine parameter storage with computation to target large acceleration and energy benefits.The cited RPU reaches 84000 GigaOps/s/W, while a memristor DPE reports a 1000× to 10,000× speed-efficiency product over a digital ASIC.
Design Space: General-purpose CNN hardware remains difficult to design because computation resources and memory bandwidth constrain a broadly applicable architecture.The paper characterizes hardware implementation as an open design space rather than a settled architecture.

7. Discussion

The discussion compares acceleration methods within their respective levels because heterogeneous benchmarks make cross-level evaluation difficult. It also identifies method-specific trade-offs in representation, optimization, convolution, and hardware acceleration.

Evaluation Perspective: Different datasets, models, benchmarks, and criteria make direct comparison between acceleration methods from different levels difficult.The paper therefore constrains comparison and analysis to methods within each level.
Structure Level: Layer decomposition and pruning often require pretrained models, and large networks may need weeks or months of training before these methods can be applied.Pruning sparse weights or connections also requires fine-tuning after pretraining.
Structure Level: Layer decomposition and pruning are frequently specialized for particular layers, including softmax, fully connected, and convolutional layers.The cited examples target different layer types rather than providing one universally applied reduction method.
Structure Level: Representation reduction can shrink networks, but hardware support determines whether the theoretical reduction produces practical speedup.BinaryConnect training on a GPU takes more time, while dedicated hardware could theoretically reduce time complexity by 60%.
Structure Level: Low-bit representation results depend on model and dataset, with small CIFAR10 experiments usually reporting less than 5% error-rate increase but larger ImageNet settings remaining difficult.Performance on large-scale ImageNet is not generally matched to that achieved on small, low-resolution datasets.
Algorithm Level: Second-order gradient methods converge faster than first-order methods but are often impractical for deep, large networks because using second-order information is complex.Approximate Hessian matrices are therefore emphasized as a simplification.
Algorithm Level: Batch, mini-batch, and online gradient methods trade off convergence behavior, redundant computation, memory requirements, and update frequency.Mini-batch methods reduce redundant computation, while online methods support lightweight memory and speed requirements.
Algorithm Level: Asynchronous distributed training improves large-cluster efficiency, but suboptimal worker communication can cause parameter divergence.Communication remains a central consideration in distributed optimization.

8. Conclusion

The paper concludes that CNN acceleration spans structure, algorithm, and implementation levels, each containing substantial unexplored design space. It also emphasizes that inconsistent benchmarks and criteria make cross-method performance evaluation challenging.

Conclusion: The survey organizes CNN acceleration into structure-level compression, algorithm-level optimization, and implementation-level hardware exploration.It links redundancy reduction, convolution optimization, and hardware-specific design considerations within this three-level view.
Conclusion: CNN acceleration remains necessary because deeper architectures and larger inputs increase training and inference time and computational expense.The conclusion connects CNN complexity with the need for faster implementation.
Conclusion: The methods at each acceleration level retain substantial exploration space for future research.This conclusion is stated across the survey’s structure, algorithm, and implementation taxonomy.
Conclusion: No generalized evaluation system currently compares acceleration performance across methods and levels using consistent benchmarks and criteria.Case-by-case datasets and level-specific criteria make performance evaluation challenging.