Source-linked AI summary
PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices
Xiaolong Ma, Fu-Ming Guo, Wei Niu, Xue Lin, Jian Tang, Kaisheng Ma, Bin Ren, Yanzhi Wang
TL;DR
PCONV addresses the gap between accurate but irregular fine-grained pruning and hardware-friendly but less accurate structured pruning. It combines sparse convolution patterns with connectivity pruning and deploys them through a compiler-assisted framework. The resulting system achieves real-time inference on large-scale DNNs on mobile devices without accuracy loss.
Problem
Existing pruning methods trade off high accuracy and sparsity against hardware-friendly regularity, while real-time large-DNN inference remains difficult on mobile devices.
Method
PCONV combines intra-convolution sparse convolution patterns and inter-convolution connectivity pruning with a compiler-assisted mobile inference framework.
Results
39.2× speedup is achieved without accuracy drop, and VGG-16 on ImageNet completes in 19.1 ms on an Adreno 640 mobile GPU.
Takeaways & Limitations
PCONV enables real-time inference of representative large-scale DNNs on mobile devices while retaining high accuracy and structured execution properties.
Takeaways & Limitations
The SCP design’s stated upper bound covers most existing effective DNNs, including ResNet-152, but is framed as an upper-bound scope rather than an unrestricted guarantee.
Abstract
from arXiv · showhide
Model compression techniques on Deep Neural Network (DNN) have been widely acknowledged as an effective way to achieve acceleration on a variety of platforms, and DNN weight pruning is a straightforward and effective method. There are currently two mainstreams of pruning methods representing two extremes of pruning regularity: non-structured, fine-grained pruning can achieve high sparsity and accuracy, but is not hardware friendly; structured, coarse-grained pruning exploits hardware-efficient structures in pruning, but suffers from accuracy drop when the pruning rate is high. In this paper, we introduce PCONV, comprising a new sparsity dimension, -- fine-grained pruning patterns inside the coarse-grained structures. PCONV comprises two types of sparsities, Sparse Convolution Patterns (SCP) which is generated from intra-convolution kernel pruning and connectivity sparsity generated from inter-convolution kernel pruning. Essentially, SCP enhances accuracy due to its special vision properties, and connectivity sparsity increases pruning rate while maintaining balanced workload on filter computation. To deploy PCONV, we develop a novel compiler-assisted DNN inference framework and execute PCONV models in real-time without accuracy compromise, which cannot be achieved in prior work. Our experimental results show that, PCONV outperforms three state-of-art end-to-end DNN frameworks, TensorFlow-Lite, TVM, and Alibaba Mobile Neural Network with speedup up to 39.2x, 11.4x, and 6.3x, respectively, with no accuracy loss. Mobile devices can achieve real-time inference on large-scale DNNs.
Introduction
PCONV targets real-time DNN inference on mobile devices by combining a new intermediate sparsity dimension with compiler-assisted execution. It addresses the trade-off between accuracy and hardware regularity in existing pruning approaches.
- Mobile inference for large DNNs remains challenging because representative models require intensive computation and high memory storage.
- Existing non-structured and structured pruning occupy opposite extremes of the accuracy–regularity design space.The paper identifies a missing intermediate sparsity dimension that combines fine-grained accuracy with coarse-grained regularity.
- PCONV combines intra-convolution pattern pruning with inter-convolution connectivity pruning to provide high accuracy and regularity.Pattern pruning creates fixed sparsity patterns within kernels, while connectivity pruning removes selected channel connections and corresponding kernels.
- PCONV integrates its sparsity structures with a compiler-assisted DNN inference framework for mobile execution.The framework is designed to bridge hardware efficiency and the new sparsity dimension.
- 39.2× speedup is achieved without accuracy drop, while PCONV reaches real-time execution for representative large-scale DNNs on mobile devices.The evaluation compares PCONV with TensorFlow Lite, TVM, and MNN across mobile CPUs and GPUs.
Background
DNN compression removes redundant model components to reduce inference cost, but existing pruning methods trade accuracy and hardware efficiency differently. Mobile acceleration frameworks and sparsity-based methods have not fully resolved real-time execution demands.
- DNN model compression removes redundancy from original models with the goal of reducing inference time.
- Non-structured pruning achieves high weight reduction and promising accuracy but leaves irregular weight distributions that complicate compiler and code optimization.
- Structured pruning produces regular, smaller matrices that reduce index overhead and improve CPU/GPU acceleration, but accuracy drops as pruning increases.
- Convolution research connects manually designed vision filters with convolutional neural networks, but prior applications of such filters differ from PCONV in structure and purpose.The cited prior work changes network structure or does not focus on weight pruning and acceleration.
- Existing mobile DNN frameworks and sparsity-acceleration studies do not fully account for model compression or remain far from real-time requirements.
Motivations
The paper is motivated by the need to combine pruning accuracy with hardware regularity, exploit vision-informed sparse patterns, and recover execution efficiency through compiler assistance.
- The missing design point is a pruning granularity that combines non-structured flexibility with structured regularity.
- Sparse convolution patterns are motivated by the potential to enhance image quality through special vision properties.
- The proposed patterns are derived from mathematical vision theory rather than treating pruning solely as redundant-information removal.
- Pattern pruning and connectivity pruning provide complementary structures for the proposed sparsity dimension.Pattern pruning acts within convolution kernels, while connectivity pruning removes selected connections between input and output channels.
- A compiler-assisted inference framework is needed to recover hardware efficiency while retaining the accuracy enabled by fine-grained pruning patterns.
Theory of Sparse Convolution Patterns (SCP)
PCONV introduces Sparse Convolution Patterns (SCPs), which impose mathematically motivated fine-grained structures within convolution kernels, alongside connectivity pruning. The designed patterns approximate vision filters such as Gaussian and Laplacian of Gaussian filters, while experiments report improved accuracy and higher compression without compromise.
- SCP definition: SCPs are binary-valued convolution patterns applied element-wise to kernels, pruning a fixed number of weights while limiting the number of pattern types.Different kernels may use different SCPs, but each filter has the same sparsity ratio.
- Connectivity pruning: Connectivity pruning removes selected connections between input and output channels, eliminating corresponding kernels to increase pruning and inference speed.It complements pattern pruning and can be integrated into the same algorithm and compiler-assisted framework.
- Vision interpretation: Designed SCPs correspond to steerable vision filters, including Gaussian and Laplacian of Gaussian filters used for smoothing, edge detection, and sharpening.The Gaussian filter performs image smoothing, while LoG is associated with image enhancement and edge detection.
- SCP construction: The four SCPs form an Enhanced Laplacian of Gaussian filter through interpolation, inheriting LoG-like denoising and sharpening characteristics.The construction uses interpolation across convolutional layers; the paper sets p = 0.75 and n = 8 for the desired filter.
- Accuracy results: PCONV achieves higher accuracy and compression than prior non-structured pruning, while exceeding structured pruning accuracy at the same compression rate without accuracy compromise.The comparison covers VGG-16 and ResNet-50 on ImageNet.
Compiler-assisted DNN Inference Framework
PCONV inference uses compiler-generated code to exploit the model’s fixed pattern and connectivity structure. The framework extracts layerwise information, reorders filters, and eliminates redundant memory accesses for CPU or GPU execution.
- Compiler-assisted DNN Inference Framework: The compiler converts the DNN into static C++ for CPU execution or OpenCL for GPU execution.Code generation follows layerwise information extraction, filter kernel reorder, and load redundancy elimination.
- Compiler-assisted DNN Inference Framework: Layerwise analysis records pattern distribution, pattern order, and channel connectivity for later compiler optimizations.These details describe kernel patterns and connections between input and output channels.
- Compiler-assisted DNN Inference Framework: Filter kernel reorder groups similar kernels across filters and identical kernels within filters to improve inter-thread and intra-thread parallelism.PCONV patterns and connections are fixed before deployment, enabling this reordering.
- Compiler-assisted DNN Inference Framework: The framework generates pattern-specific data-access code to directly load valid inputs associated with non-zero kernel elements.Structured pattern distribution reduces data-access-code calling frequency and memory overhead.
Experimental Results
Experiments evaluate PCONV on mobile CPU and GPU across VGG-16, ResNet-50, and MobileNet-v2, comparing optimized baselines and examining throughput and pattern-count effects. PCONV accelerates inference substantially, reaches real-time VGG-16 ImageNet execution, and maintains favorable accuracy-performance behavior.
- Experimental Results: PCONV is evaluated against TFLite, TVM, and MNN on a Samsung Galaxy S10 with Snapdragon 855 CPU and GPU.The evaluated models are VGG-16, ResNet-50, and MobileNet-v2, with convolutional layers emphasized because they exceed 95% of DNN inference time.
- Experimental Results: 39.2× maximum CPU speedup over TFLite and 11.4× maximum GPU speedup over TVM are reported for PCONV.The reported ranges are 9.4×–39.2× over TFLite on CPU and 2.5×–11.4× over TVM on GPU.
- Experimental Results: 19.1ms per VGG-16 ImageNet image on GPU corresponds to 52.4 frames/sec and meets the stated 30-frames/sec real-time requirement.The comparison baselines use fully optimized configurations, including Winograd optimization.
- Experimental Results: PCONV outperforms MNN in layerwise GFLOPS throughput on both mobile CPU and GPU.The comparison uses nine unique VGG-16 convolutional layer sizes; four repeated-size layers are omitted.
- Experimental Results: Pattern-count increases are not necessarily associated with accuracy loss, but execution performance drops quickly, especially on ImageNet.The authors characterize their designed SCPs as providing ideal performance with negligible accuracy loss.
Conclusion
PCONV combines flexible and structured sparsity in a compiler-assisted framework for accelerating large-scale DNN inference on mobile devices. Its design targets real-time performance while preserving accuracy.
- PCONV combines non-structured pruning’s flexibility with structured pruning’s hardware friendliness.The resulting structure supports optimized memory access, balanced workload, and computation parallelism.
- A compiler-assisted DNN inference framework exploits PCONV’s structural characteristics to achieve high inference speed on large-scale DNNs.
- PCONV enables real-time DNN inference on mobile devices without accuracy compromise.