Source-linked AI summary
PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning
Wei Niu, Xiaolong Ma, Sheng Lin, Shihao Wang, Xuehai Qian, Xue Lin, Yanzhi Wang, Bin Ren
TL;DR
Mobile DNN inference must meet real-time requirements despite constrained processors and a trade-off between pruning accuracy and hardware efficiency. PatDNN combines pattern-based pruning with compiler-optimized execution, achieving up to 44.5× speedup without accuracy compromise and real-time execution of large-scale DNNs on mobile devices.
Problem
Real-time mobile DNN inference requires high accuracy despite limited embedded resources and the conflicting hardware properties of non-structured and structured pruning.
Method
PatDNN uses fine-grained pruning patterns within coarse-grained structures, solved with an extended ADMM framework and paired with compiler-based execution optimizations.
Results
PatDNN achieves up to 44.5× speedup without accuracy compromise and real-time execution of representative large-scale DNNs on mobile devices.
Takeaways & Limitations
Pattern-based pruning and compiler optimization provide an end-to-end route to combining high DNN accuracy with hardware-efficient mobile execution.
Takeaways & Limitations
Structured filter/channel pruning can still produce a 1.0% Top-5 accuracy degradation at 3.8× weight reduction on VGG-16 CONV layers using ImageNet.
Abstract
from arXiv · showhide
With the emergence of a spectrum of high-end mobile devices, many applications that formerly required desktop-level computation capability are being transferred to these devices. However, executing the inference of Deep Neural Networks (DNNs) is still challenging considering high computation and storage demands, specifically, if real-time performance with high accuracy is needed. Weight pruning of DNNs is proposed, but existing schemes represent two extremes in the design space: non-structured pruning is fine-grained, accurate, but not hardware friendly; structured pruning is coarse-grained, hardware-efficient, but with higher accuracy loss. In this paper, we introduce a new dimension, fine-grained pruning patterns inside the coarse-grained structures, revealing a previously unknown point in design space. With the higher accuracy enabled by fine-grained pruning patterns, the unique insight is to use the compiler to re-gain and guarantee high hardware efficiency. In other words, our method achieves the best of both worlds, and is desirable across theory/algorithm, compiler, and hardware levels. The proposed PatDNN is an end-to-end framework to efficiently execute DNN on mobile devices with the help of a novel model compression technique (pattern-based pruning based on extended ADMM solution framework) and a set of thorough architecture-aware compiler- and code generation-based optimizations (filter kernel reordering, compressed weight storage, register load redundancy elimination, and parameter auto-tuning). Evaluation results demonstrate that PatDNN outperforms three state-of-the-art end-to-end DNN frameworks, TensorFlow Lite, TVM, and Alibaba Mobile Neural Network with speedup up to 44.5x, 11.4x, and 7.1x, respectively, with no accuracy compromise. Real-time inference of representative large-scale DNNs (e.g., VGG-16, ResNet-50) can be achieved using mobile devices.
1 Introduction
Mobile DNN inference needs real-time performance despite constrained embedded processors and a trade-off between pruning accuracy and hardware efficiency. PatDNN addresses this gap with pattern-based pruning and compiler optimizations, achieving efficient, accurate mobile execution.
- Motivation: Real-time DNN inference is challenging on mobile devices because embedded processors have limited computing resources.TVM requires 242ms for VGG-16 inference on an Adreno 640 GPU, while TensorFlow Lite does not support it in the cited setting.
- Motivation: Non-structured pruning offers fine-grained compression but limits hardware parallelism, whereas structured pruning is hardware-friendly but causes greater accuracy loss.Compressed-weight indices hinder throughput in non-structured pruning; filter and channel pruning provide regular models but reduce accuracy.
- Approach: PatDNN introduces fine-grained pruning patterns within coarse-grained structures to explore a more flexible accuracy–hardware-efficiency trade-off.Its key challenge is recovering the hardware efficiency lost through fine-grained patterns.
- Approach: PatDNN combines pattern-based training with extended ADMM and compiler-based code generation, including layerwise representation, filter reordering, load-redundancy elimination, and parameter tuning.The framework targets both mobile CPUs and GPUs.
- Contributions: PatDNN’s pattern-based pruning and compiler optimizations are designed to combine the accuracy benefits of non-structured pruning with the efficiency of structured pruning.The paper identifies optimization opportunities enabled by its pattern-based design that prior execution frameworks do not provide.
- Evaluation: 44.5× speedup was achieved without accuracy compromise across comparisons with TensorFlow Lite, TVM, and Alibaba Mobile Neural Networks.The evaluation uses VGG-16, ResNet-50, and MobileNet-V2 on ImageNet and CIFAR-10; PatDNN also reports 18.9ms VGG-16 inference on an Adreno 640 GPU.
2 Background and Motivation
Mobile DNN inference remains difficult because embedded processors have limited resources, while existing pruning methods trade accuracy against hardware efficiency. PatDNN introduces fine-grained patterns within coarse-grained structures and uses compiler optimizations to recover execution efficiency.
- Pruning methods: Non-structured pruning removes arbitrary weights, achieving high pruning rates but causing irregular computation, memory access, and hardware execution.Irregular sparse kernels require control flow, can cause thread divergence and load imbalance, and often reduce data locality and cache performance.
- Pruning methods: Structured pruning removes whole filters or channels, producing regular models that are hardware friendly but can incur greater accuracy loss.Filter and channel pruning are equivalent across adjacent layers and are compatible with the Winograd algorithm.
- ADMM-based compression: ADMM-NN achieves strong non-structured compression, but its structured extension causes 1.0% Top-5 accuracy degradation for 3.8× weight reduction on VGG-16 CONV layers.The reported evaluation uses ImageNet, and the accuracy loss is described as non-negligible for many applications.
- PatDNN motivation: PatDNN adds fine-grained pruning patterns inside coarse-grained structures to explore a previously unavailable accuracy–hardware-efficiency trade-off.Its compiler optimizations are intended to close the performance gap between fully structured pruning and pattern-based semi-structured pruning.
3 Overview of PatDNN
PatDNN combines fine-grained pruning patterns with compiler optimizations to balance accuracy and hardware efficiency. Its framework uses kernel-pattern and connectivity pruning alongside architecture-aware execution code generation.
- Pattern-based pruning: Pattern-based pruning seeks both flexibility and regularity, enabling accuracy-oriented pruning while supporting parallelism and compiler optimization.Flexibility supports instruction-level and thread-level parallelism; regularity supports efficient hardware execution and redundant-load elimination.
- Pattern-based pruning: Kernel pattern pruning reserves a fixed number of non-zero weights per kernel, selected from predefined patterns.For example, 4-entry pruning reserves four non-zero weights from an original 3 × 3 kernel, while allowing each kernel to choose among predefined patterns.
- Pattern-based pruning: Connectivity pruning removes selected connections between input and output channels, eliminating corresponding kernels to increase pruning beyond kernel pattern pruning.The compiler can group removed kernels and associated computations through reordering without affecting other computations.
- Overview of PatDNN Acceleration Framework: Compiler optimizations recover hardware efficiency from fine-grained patterns, allowing PatDNN to combine benefits associated with non-structured and structured pruning.The framework uses pattern-aware execution mechanisms, including filter-kernel reordering, compressed weight storage, and redundant-load elimination.
- Overview of PatDNN Acceleration Framework: PatDNN integrates pattern-based training with execution code generation, applying graph and layerwise optimizations to pruned DNN models.The training stage uses kernel pattern and connectivity pruning with an extended ADMM framework; code generation applies multiple optimizations based on the patterns.
4 PatDNN Training w/ Pattern-based Pruning
PatDNN trains compressed DNNs by combining kernel pattern pruning with connectivity pruning in an extended ADMM framework. The method balances pattern-set flexibility, compression, and accuracy, with 8-pattern configurations achieving substantial weight reduction without Top-5 accuracy loss.
- Pattern-based training: PatDNN selects candidate kernel patterns, assigns patterns or prunes whole kernels, and trains the resulting pattern-based weights.The procedure supports both pretrained models and training from scratch.
- Pattern-set design: 6–8 patterns provide a desirable tradeoff between code-generation efficiency and accuracy flexibility.Too many patterns complicate efficient code generation, while too few can degrade accuracy.
- Pattern-set design: The pattern set is formed from frequent four-entry natural patterns, retaining the central 3 × 3 kernel weight and selecting the Top-k patterns across kernels.Natural patterns are derived from the largest-magnitude weights, including the central weight.
- Pruning formulation: Kernel pattern pruning constrains every kernel to one pattern, while connectivity pruning limits each layer’s number of nonzero kernels to α_k.Both constraints must be satisfied simultaneously.
- Optimization algorithm: The extended ADMM solution alternates weight optimization, analytical projection subproblems, and dual-variable updates until convergence.The weight subproblem can use stochastic gradient descent, while pruning projections are polynomial-time solvable.
- Accuracy and compression: 8-pattern joint pruning with 3.6× connectivity pruning achieves about 8× VGG-16 CONV weight reduction and 4.4× ResNet-50 CONV weight reduction without Top-5 accuracy loss.The reduction rate is close to ADMM-based non-structured pruning and exceeds prior heuristic-based pruning on CONV layers.
5 PatDNN Inference Code Optimization
PatDNN uses sparsity-aware layer representations and compiler transformations to recover execution efficiency lost from fine-grained pattern pruning. Reordering, compact storage, redundancy elimination, and tuning produce branchless, balanced execution for mobile CPUs and GPUs.
- Compiler flow: PatDNN’s compiler converts computational graphs and layerwise representations into optimized vectorized C++ and OpenCL execution code.The flow progressively applies reordering, FKW storage, load redundancy elimination, and auto-tuning.
- Layerwise representation: The layerwise representation records pattern, connectivity, layout, and layer-specific information needed for aggressive pruning-aware optimization.It also captures computation parameters such as kernel size, stride, and dilation.
- Filter kernel reorder: Filter kernel reorder groups similar filters and identical-pattern kernels, eliminating branches while improving instruction parallelism, thread similarity, and load balance.Filter reorder uses filter length and positional pattern similarity; kernel reorder groups identical patterns within filters.
- Compressed weight storage: FKW stores reordered sparse weights in a higher-level compact format that uses less index-array overhead than CSR and supports branchless execution.FKW is designed around the known pattern layout produced by filter kernel reorder.
- Load redundancy elimination: Pattern regularity enables load-redundancy elimination that is difficult or impossible for irregular sparse models with unpredictable memory accesses.The preserved pattern structure exposes reusable access patterns for register-load optimization.
6 Evaluation
PatDNN outperforms TFLite, TVM, and MNN across evaluated mobile CPU and GPU workloads without accuracy compromise. Its optimizations convert pruning-induced computation and memory reductions into real-time inference for large DNNs, although TFLite lacks GPU support for VGG on ImageNet.
- Overall performance: PatDNN outperforms TFLite, TVM, and MNN on all six evaluated DNNs across mobile CPU and GPU cases.The evaluation covers VGG-16, ResNet-50, and MobileNet-V2 on ImageNet and CIFAR-10.
- Overall performance: 44.5× maximum CPU speedup over TFLite and 11.4× maximum GPU speedup over TVM demonstrate PatDNN’s strongest framework comparisons.On CPU, speedups range from 12.3× to 44.5× over TFLite; on GPU, they range from 2.8× to 11.4× over TVM.
- Real-time inference: 18.9 ms VGG inference on ImageNet’s Adreno 640 GPU remains within the 33 ms/frame real-time requirement after including other network layers.The 18.9 ms figure covers CONV layers; the paper states the complete inference still meets the usual 30 frames/sec target.
- Optimization contributions: Pattern pruning reduces computation by 3× to 8×, while compiler optimizations convert that reduction into performance gains that CSR sparsity alone does not achieve.The cited optimizations include filter kernel reorder, load redundancy elimination, auto-tuning, and compressed weight storage.
- Optimization contributions: On VGG CONV layers, reorder, load-redundancy elimination, and parameter tuning provide CPU speedups of 1.6×–3.0×, 1.6×–2.8×, and 1.2×–1.9×, respectively.The corresponding GPU ranges are 2.7×–6.1×, 1.5×–3.3×, and 1.4×–3.8×.
- GFLOPS and memory performance: PatDNN’s sparse implementation matches its optimized dense baseline on CPU GFLOPS and exceeds it on GPU.The result indicates comparable CPU memory performance and better GPU memory performance for the sparse implementation.
- Portability: PatDNN remains more stable across two additional platforms, where pruning reduces computation and memory accesses and therefore memory-bandwidth pressure.The portability study reports that competing frameworks slow substantially on Magic 2 relative to Snapdragon 855.
7 Discussion
PatDNN’s techniques are presented as generalizable beyond the evaluated mobile platforms, while portability and sparse-DNN support remain important discussion points.
- Generality: PatDNN was also tested on Raspberry Pi 4, showing similar speedup over frameworks such as TVM.
- Generality: The authors identify portability improvements through integration with TVM as a promising research direction.
- Dense vs. Sparse DNNs: General end-to-end frameworks such as TFLite, TVM, and MNN do not support sparse DNN execution.
- Dense vs. Sparse DNNs: Adding random pruning and CSR-based compressed storage to these frameworks is expected to provide little speed improvement.
- Dense vs. Sparse DNNs: Coarse-grained structured pruning is identified as a potential avenue for improving sparse-DNN performance.
8 Conclusion
PatDNN combines pattern-based pruning with architecture-aware code generation to target real-time DNN execution on mobile devices. It outperforms other end-to-end frameworks by up to 44.5× without accuracy compromise and achieves real-time execution for large-scale DNNs.
- PatDNN combines extended-ADMM pattern-based pruning with optimized code generation and architecture-aware execution optimizations.
- The framework is designed to benefit from both high accuracy and hardware efficiency.
- Up to 44.5× speedup over other state-of-the-art end-to-end DNN frameworks was achieved with no accuracy compromise.
- Real-time execution of large-scale DNNs on mobile devices was achieved.