Source-linked AI summary
DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
Shuochao Yao, Yiran Zhao, Aston Zhang, Lu Su, Tarek Abdelzaher
TL;DR
Deep learning can improve sensing applications, but embedded devices face high memory, execution-time, and energy demands. DeepIoT learns dropout probabilities from parameter redundancies to compress diverse network structures into smaller dense matrices. Across sensing tasks on Intel Edison, it substantially reduces model size, running time, and energy without accuracy loss.
Problem
High memory consumption, execution time, and energy demand impede deploying trained deep-learning sensing applications on low-end embedded devices.
Method
DeepIoT uses a compressor neural network and compressor-critic optimization to learn dropout probabilities from parameter redundancies and minimize hidden elements across network layers.
Results
90% to 98.9% model-size reductions, 71.4% to 94.5% shorter running times, and 72.2% to 95.7% lower energy consumption are achieved without loss of accuracy.
Takeaways & Limitations
DeepIoT generates concise network structures that reduce sensing-system resource consumption while preserving performance on resource-constrained embedded devices.
Takeaways & Limitations
DeepIoT focuses on structure compression and does not evaluate combining it with weight quantization or exploiting heterogeneous processors such as DSPs.
Abstract
from arXiv · showhide
Recent advances in deep learning motivate the use of deep neutral networks in sensing applications, but their excessive resource needs on constrained embedded devices remain an important impediment. A recently explored solution space lies in compressing (approximating or simplifying) deep neural networks in some manner before use on the device. We propose a new compression solution, called DeepIoT, that makes two key contributions in that space. First, unlike current solutions geared for compressing specific types of neural networks, DeepIoT presents a unified approach that compresses all commonly used deep learning structures for sensing applications, including fully-connected, convolutional, and recurrent neural networks, as well as their combinations. Second, unlike solutions that either sparsify weight matrices or assume linear structure within weight matrices, DeepIoT compresses neural network structures into smaller dense matrices by finding the minimum number of non-redundant hidden elements, such as filters and dimensions required by each layer, while keeping the performance of sensing applications the same. Importantly, it does so using an approach that obtains a global view of parameter redundancies, which is shown to produce superior compression. We conduct experiments with five different sensing-related tasks on Intel Edison devices. DeepIoT outperforms all compared baseline algorithms with respect to execution time and energy consumption by a significant margin. It reduces the size of deep neural networks by 90% to 98.9%. It is thus able to shorten execution time by 71.4% to 94.5%, and decrease energy consumption by 72.2% to 95.7%. These improvements are achieved without loss of accuracy. The results underscore the potential of DeepIoT for advancing the exploitation of deep neural networks on resource-constrained embedded devices.
1 INTRODUCTION
DeepIoT targets the memory, execution-time, and energy demands that impede deploying deep-learning sensing applications on low-end embedded devices. It learns compact network structures across common neural architectures while preserving sensing performance.
- High memory consumption, execution time, and energy demand impede deploying trained deep-learning sensing networks on target embedded devices.
- DeepIoT minimizes the number of hidden elements in each layer, producing small dense matrices instead of large sparse parameter matrices.Small dense matrices avoid storing element indices and are efficiently optimized for processing.
- DeepIoT applies dropout-based compression to fully-connected neurons, convolutional filters, recurrent dimensions, and their combinations.Dropout probabilities determine which hidden elements are pruned to generate a thinned network structure.
- The compressor neural network learns parameter redundancies from layer parameters and generates dropout probabilities for the original network.It is optimized jointly with the original network through a compressor-critic framework that minimizes the sensing application's loss.
- Experiments on Intel Edison cover handwriting, vision, speech, human activity recognition, and user identification sensing tasks.The first experiment set evaluates interaction modalities, while the second examines human-centric context sensing in resource-constrained scenarios.
- 90% to 98.9% model-size reductions, 71.4% to 94.5% shorter running times, and 72.2% to 95.7% lower energy consumption are achieved without loss of accuracy.Compared with state-of-the-art baselines, DeepIoT also reduces model size by 11.6% to 83.2%, running time by 60.9% to 87.9%, and energy consumption by 64.1% to 88.7%.
2 RELATED WORK
Prior embedded-network compression methods use approaches such as low-rank factorization, sparse coding, magnitude pruning, frequency-domain representations, and distillation. DeepIoT instead provides a dropout-based framework for reducing parameter redundancies and compressing network structures.
- Embedded sensing research seeks to run progressively more capable applications under increasingly pronounced resource constraints.
- Magnitude-based pruning removes low-magnitude connections iteratively but requires sparse-matrix support and can risk irretrievable network damage under aggressive pruning.
- Factorization-based methods exploit low-rank parameter structure, but may provide lower compression ratios and risk hurting final network performance.
- DeepIoT is presented as a dropout-based framework that reduces parameter redundancies and generates concise structures for transplanting large networks onto constrained devices.
3 SYSTEM FRAMEWORK
DeepIoT compresses neural-network structures by learning dropout probabilities from weight redundancies and jointly optimizing a compressor with the original network. The framework applies across fully-connected, convolutional, recurrent, and combined architectures while progressively producing smaller dense structures.
- Compression pipeline: DeepIoT inserts dropout into internal layers, excluding fixed-dimension input and output layers, to remove hidden elements during compression.The dropout masks control which hidden elements remain active.
- Compression pipeline: The compressor learns weight redundancies across layers and generates dropout probabilities for hidden elements in the original network.The binary mask and probabilities represent the learned stochastic dropout policy.
- Compressor-critic optimization: The compressor and original network are optimized iteratively: the compressor seeks efficient structures, while the original network maintains sensing performance.DeepIoT uses the critic’s objective and jointly updates both networks.
- Dropout policy: Individual Bernoulli probabilities provide finer-grained structure compression than a single shared dropout probability.Unlike conventional dropout’s commonly constant probability, DeepIoT assigns probabilities to different hidden elements.
- Architecture coverage: DeepIoT drops neurons, convolutional kernels, or recurrent hidden dimensions, enabling compression of fully-connected, convolutional, recurrent, and combined networks.Convolution is reformulated as matrix multiplication, allowing kernels to be pruned; LSTM compression prunes hidden dimensions.
- Compression schedule: The compression process softly suppresses dropout probabilities until the remaining parameter percentage falls below a user-defined threshold.A decay factor γ ∈ (0, 1) and threshold τ govern gradual suppression and deterministic mask generation.
4 IMPLEMENTATION
DeepIoT is implemented and evaluated on CPU-only Intel Edison devices, with compressed models exported from workstation-based training and compression. The implementation compares dense and sparse baseline representations under their relevant execution trade-offs.
- Hardware and runtime: Intel Edison experiments use a 500 MHz dual-core Intel Atom CPU, 1GB memory, and 4GB flash storage.All neural-network models run solely on the device CPU.
- Evaluation setup: Original networks and baseline compression processes run on a workstation, while compressed models are exported to Intel Edison flash storage for evaluation.The workstation uses an NVIDIA GeForce GTX Titan X for training.
- Software implementation: Compressed models run through Theano on Intel Edison CPU, with BLAS and Sparse BLAS optimizing dense and sparse matrix operations.No additional runtime optimization is applied to compressed models.
- Memory targeting: DeepIoT automatically generates a compressed network sized to fit a configured fraction of the embedded device’s available memory.The needed compression ratio is computed from device memory information and the target compressed size.
- Baseline comparison: Sparse baselines are evaluated in both sparse and nonsparse forms because sparse multiplication can be slower when more than 20% of elements are nonzero.The same trade-off applies to CPU convolution implemented as matrix multiplication.
5 EVALUATION
DeepIoT is evaluated on handwriting, vision, and speech recognition using compressed LeNet-5, VGGNet, and deep bidirectional LSTM models. Across these tasks, it preserves accuracy while improving memory, execution-time, and energy tradeoffs over baseline compression methods.
- Handwritten digits recognition with LeNet5.: DeepIoT achieves at least ×2 better accuracy–memory tradeoffs than two baselines on LeNet-5.The paper attributes this to global redundancy modeling and direct hidden-unit pruning, which enables small dense matrices.
- Handwritten digits recognition with LeNet5.: 14.2ms inference reduces LeNet-5 execution time by 71.4% without loss of accuracy.SparseSep is faster than DyNS but incurs around 0.2% test-error degradation; convolution operations dominate execution time.
- Handwritten digits recognition with LeNet5.: 73.7% lower energy consumption is achieved by DeepIoT on LeNet-5 without loss of accuracy.Energy consumption is more sensitive to operation count than model size in this evaluation.
- Image recognition with VGGNet.: DeepIoT beats two baselines on VGGNet while retaining the original testing accuracy, with global information shared across layers.Its standard-matrix representation avoids the index overhead associated with sparse matrices.
- Image recognition with VGGNet.: 82.2ms prediction reduces VGGNet execution time by 94.5% without loss of accuracy, while energy consumption falls by 95.7%.The execution-time comparison differs from LeNet-5 because VGGNet uses smaller convolutional kernels; SparseSep also fails to preserve performance without fine-tuning.
- Speech recognition with deep Bidirectional LSTM.: DeepIoT reduces deep bidirectional LSTM execution time by 86.4% and energy consumption by 87% without degrading word error rate.The recurrent model remains above 5% of its original parameters but still achieves a 20-fold reduction and less than 10% of the original parameters.
6 DISCUSSION
DeepIoT simplifies network structures for resource-constrained devices while preserving accuracy, with compression and fine-tuning performed before deployment. The paper focuses on structure pruning and leaves quantization and heterogeneous processor acceleration outside its scope.
- 6 DISCUSSION: DeepIoT generates a simplified network by deciding which hidden elements to drop in each layer.The process includes fine-tuning after compression, and both steps occur on a workstation rather than the target device.
- 6 DISCUSSION: Compression and fine-tuning are performed ahead of time, so embedded and mobile devices do not bear that computational burden.
- 6 DISCUSSION: DeepIoT focuses on structure or weight pruning, while weight quantization and heterogeneous processor acceleration remain outside this paper’s scope.The paper describes pruning and quantization as non-conflicting methods, but does not evaluate their combination or hardware speed-up.
7 CONCLUSION
DeepIoT learns succinct network structures through dropout and a compressor-critic framework that models parameter redundancies globally. Across the experiments, it outperforms baseline compression algorithms and reduces resource consumption without hurting performance.
- 7 CONCLUSION: DeepIoT jointly optimizes the original network and a compressor neural network to learn parameter redundancies and generate dropout probabilities.The resulting stochastic hidden elements produce a more succinct network structure for sensing applications.
- 7 CONCLUSION: DeepIoT outperforms the baseline compression algorithms by a significant margin in all experiments.
- 7 CONCLUSION: The compressed structures greatly reduce sensing-system resource consumption without hurting performance.The authors state that this makes many state-of-the-art deep neural networks deployable on resource-constrained embedded and mobile devices.