Source-linked AI summary
BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services
Amir Erfan Eshratifar, Amirhossein Esmaili, Massoud Pedram
TL;DR
Mobile-cloud DNN inference can incur substantial wireless latency and energy costs because feature data must be transmitted to the cloud. BottleNet reduces that feature size with learnable bottleneck units and compression-aware training, achieving average 30× latency and 40× mobile-energy improvements over cloud-only inference with negligible accuracy loss.
Problem
Wireless transfer of feature tensors remains a major latency and mobile-energy cost in collaborative intelligence, while lossy compression can adversely affect accuracy.
Method
BottleNet inserts learnable channel and spatial reduction and restoration units around a lossy compressor and decompressor, with end-to-end compression-aware training.
Results
30× average improvement in end-to-end latency and 40× average improvement in mobile energy consumption are achieved versus the cloud-only approach with negligible accuracy loss.
Takeaways & Limitations
BottleNet provides a partitioning scheme that reduces communication costs and can select partition points for latency and mobile-energy objectives under varying deployment conditions.
Takeaways & Limitations
Future work is needed to study adaptability across underlying deep models, alternative compressors beyond JPEG, and further feature-dimension reduction architectures.
Abstract
from arXiv · showhide
Recent studies have shown the latency and energy consumption of deep neural networks can be significantly improved by splitting the network between the mobile device and cloud. This paper introduces a new deep learning architecture, called BottleNet, for reducing the feature size needed to be sent to the cloud. Furthermore, we propose a training method for compensating for the potential accuracy loss due to the lossy compression of features before transmitting them to the cloud. BottleNet achieves on average 30x improvement in end-to-end latency and 40x improvement in mobile energy consumption compared to the cloud-only approach with negligible accuracy loss.
1 INTRODUCTION
Collaborative intelligence splits DNN inference between mobile devices and the cloud, but transmitted feature data remains a major latency and energy cost. BottleNet inserts learnable reduction, lossy compression, decompression, and restoration units to reduce transmitted feature size while compensating for accuracy loss.
- Collaborative intelligence computes initial DNN layers on mobile devices and sends intermediate feature tensors to the cloud for the remaining computation.
- BottleNet adds a learnable reduction unit, conventional compressor, decompressor, and learnable restoration unit around the mobile-cloud partition.
- The approach targets models with large intermediate feature tensors, where ordinary partitioning may require substantial mobile computation before transmission.
- Lossy feature compression can save more bits than lossless compression but may reduce accuracy, motivating compression-aware training.
- 30× lower end-to-end latency and 40× lower mobile energy consumption are achieved on average versus the cloud-only approach, with negligible accuracy loss.
- The bottleneck unit reduces feature dimensions along channel or spatial axes before transmission and restores them on the cloud.
2 PROPOSED METHOD
BottleNet inserts a bottleneck unit into a DNN to reduce intermediate feature dimensions before lossy compression and cloud transmission. Its compression-aware training treats compression as an identity function during backpropagation so the full model remains end-to-end differentiable.
- Bottleneck Unit: Channel-wise reduction maps (batch_size, w, h, c) to (batch_size, w, h, c′), while spatial reduction maps it to (batch_size, w′, h′, c).Channel reduction uses c′ ≪ c; spatial reduction uses w′ < w and h′ < h.
- Bottleneck Unit: The bottleneck unit combines learnable reduction, compression, decompression, and restoration units to create a transmitted compressed feature representation.Reduction and compression occur on the mobile device; decompression and restoration occur on the cloud.
- Non-differentiable Lossy Compression Aware Training: Lossy compression is used during forward propagation but approximated as an identity function during backpropagation to preserve gradient flow.This addresses the non-differentiability introduced by quantization and enables end-to-end training.
- Compression Processing: Features are quantized to unsigned n-bit values and reshaped into 2-D tensors before compression.The reshaping aims to keep the aspect ratio as square as possible for maximum compression ratio.
- Bottleneck Placement: BottleNet evaluates channel and spatial reduction factors while selecting the bottleneck location and reduction sizes through training, profiling, and selection.The bottleneck is considered after each of M candidate layers.
3 EVALUATION
The evaluation prototypes BottleNet on ResNet-50 using a Jetson TX2 mobile platform, a GTX 1080 Ti server, and 3G, 4G, and Wi-Fi settings. Experiments examine partitions across residual blocks under an accuracy-loss constraint.
- Experimental Setup: The evaluation uses an NVIDIA Jetson TX2 mobile board and a NVIDIA GeForce GTX 1080 Ti server, with measured or modeled wireless power across 3G, 4G, and Wi-Fi.The wireless transmission power model has an estimation error rate below 6%.
- Implementation: Inference networks are implemented with TensorRT in FP16 mode, with 8-bit quantization applied before lossy compression.The mobile GPU does not support INT8 inference operations.
- Experimental Setup: BottleNet is evaluated on ResNet-50, which contains 16 residual blocks, with one candidate model placing the bottleneck after each block.These placements produce 16 different network partitions.
- Experimental Setup: The experiments use miniImageNet with 100 classes, 600 examples per class, an 85% training split, and 224×224 random crops.Models are trained for 90 epochs.
- Accuracy Constraint: 76% is the target ResNet-50 accuracy on miniImageNet without a bottleneck unit.The target is used to define acceptable accuracy loss for partition selection.
- Accuracy Constraint: With a 2% acceptable accuracy loss, channel sizes c′ of 1, 2, 5, and 10 are selected for bottleneck placements after RB groups 1–3, 4–7, 8–13, and 14–16.The spatial reduction factor is s=2 for all 16 partitions.
2 N: number of layers in the DNN
The notation defines N as the number of layers in the DNN.
- N denotes the number of layers in the DNN.
8 Kcloud: current load level of cloud
The notation specifies load-dependent latency and power terms for mobile and cloud execution at partition j.
- t_mobile(j,K_mobile) and p_mobile(j,K_mobile) denote mobile latency and power for partition j and mobile load K_mobile.
- t_cloud(j,K_cloud) denotes cloud latency for partition j and cloud load K_cloud.
12 PU : wireless network up-link power consumption
BottleNet evaluates bottleneck placement and compression settings to reduce transmitted feature data while maintaining acceptable accuracy. Across wireless settings, the proposed partitions improve latency and mobile energy relative to cloud-only inference.
- Evaluation: The selected partitions place bottleneck units after residual blocks using configurations with accuracy loss below 2%.Latency, mobile energy consumption, and offloaded data size are evaluated across wireless networks without congestion.
- Results: 63×, 21×, and 8× end-to-end latency improvements over cloud-only inference are achieved on 3G, 4G, and Wi-Fi, respectively.The comparison uses the proposed method against the cloud-only approach.
- Results: 47×, 41×, and 31× mobile energy-consumption improvements over cloud-only inference are achieved on 3G, 4G, and Wi-Fi, respectively.The best partition across wireless settings places the bottleneck unit after RB1.
- Compression-aware training: Compression-aware training reduces accuracy loss to almost zero above JPEG quality 20, whereas naive training has close to 18% loss at quality 20.The experiments use JPEG quality 20 to maximize bit savings without accuracy loss.
- Server-load adaptation: At run time, the mobile can change the partition point according to server load while maintaining acceptable accuracy and offloading less data than cloud-only inference.The best partition can be selected by periodically pinging the server during mobile idle periods.
4 CONCLUSION AND FUTURE WORK
BottleNet introduces a bottleneck unit and compression-aware end-to-end training to reduce mobile–cloud feature-transfer costs while maintaining acceptable accuracy. Across wireless settings, it improves latency and mobile energy consumption relative to cloud-only deployment, while future work remains on broader model and compression adaptability.
- Conclusion: 30× average end-to-end latency improvement and 40× average mobile energy-consumption improvement are achieved versus the cloud-only approach for ResNet-50.Accuracy loss is less than 2%.
- Conclusion: BottleNet creates a neural-network bottleneck with a learnable reduction unit, compressor, decompressor, and restoration unit.The architecture is trained end-to-end using compression-aware training to save bits while retaining acceptable accuracy.
- Conclusion: The method selects a partition point at run time for minimum end-to-end latency and/or mobile energy consumption across hardware, wireless, and load settings.The stated adaptability covers DNN architecture, hardware platform, wireless network settings, and mobile and server load levels.
- Future Work: Future work should study adaptability to various deep models, alternative compression techniques beyond JPEG, and further feature-tensor dimension reduction.The proposed reduction-unit architectures also provide a basis for exploring more extensive dimension reduction.