Source-linked AI summary
Edge-Host Partitioning of Deep Neural Networks with Feature Space Encoding for Resource-Constrained Internet-of-Things Platforms
Jong Hwan Ko, Taesik Na, Mohammad Faisal Amir, Saibal Mukhopadhyay
TL;DR
The paper addresses how to provide DNN inference in edge-host IoT systems while limiting edge computation, memory, transmission demand, and energy. It partitions the network, encodes intermediate features, and evaluates the resulting trade-offs. The approach improves edge energy-efficiency and throughput over entire-inference baselines, with the strongest reported gains reaching 15.3x and 16.5x over host inference.
Problem
Edge-only inference imposes costly computation and memory demands, while host-only inference requires bandwidth-intensive image transmission in edge-host IoT systems.
Method
The paper partitions a DNN at an intermediate layer, encodes its output features for transmission, and fine-tunes the host partition using encoded features.
Results
For AlexNet, lossy feature encoding reduced last-convolutional-layer feature size 28x with 1% accuracy loss, while the approach improved energy efficiency 15.3x and throughput 16.5x over host inference.
Takeaways & Limitations
Partitioning near the end of convolutional layers with feature encoding can improve edge energy-efficiency and throughput, with the best partition depending on network complexity and transmission bandwidth.
Takeaways & Limitations
Further compression can improve transmission throughput but causes additional accuracy loss, and changing channel bandwidth may require dynamically shifting the partitioning layer.
Abstract
from arXiv · showhide
This paper introduces partitioning an inference task of a deep neural network between an edge and a host platform in the IoT environment. We present a DNN as an encoding pipeline, and propose to transmit the output feature space of an intermediate layer to the host. The lossless or lossy encoding of the feature space is proposed to enhance the maximum input rate supported by the edge platform and/or reduce the energy of the edge platform. Simulation results show that partitioning a DNN at the end of convolutional (feature extraction) layers coupled with feature space encoding enables significant improvement in the energy-efficiency and throughput over the baseline configurations that perform the entire inference at the edge or at the host.
1 INTRODUCTION
The paper addresses edge-host DNN inference under constrained edge resources by partitioning the network and encoding intermediate features before transmission. This approach targets improved edge energy-efficiency and throughput while preserving accuracy.
- IoT edge devices increasingly stream high-rate sensor data while requiring DNNs for visual-data processing.
- Host-only inference requires transmitting images, whereas edge-only inference reduces transmission but imposes substantial computation and memory demands.
- The proposed design partitions inference at an intermediate layer, transmitting encoded feature maps from the edge to the host for the remaining inference.
- The design guideline places convolutional layers at the edge and fully connected layers at the host, with lossless or lossy feature encoding and split retraining.
- 28x feature-size reduction with 1% accuracy loss and 11% lower transmission demand after fine-tuning were reported for AlexNet.
- Compared with host inference, the approach improved energy efficiency 15.3x and throughput 16.5x; compared with edge inference, improvements were 2.3x and 2.5x.
2 RELATED WORK
Prior work either processes DNN inference at the host or edge, while earlier distributed partitioning lacked energy-throughput analysis and intermediate-feature encoding. This paper adds a layer-wise design guideline incorporating bandwidth, encoding, and accuracy effects.
- Host-based IoT inference centralizes decisions but depends on transmitting sensed visual data and reliable communication.
- Edge-based inference avoids transmitting full inputs but remains limited by the large resource demand of deep neural networks.
- Earlier distributed inference partitioned a shallow network section but did not analyze system energy and throughput or encode intermediate features.
- This paper develops a layer-wise partitioning guideline under accuracy and bandwidth constraints while incorporating feature encoding effects on transmission demand and classification accuracy.
3 PARTITIONING OF INFERENCE WITH FEATURE SPACE ENCODING
The paper treats a DNN as an information-encoding pipeline and partitions inference between edge and host, encoding intermediate features before transmission. It analyzes layer placement and lossless or lossy feature encoding to balance computation, memory access, bandwidth, energy, and accuracy.
- DNN as an Information Encoding Pipeline: The edge extracts features through early DNN layers, encodes the intermediate output, and transmits it to the host for the remaining inference.This partitions the network while treating the edge as an encoding engine.
- DNN as an Information Encoding Pipeline: Feature-space characteristics vary by layer: max-pooling reduces output size, while deeper features become sparser and lower in entropy.These changes affect how effectively conventional encoding methods compress different intermediate feature maps.
- Edge-Host Partitioning of Inference: Partitioning earlier convolutional layers leaves larger feature maps to transmit, whereas including fully connected layers at the edge increases memory-access demand and degrades throughput and energy efficiency.The resulting guideline places the partition at the end of the convolutional layers.
- Feature Space Encoding: Lossless run-length plus Huffman encoding reconstructs features perfectly but provides a limited 3-10x compression ratio.Its compression improves for deeper layers because their feature maps contain more zeros.
- Feature Space Encoding: At 1% accuracy loss, lossy JPEG encoding achieves 5-50x compression, with deeper low-entropy features generally obtaining higher compression ratios.Lossy encoding trades accuracy for lower transmission bandwidth and energy demand.
- Fine-Tuning of the Partitioned Network: Fine-tuning the host-side partition using encoded training features increases the compression ratio by 11% at 1% accuracy loss.For AlexNet conv5, the target accuracy is reached with quality factors between 30 and 35.
4 SIMULATION RESULTS
Simulation results show that partitioning and feature encoding improve edge throughput and energy efficiency, with the best partition depending on network type and transmission conditions.
- Simulation Setup: The inference engine uses 144 16-bit MAC units, a JPEG encoder, an on-chip feature-map buffer, and a JPEG decoder for compressed weights.The design was synthesized as a 28nm ASIC for energy and throughput analysis.
- Analysis without Feature Space Encoding: 1.2x and 1.15x higher energy efficiency and throughput occur at fc6 than with entire inference at the edge, while gains reach 4.5x and 7.5x versus host inference.Without feature encoding, fc6 is the best partition for AlexNet.
- Analysis with Feature Space Encoding: Feature encoding makes conv5 the optimal partition, reducing transmission latency and energy with negligible encoding overhead when convolutional features otherwise bottleneck performance.Further compression can improve transmission throughput but causes additional accuracy loss.
- Effect of Network Types: For VGG-16 and ResNet-50, the preferred partition reflects network structure: fully-connected memory demand harms edge inference, while removing ResNet-50’s final layer avoids that demand.ResNet-50’s smaller feature sizes make full edge inference more efficient than host inference, although partitioning still improves efficiency by removing the final layer.
- Effect of Transmission Channel: At 1 Mbps and 22 Mbps, convolutional-layer partitioning outperforms edge and host inference; higher bandwidth shifts the throughput-optimal partition earlier, while lower bandwidth favors deeper partitioning.The optimal layer shifts according to transmission bandwidth and transmitter energy.
5 CONCLUSIONS
The paper concludes that edge-host DNN partitioning with intermediate feature encoding improves edge energy efficiency and throughput relative to whole-network inference at either platform.
- 5 CONCLUSIONS: The proposed approach partitions DNN inference between edge and host platforms and encodes intermediate-layer features before transmission.Fine tuning is used with feature encoding in the demonstrated approach.
- 5 CONCLUSIONS: The reported improvement is relative to performing the entire inference at either the edge or the host platform.The conclusion frames the benefit as improved edge energy efficiency and throughput.