Source-linked AI summary
Distributed Deep Neural Networks over the Cloud, the Edge and End Devices
Surat Teerapittayanon, Bradley McDanel, H. T. Kung
TL;DR
DDNN addresses the challenge of providing accurate DNN inference across resource-constrained, geographically distributed devices without always sending raw sensor data to the cloud. It maps and jointly trains DNN sections across devices, edge, and cloud, enabling local exits and automatic aggregation. In the evaluation, DDNN reduced communication cost by over 20x relative to raw-sensor cloud offloading while maintaining high accuracy.
Problem
DNN deployment must balance cloud accuracy and centralized processing costs against end-device limits on communication, resources, latency, and privacy.
Method
DDNN maps sections of one DNN across end devices, edge, and cloud and jointly trains them with local exit thresholds and aggregation schemes.
Results
Communication cost was reduced by over 20x versus offloading raw sensor input to a cloud DNN performing all inference.
Takeaways & Limitations
DDNN provides a distributed framework that combines local inference, cloud processing, automatic sensor fusion, fault tolerance, and privacy benefits.
Takeaways & Limitations
The current DDNN uses binary layers throughout; future work will investigate larger multimodal datasets, more end devices, and mixed-precision cloud layers.
Abstract
from arXiv · showhide
We propose distributed deep neural networks (DDNNs) over distributed computing hierarchies, consisting of the cloud, the edge (fog) and end devices. While being able to accommodate inference of a deep neural network (DNN) in the cloud, a DDNN also allows fast and localized inference using shallow portions of the neural network at the edge and end devices. When supported by a scalable distributed computing hierarchy, a DDNN can scale up in neural network size and scale out in geographical span. Due to its distributed nature, DDNNs enhance sensor fusion, system fault tolerance and data privacy for DNN applications. In implementing a DDNN, we map sections of a DNN onto a distributed computing hierarchy. By jointly training these sections, we minimize communication and resource usage for devices and maximize usefulness of extracted features which are utilized in the cloud. The resulting system has built-in support for automatic sensor fusion and fault tolerance. As a proof of concept, we show a DDNN can exploit geographical diversity of sensors to improve object recognition accuracy and reduce communication cost. In our experiment, compared with the traditional method of offloading raw sensor data to be processed in the cloud, DDNN locally processes most sensor data on end devices while achieving high accuracy and is able to reduce the communication cost by a factor of over 20x.
I. INTRODUCTION
DDNNs address the tension between cloud accuracy and the communication, latency, privacy, and resource constraints of end-device inference by jointly mapping one DNN across cloud, edge, and geographically distributed devices.
- Cloud offloading incurs communication, latency, and privacy costs, whereas simple end-device models can reduce accuracy.
- Distributed hierarchies support coordinated local and central decisions for geographically distributed IoT devices.
- A straightforward hierarchy partition can exceed communication budgets, while end devices also face memory, battery, and accuracy constraints.
- DDNN maps one DNN across cloud, edge, and geographically distributed end devices, then jointly trains the sections.
- Shallow DDNN portions enable fast localized inference, while deeper cloud portions support further processing and classification.
- DDNNs support sensor fusion, privacy, fault tolerance, neural-network scaling, and geographical scaling through distributed computing.
II. RELATED WORK
The related-work discussion situates DDNN within distributed computing hierarchies and deep-learning methods that distribute computation or enable early exits.
- Prior work covers distributed computing hierarchies and deep-learning algorithms that enable distributed execution.
A. Distributed Computing Hierarchy
DDNN maps a jointly trained DNN across heterogeneous devices, supporting local exits, multi-device aggregation, and edge-cloud scaling.
- The framework overview describes DDNN architecture, training, and inference across distributed computing hierarchies.
- DDNN maps a trained DNN onto heterogeneous physical devices distributed across local devices, the edge, and the cloud.
- End devices can classify confident samples locally, while harder cases send intermediate DNN outputs to the cloud for further inference.
- Geographically distributed end devices aggregate local outputs before an exit point, with joint training determining the aggregation.
- An edge layer can aggregate and classify device outputs or forward intermediate results to the cloud when more processing is needed.
B. DDNN Aggregation Methods
DDNN aggregates outputs from multiple devices using max pooling, average pooling, or concatenation, each encoding different assumptions about information combination.
- DDNN offers max pooling, average pooling, and concatenation for aggregating outputs from multiple end devices.
- Max pooling: Max pooling forms each output component by taking the maximum corresponding component across input vectors.
- Average pooling: Average pooling forms each output component from the corresponding component average and may reduce noisy device inputs.
- Concatenation: Concatenation retains all input information for higher layers but expands vector dimensionality, requiring an additional linear layer to restore dimensions.
C. DDNN Training
DDNN training jointly optimizes losses from multiple exit points so the distributed network can support accurate inference at different hierarchy depths. The method combines exit losses in a weighted objective and can be trained centrally despite distributed inference.
- DDNN combines the loss from each exit during backpropagation to jointly train the entire network.Each exit is intended to achieve good accuracy relative to its depth.
- The training objective uses a softmax cross entropy loss for each exit branch.Each branch computes the network function from an entry point to its corresponding exit.
- The joint optimization minimizes a weighted sum of the loss functions from all exit points.N denotes the total number of exits, and w_n is the weight associated with exit n.
D. DDNN Inference
DDNN inference uses confidence thresholds at successive exit points to decide whether samples can be classified locally or require higher-level processing. Multiple devices can send progressively more detailed summaries through local aggregation, the edge, and the cloud.
- Inference uses one preconfigured threshold T at each exit point to measure prediction confidence.The paper uses normalized entropy, whose threshold can be selected on a validation set for best accuracy.
- Normalized entropy ranges from 0 to 1, with values near 0 indicating confidence and values near 1 indicating uncertainty.At each exit, the entropy is compared with that exit's threshold.
- Samples with η > T fall back to a higher exit until the final exit, which always performs classification.
- With multiple end devices, local aggregation first tests whether combined summaries suffice before requesting more detailed information from the edge or cloud.The procedure proceeds from local summaries to edge processing and then cloud processing when additional computation is needed.
- Communication cost includes per-class probability transmission for every sample and device-to-cloud transmission only for samples not exited locally.The latter occurs for the fraction 1 − l of samples that continue to the cloud.
F. Accuracy Measures
The evaluation distinguishes accuracy at local, edge, cloud, and mixed exit policies, including separately trained individual device models. It reports collaborative accuracy gains, graceful cloud offloading, fault tolerance, and lower communication costs than sending all sensor inputs to the cloud.
- Local, Edge, and Cloud Accuracy measure performance when 100% of samples exit at the respective local, edge, or cloud point.
- Overall Accuracy measures classification when samples exit at multiple hierarchy points according to entropy thresholds.The threshold T determines how samples are distributed across local, edge, and cloud exits.
- Individual Accuracy evaluates a separately trained end-device model without relying on DDNN local or cloud exits.Each individual model contains a ConvP block followed by a fully connected block.
- DDNNs allow multiple end devices to collaborate and improve accuracy at both local and cloud exits.
- DDNNs offload difficult samples to the cloud, extending end-device capability without requiring every sample to be classified locally.
- Missing one end device does not dramatically affect accuracy, while performance gradually degrades as more devices are lost.
- DDNNs reduce end-device communication costs compared with systems that offload all input sensor data to the cloud.
A. DDNN Evaluation Architecture
The evaluation uses binary neural-network blocks mapped across six camera-equipped end devices and the cloud. The dataset synchronizes views of the same objects across devices, with blank images marking absent objects.
- Binary blocks: Binary FC and ConvP blocks reduce the memory requirements for neural-network layers running on end devices.FC blocks combine a fully connected layer, batch normalization, and binary activation; ConvP blocks combine convolution, pooling, batch normalization, and binary activation.
- Binary blocks: The ConvP block uses a 3x3 convolution kernel with stride 1 and padding 1, followed by 3x3 pooling with stride 2 and padding 1.
- Distributed evaluation: The evaluation architecture maps FC and ConvP blocks across six end devices and the cloud for distributed inference.The dataset contains images acquired simultaneously from six cameras at different locations, with each camera attached to an end device.
- Dataset: Each synchronized sample contains cropped 32x32 RGB views of one object, while devices without that object contribute blank images labeled -1.The evaluated object classes are car, bus, and person.
C. Impact of Aggregation Schemes
The study compares max pooling, average pooling, and concatenation at local and cloud exits. MP-CC is selected because max pooling suits class outputs locally, while concatenation preserves viewpoint-specific features for cloud processing.
- Aggregation schemes: Three aggregation methods—max pooling, average pooling, and concatenation—are evaluated at both local and cloud exit points.The scheme name records the local aggregator first and the cloud aggregator second; MP-CC uses local max pooling and cloud concatenation.
- Observed trade-offs: MP-MP performs well locally but poorly in the cloud because max pooling discards viewpoint-specific activation features.Local outputs represent corresponding class responses, whereas cloud activations represent different visual features across devices.
- Observed trade-offs: MP-CC improves local classification accuracy because cloud concatenation lets all devices receive gradients during backpropagation.
- Observed trade-offs: CC-CC yields poor local accuracy but high cloud accuracy because concatenation preserves the most information for cloud neural-network processing.Local concatenation does not enforce correspondence between outputs for the same class across devices.
- Selected configuration: Average pooling generally performs worse than max pooling locally because absent-object devices dilute strong responses from devices where the object is visible.The paper therefore uses MP-CC in the remaining experiments.
D. Entropy Threshold
The entropy threshold controls whether samples exit locally or continue to the cloud, trading communication and latency against accuracy. With T = 0.8, the system achieves 97% accuracy while exiting 60.82% of samples locally.
- T = 0 means no samples exit locally, whereas T = 1 means all samples exit at the local point.
- As more samples exit locally, overall accuracy decreases because the local exit is typically less accurate than the cloud exit.
- 97% accuracy is achieved at T = 0.8 while 60.82% of samples exit locally, substantially reducing communication.
- The T = 0.8 threshold combines local and cloud classification effectively, including samples classified correctly locally but incorrectly by the cloud classifier.
- The “Overall” accuracy is roughly equivalent to always using the cloud exit, while 60.82% of samples exit locally at reduced communication cost.
F. Impact of Cloud Offloading on Accuracy Improvements
Cloud offloading improves accuracy by extending processing beyond resource-constrained local models, while DDNN keeps device memory and communication requirements small. The system also retains high accuracy under end-device failures.
- The end-device neural-network layers require under 2 KB of memory while approximately 75% of samples exit locally.
- Around a 5% accuracy improvement over the local aggregator is achieved by offloading difficult samples to the cloud.This improvement remains when end devices use larger models with more filters.
- Under any individual end-device failure, the system still achieves over 95% overall classification accuracy.
- Reducing the number of end devices from 6 to 4 decreases overall accuracy by only 4%, suggesting robustness to multiple failures.
- 3072 bytes are required to offload each raw 32x32 RGB image, compared with 140 bytes per sample for the largest evaluated DDNN model.The DDNN communication cost is therefore reduced by over 20x relative to raw-input cloud offloading.
V. DDNN PROVISION FOR HORIZONTAL AND VERTICAL SCALING
DDNN maps jointly trained neural-network sections across end devices, the edge, and the cloud to support both horizontal scaling across devices and vertical scaling across hierarchy levels. The evaluation reports high accuracy, fault tolerance, and reduced communication, while current layers remain binary.
- Feature pooling through local and cloud aggregators combines inputs from multiple devices and improves accuracy at both local and cloud levels.
- A single DNN is jointly trained and mapped across a distributed hierarchy, with shallow processing at devices or the edge and deeper processing in the cloud.
- DDNN scales horizontally across end devices and vertically across the cloud, edge, and device hierarchy while supporting automatic sensor fusion and fault tolerance.
- The framework reduces communication by exiting many samples locally and sending compact binary feature representations to the cloud when further processing is required.For the evaluation dataset, communication is reduced by over 20x compared with raw sensor-input cloud offloading.
- Current DDNN layers are binary, and future work will investigate larger datasets, more input modalities, more end devices, and mixed-precision cloud layers.