Machine Learning at the Network Edge: A Survey

M. G. Sarwar Murshed; Christopher Murphy; Daqing Hou; Nazar Khan; Ganesh Ananthanarayanan; Faraz Hussain

Source-linked AI summary

Machine Learning at the Network Edge: A Survey

M. G. Sarwar Murshed, Christopher Murphy, Daqing Hou, Nazar Khan, Ganesh Ananthanarayanan, Faraz Hussain

arXiv:1908.00080v4Jul 31, 2019cs.LGcs.CVcs.NIstat.ML

TL;DR

Resource-constrained devices make local ML deployment difficult, while cloud offloading introduces latency, communication, and privacy concerns. This survey synthesizes techniques and systems for training and deploying ML at the edge, reporting reduced data transfer and improved privacy and response times, while noting unresolved storage, metric-sensitivity, and data-heterogeneity limitations.

Problem
Resource-constrained devices have limited computational capabilities, making ML deployment difficult and motivating processing closer to data sources.
Method
The survey reviews techniques for training and deploying ML on edge computing devices, along with relevant applications, frameworks, software, and hardware.
Results
Edge-based ML systems have been shown to improve user-data privacy and security and system response times, while one edge-cloud system reduced transferred data by 96% versus cloud-only processing.
Takeaways & Limitations
The survey provides an operational overview of intelligent edge systems spanning compression techniques, applications, and common deployment platforms.
Takeaways & Limitations
The influence of data heterogeneity on distributed-model accuracy remains an open research issue.

Abstract

from arXiv · show

Resource-constrained IoT devices, such as sensors and actuators, have become ubiquitous in recent years. This has led to the generation of large quantities of data in real-time, which is an appealing target for AI systems. However, deploying machine learning models on such end-devices is nearly impossible. A typical solution involves offloading data to external computing systems (such as cloud servers) for further processing but this worsens latency, leads to increased communication costs, and adds to privacy concerns. To address this issue, efforts have been made to place additional computing devices at the edge of the network, i.e close to the IoT devices where the data is generated. Deploying machine learning systems on such edge computing devices alleviates the above issues by allowing computations to be performed close to the data sources. This survey describes major research efforts where machine learning systems have been deployed at the edge of computer networks, focusing on the operational aspects including compression techniques, tools, frameworks, and hardware used in successful applications of intelligent edge systems.

1 INTRODUCTION

Resource-constrained IoT devices generate substantial data but cannot readily run machine-learning models, making cloud offloading costly, slow, and privacy-sensitive. This survey examines edge-based ML applications, deployment techniques, platforms, hardware, and distributed intelligence across the end-edge-cloud architecture.

Cloud offloading increases communication costs, delays system response, and exposes private data to compromise.
Edge computing places computational resources near data sources so devices can process data locally or transmit only necessary information.
An edge-device may be an end-device or edge-server; edge-servers have more resources and support more complex tasks, while remote servers handle excessively intensive computations.
36% less expensive, the edge-cloud wind-farm system cost $28,927 versus $80,531 for cloud-only processing.
96% less data required transfer in the edge-cloud wind-farm system than in the cloud-only system.
The survey covers edge-intelligence applications, ML adaptation, distributed training and inference, hardware, and the supporting end-edge-cloud software ecosystem.

2 MACHINE LEARNING AT THE EDGE: TRAINING, COMPRESSION, AND DEPLOYMENT

Machine learning at the edge addresses the difficulty of deploying models on resource-constrained devices by adapting models and placing computation near IoT data sources. The survey covers classical and deep learning, compression, distributed learning, and deployment infrastructure.

Large computing systems can provide real-time computation, but deploying them on small devices such as robots can significantly affect performance.
Resource-constrained devices generate increasing amounts of edge data but have limited computational power, storage, and training capacity.
In an end-edge-cloud hierarchy, centralized training is constrained by limited bandwidth, motivating edge servers near IoT devices and distributed training across edge devices.
Model compression makes machine learning models lighter and faster so they fit and execute efficiently on devices with limited memory and compute power.
The survey reviews classical and deep learning from a resource-constrained edge perspective and summarizes major contributions in machine learning at the network edge.

2.1 Lighter and Faster Architectures

Lighter and faster architectures reduce the memory, parameter, and computation demands that make neural networks difficult to deploy on edge devices. The survey discusses efficient CNN and RNN designs, including depthwise separable convolutions, EfficientNet variants, and FastGRNN.

Edge deployment requires small models with few trainable parameters, low inference time, low energy use, and minimal accuracy loss.
Convolutional neural networks: Depthwise separable convolutions replace standard convolutional operations to reduce CNN model size and computational cost.
Convolutional neural networks: Depthwise separable convolutions reduce computational cost by 1/N + 1/K^2 and parameters by 1/N + (M/(K^2 ∗ M + 1)) compared with standard convolution.
Convolutional neural networks: EfficientNet achieved state-of-the-art ImageNet accuracy with 8 times less memory and 6 times faster inference, while EfficientNetLite6 enabled real-time classification on a smartphone CPU.
Recurrent neural networks: FastRNN and FastGRNN achieved 18–42 times faster prediction than compared models on an Arduino MKR1000 and were benchmarked across several problems.

2.2 Distributed Training

Distributed training keeps data and computation closer to edge devices while coordinating model updates across devices, aggregators, and cloud servers. The surveyed approaches target communication overhead, heterogeneous clients, and privacy during collaborative learning.

Distributed training can reduce communication overhead by using end-device data at edge servers and can improve privacy because end-device data is not centralized.
Wang et al. trained machine learning models at the edge without external cloud computation, achieving performance close to optimum across three datasets.
Distributed training for algorithms based on gradient descent: Gradient-based distributed learning performs local updates on edge devices, aggregates locally updated models using a weighted average, and repeats the process.
Federated learning: Federated learning keeps training data on end devices while a server aggregates updated models, but it faces communication overhead and interoperability challenges.
Federated learning: FedCS uses intelligent client selection to address slow federated learning caused by heterogeneous clients with very low resources.
Communication-efficient training: Deep gradient compression found 99.9% of gradient exchanges redundant and reduced updates 270 to 600 times, including ResNet-50 gradients from 97 MB to 0.35 MB.
Privacy-aware learning: Privacy-aware approaches use encrypted updates, differential privacy, noisy activations, or blockchains, while blockchain mining remains constrained by edge-device compute and energy requirements.

2.3 Model Compression

Model compression enables machine-learning models to run on memory- and energy-constrained IoT devices while limiting accuracy loss. The survey covers quantization, pruning, and specialized compact models, including ProtoNN and Bonsai.

Quantization reduces parameter precision, while pruning reduces parameter count, lowering memory use and potentially improving computation speed and storage efficiency.
ProtoNN projects data into a lower-dimensional space, selects representative prototypes, and jointly learns these components to compensate for accuracy loss.
For models smaller than 2 kB, ProtoNN outperformed compressed baselines; with 16–32 kB, it matched them across binary, multiclass, and multilabel datasets.
Compared with the best uncompressed models, ProtoNN was 1–2% less accurate while using 1–2 orders of magnitude less memory in multiclass and multilabel settings.
On an Arduino Uno, ProtoNN was more efficient in energy use and prediction time than existing methods, while Bonsai used 70 bytes for binary and 500 bytes for 62-class models.
Bonsai combines a shallow sparse tree with low-dimensional projection and enhanced branching to retain accuracy despite reducing model size.

2.4 Distributed deep learnings inference

Distributed inference places DNN computation across end-devices, edge-servers, and cloud infrastructure rather than relying exclusively on cloud processing. Inference can be distributed vertically across this architecture or horizontally across devices at the same level.

Distributed DNN inference places computation across end-devices, edge-servers, and cloud infrastructure.
Distributed models must keep inter-device communication costs low to support successful inference.
Inference is distributed vertically along the end-edge-cloud architecture or horizontally across multiple devices at one architectural level.
DDNNs aggregate summary information from end-devices and use jointly trained exit points for local, edge, or cloud classification.
Early exit of inference (EEoI): Early exit of inference exploits vertical resources to balance accuracy against computational effort and latency.

Mobile Deep Inference (MODI) Platform

The MODI platform dynamically selects where and which model to run according to changing inference conditions. It uses compressed models under resource constraints and can offload inference to edge-servers when networks are slow.

MODI addresses variable network conditions because fixed on-device-only and remote-only inference approaches are sub-optimal.
MODI stores compressed and uncompressed models in a centralized manager and selects models and devices using memory, bandwidth, and power.
Resource-constrained environments use compressed models, while unconstrained environments use uncompressed models for higher accuracy.
2.4 times speedup was reported for on-device inference, while MODI also offloads remote inference to edge-servers when networks are slow.
MODI runs jointly on end-devices and servers to provide suitable models based on device resources and installed applications.

Early exit of inference (EEoI)

Research on distributed inference partitions or coordinates DNN computation across devices and architectural levels to reduce latency, communication, memory, and computation costs. Reported systems combine early exits, aggregation, partitioning, layer fusion, coded computation, and resource-aware scheduling.

Distributed deep neural networks: DDNNs distribute DNN sections across end-devices and higher network levels, with joint training supporting features useful to subsequent classifiers.
Early exit of inference (EEoI): Early-exit classifiers at end-devices and edge-servers enable fast inference when lower classifiers classify samples accurately; otherwise, features continue upward.
Distributed deep neural networks: DDNN aggregation across end-devices and edge-servers reduced communication cost by a factor of more than 20 for multi-view multi-camera classification.
Horizontally-distributed inference: 2.17-4.28 times acceleration was reported for MoDNN using 2 to 4 mobile devices after partitioning convolutional and fully connected layer inputs.
Horizontally-distributed inference: Communication-aware layer fusion reduced communication cost by 14.8% and accelerated inference by 1.15 times for YOLOv2.
Horizontally-distributed inference: Coded distributed computing supported robust DNN execution and low-latency recovery from errors caused by delays in distributed computations.
Horizontally-distributed inference: Partitioning models across nearby idle robots achieved an average speedup of 6 times over local robot or remote cloud execution.

3 MACHINE LEARNING APPLICATIONS AT THE EDGE

Edge-based machine learning applications process or filter data near its source across video analytics, image recognition, traffic monitoring, drones, food recognition, and mobile applications. These systems report improvements in throughput, accuracy, latency, recall, energy use, communication, or edge-server demand.

Video analytics: Edge ML applications target real-time video analytics for self-driving cars, traffic safety, surveillance, and augmented reality.
Video analytics: Rocket uses modular vision processing and resource management across edge-devices and cloud servers, with deployments including traffic analytics.
Video analytics: Rocket has been applied to smart crosswalks, connected kitchens, traffic dashboards, and retail intelligence.
Video analytics: Drones select interesting frames locally, while edge-devices train SVMs that are sent back to predict whether DNN outputs are correct.
Video analytics: 71% greater throughput efficiency than a cloud-based model was reported for an edge-cloud object-recognition system.
Traffic and image recognition: A dashboard-camera CNN trained on 8000 car images achieved 90% accuracy for detecting moving vehicles.
Image recognition and privacy: Edge preprocessing can remove personally identifiable information or adjust image and video resolution before external upload.
Image recognition and caching: Precog predicted future classification requests and cached classifier parts on end-devices, reducing cloud offloads, latency, and communication.

3.2 Automatic Speech Recognition

Edge speech-recognition systems target offline voice assistance by adapting or compressing neural models for resource-constrained devices. Reported systems reduce computational, latency, memory, or network burdens while maintaining useful recognition accuracy.

Offline speech recognition: Keyword spotting supports offline speech recognition by extracting voice features and classifying them into keyword probabilities.The classifier generates a probability for each keyword from extracted voice features.
Deployment constraints: DNN-based keyword spotting is difficult to deploy on resource-constrained devices, motivating efficient designs such as EdgeSpeechNets.EdgeSpeechNets is described as a highly efficient DNN for this setting.
EdgeSpeechNets: Approximately 97% accuracy was achieved on Google Speech Commands with an approximately 1MB memory footprint.The result is reported for EdgeSpeechNets relative to state-of-the-art DNNs.
EdgeSpeechNets: 36 times fewer mathematical operations, 10 times lower prediction latency, 7.8 times smaller network size, and 16 times smaller memory footprint were reported for EdgeSpeechNets.These reductions were reported against state-of-the-art DNNs on an edge device.
Deep KWS: Deep KWS uses a feature extractor, deep neural network, and posterior handling module to convert audio into final frame-level recognition scores.The feature vector feeds the DNN, and posterior scores are processed into final output scores.
Deep KWS: Deep KWS achieved a reported relative improvement over traditional HMM-based solutions.The supplied passage does not include the improvement value.

3.7 Smart Homes and Cities

Edge-based machine learning systems support smart homes, factories, cities, and surveillance by processing or compressing data near its source. Applications include energy management, safety monitoring, fault detection, privacy protection, and threat response.

Smart homes: Fall detection combines skeleton extraction, ML prediction, compression on a Raspberry Pi, and cloud-based detection before notifying users.The architecture distributes processing between the edge and cloud.
Smart cities: Smart-city edge systems use SVMs to detect hazardous pipeline events and quickly shut gas supplies after leakage or fire threats.The architecture analyzes big data at the network edge across connected sensors.
Smart homes: Edge-based energy management uses weather prediction and scheduling to improve renewable-energy use in smart homes.A Raspberry Pi runs the framework at users’ locations, helping protect privacy-sensitive data.
Smart factories: An LSTM deployed on an edge device with 1 GB memory and 16 GB flash storage detects robotic-arm faults in smart factories.The system targets faults in the working sequence of a robotic arm.
Surveillance: Edge surveillance processes sensitive data locally and reduces transferred data; the MISS prototype reports both outcomes in edge-cluster testing.The system was tested on single-camera, cloud, and edge-cluster setups.
Human-machine collaboration: Human-cognition augmentation remains bounded by unresolved security, privacy, and ethics questions before deployment.The survey identifies these concerns for adaptive human-machine collaboration and memory augmentation.

4 MACHINE LEARNING FRAMEWORKS, SOFTWARE, AND HARDWARE

The survey reviews frameworks, software, and hardware for building and deploying machine learning across resource-constrained end-edge-cloud systems. Reported tools use lightweight models, distributed execution, compression, and specialized accelerators to support edge inference.

Scope: The survey covers frameworks and software for lightweight models, distributed learning, and inference across the device-edge-cloud architecture.Frameworks are summarized in Table 4, followed by hardware in Table 5.
Mobile frameworks: CoreML runs trained models on iOS edge devices without a dedicated server, while TensorFlow Lite targets embedded and mobile deployment.CoreML translates trained DNNs into a deployable CoreML format.
Model-building platforms: AI2GO has built more than 100 custom models for on-device inference and supports object detection, food classification, and other applications.The platform targets specialized hardware and supports deep learning without a cloud server.
Model-building platforms: AI2GO inference on a Raspberry Pi was reported as 2 times faster than TensorFlow Lite using MobileNets v1 SSD 0.75 depth.The comparison uses the MobileNets v1 SSD 0.75 depth model.
Distributed inference: DeepThings uses Fused Tile Partitioning to distribute CNN computation across edge devices, with YOLOv2 deployed on Raspberry Pi 3 devices.A gateway collects processed data and generates results after edge devices complete their tasks.
Compression: DeepIoT compresses neural networks by more than 90%, shortens execution time by more than 71%, and reduces energy consumption by 72.2% to 95.7%.The framework removes layer redundancy while retaining nearly the same algorithm performance.
Compression: SparseSep reportedly runs inference 13.3 times faster with 11.3 times less memory than conventional neural networks.It targets resource-constrained wearable and ARM Cortex hardware with little impact on inference accuracy.
Hardware: Specialized edge hardware includes devices such as Coral Dev Board, which was reported 31 times faster than Raspberry Pi for MobileNetV2 SSD.The same comparison reported 10 times versus Movidius NCS and 3.5 times versus Nvidia Jetson Nano (TF-TRT).

5 CHALLENGES AND FUTURE DIRECTIONS

The survey identifies data heterogeneity, non-stationarity, distributed coordination, training constraints, trust, fairness, and hardware integration as challenges for intelligent edge systems. It highlights computation-aware networking, specialized hardware, and new learning methods as future directions.

Data challenges: Edge systems must learn across heterogeneous image, text, sound, and motion data generated by diverse IoT sensors.Multimodal deep learning is one approach for learning across modalities such as audio and video.
Data challenges: Non-stationary sensor streams arise when seasonality, periodicity, or hardware and software failures change data distributions.The survey identifies novel algorithms for handling these changing streams as an important research area.
Distributed training: Distributed learning across cloud servers, edge servers, and IoT devices can provide acceptable accuracy on naturally distributed datasets, but data heterogeneity remains open.Fraud detection and market analysis are given as examples of naturally distributed datasets.
Training: Most systems train in the cloud because edge devices have limited memory and computational capability, while pruned edge-trained models often lose accuracy.Power-efficient neural-network training on edge devices remains an active research area.
Training: Future work includes mapping model sections across distributed architectures and using specialized hardware such as ARM ML processors.The goal is to speed deep-learning training and inference.
Networking: Computation results and data sharing across edge devices motivate computation-aware networking and integration with 5G URLLC services.Software-defined networking and network-function virtualization are identified as appealing research areas.
Trust and explainability: Sensitive applications require accuracy and explainability, while edge deployment is described as inherently less accurate in resource-constrained environments.The survey frames avoiding permanently harmful decisions as a primary challenge.
Fairness: Fairness methods remain affected by dataset bias, and optimizing fairness for one attribute can reduce fairness for another or reduce performance.The survey also calls for fairness criteria and optimization in ML software libraries and frameworks.

6 CONCLUSION

Edge-based ML is a fast-growing area with challenges and opportunities. This survey reviews techniques, architectures, applications, frameworks, hardware, challenges, and future directions for deploying ML at the network edge.

Edge-based ML is a fast-growing research area with numerous challenges and opportunities.
Using edge devices for ML has been shown to improve user-data privacy, security, and system response times.
The survey provides a comprehensive overview of techniques for training and deploying ML systems at the network edge.
It highlights ML architectures designed specifically for resource-constrained edge computing devices and reviews important edge-based ML applications.
The paper describes widely adopted frameworks and resource-constrained devices used to develop and deploy edge-based deep learning models.
The survey lists challenges in edge-based ML and outlines several directions for future work.