PaperlayerView exact revision on arXiv

Source-linked AI summary

Bringing AI To Edge: From Deep Learning's Perspective

Di Liu, Hao Kong, Xiangzhong Luo, Weichen Liu, Ravi Subramaniam

arXiv:2011.14808v1cs.LGcs.AI

TL;DR

Edge intelligence faces a computational gap between computation-intensive DNNs and less-capable edge systems, while prior reviews cover only subsets of the relevant techniques. This paper surveys lightweight architectures, compression, hardware-aware NAS, and adaptive models, then discusses future directions from observations and experiments. It highlights that FLOPs and latency can diverge substantially across architectures and hardware.

  • Problem

    Edge intelligence faces a computational gap between computation-intensive DNNs and less-capable edge systems, while prior reviews cover only subsets of the relevant techniques.

  • Method

    The paper synthesizes lightweight architectures, compression methods, knowledge distillation, hardware-aware design, adaptive models, and edge-learning approaches.

  • Results

    Models with the same latency can differ in FLOPs by up to 26%, while models with the same FLOPs can range from below 80ms to 120ms latency on Nvidia Jetson Xavier.

  • Takeaways & Limitations

    Edge DNN design should consider hardware-relevant metrics beyond accuracy and FLOPs, including latency, power, and security.

  • Takeaways & Limitations

    The review focuses on models trained on powerful servers and deployed for inference, while on-device learning remains challenging because training is more computation- and memory-intensive.

Abstract

from arXiv · show

Edge computing and artificial intelligence (AI), especially deep learning for nowadays, are gradually intersecting to build a novel system, called edge intelligence. However, the development of edge intelligence systems encounters some challenges, and one of these challenges is the \textit{computational gap} between computation-intensive deep learning algorithms and less-capable edge systems. Due to the computational gap, many edge intelligence systems cannot meet the expected performance requirements. To bridge the gap, a plethora of deep learning techniques and optimization methods are proposed in the past years: light-weight deep learning models, network compression, and efficient neural architecture search. Although some reviews or surveys have partially covered this large body of literature, we lack a systematic and comprehensive review to discuss all aspects of these deep learning techniques which are critical for edge intelligence implementation. As various and diverse methods which are applicable to edge systems are proposed intensively, a holistic review would enable edge computing engineers and community to know the state-of-the-art deep learning techniques which are instrumental for edge intelligence and to facilitate the development of edge intelligence systems. This paper surveys the representative and latest deep learning techniques that are useful for edge intelligence systems, including hand-crafted models, model compression, hardware-aware neural architecture search and adaptive deep learning models. Finally, based on observations and simple experiments we conducted, we discuss some future directions.

I. INTRODUCTION

Edge intelligence brings computation-intensive deep learning to diverse, resource-constrained edge systems, creating a computational gap that software techniques seek to bridge. This paper reviews lightweight models, compression, hardware-aware NAS, and adaptive models as complementary approaches.

  • Motivation: Cloud inference can suffer from latency and privacy problems, which are especially critical for applications such as self-driving cars with rigorous real-time constraints.
  • Motivation: Edge systems range from powerful computation units to small, battery-powered devices, with scenario-specific latency, power, and capability constraints.
  • Motivation: DNNs are increasingly deep and wide, with billions of parameters and millions of FLOPs, creating a computational gap with less-capable edge systems.
  • Challenges: Hardware accelerators can bridge the gap, but chip development costs 30-80 million dollars and takes 2-3 years.
  • Challenges: Customized accelerators may sacrifice generality, while poor ecosystems and programmability can prevent performance gains from compensating for engineering effort.
  • Paper scope: The review covers lightweight DNN design, model compression, hardware-aware NAS, and adaptive models, addressing gaps in earlier reviews that treated only subsets of these topics.

B. Structure of This Paper

The paper combines a focused survey of edge-oriented deep learning techniques with preliminaries and future-design observations. It emphasizes computer vision models and organizes the discussion around lightweight architectures, compression, NAS, adaptive computation, and future directions.

  • Scope and contribution: The review complements existing edge-computing surveys by focusing on deep learning techniques for efficient DNN deployment on edge systems.
  • Organization: The paper covers lightweight architectures, network pruning and quantization, knowledge distillation, hardware-aware NAS, adaptive DNNs, and future edge-DNN design.
  • Preliminaries: The presentation primarily uses computer vision because its edge-intelligence models are more mature than lightweight NLP models.
  • Organization: The paper includes peer-reviewed work through 2020 and uses a schematic structure to organize its sections.
  • Preliminaries: The preliminaries introduce DNN models and explain that convolutional layers are the major resource and time cost in CNNs.
  • Preliminaries: For computer vision, the paper uses DNN and CNN interchangeably and presents convolution, pooling, activation, and fully connected layers.

III. LIGHTWEIGHT NETWORK DESIGN

Lightweight network design develops efficient DNN architectures for resource-constrained edge systems while preserving predictive accuracy. The surveyed models reduce computation through specialized modules, efficient convolutions, channel operations, or coordinated architectural scaling.

  • Motivation: Lightweight architectures target efficient edge deployment without substantially compromising predictive accuracy.The section presents novel architectures and operators as a response to the computational complexity of increasingly large DNNs.
  • SqueezeNet: SqueezeNet splits convolution into squeeze and expand layers, replacing some 3 × 3 convolutions with lower-complexity 1 × 1 operations.Its applicability was validated on FPGA hardware with limited on-chip memory.
  • MobileNet: MobileNet reduces computational cost by replacing conventional convolution with depth-wise separable convolution.The operation factorizes k × k × n convolution into depth-wise and point-wise convolutions.
  • Efficient architectures: ShuffleNet uses group convolution and channel shuffle, while ShuffleNetV2 uses channel split to improve efficiency and accuracy.GhostNet instead generates additional features from a small set of intrinsic features based on feature correlation.
  • EfficientNet: EfficientNet uses compound scaling across depth, channel width, and resolution to improve predictive performance under a target FLOPs budget.The resulting scaled models achieve comparable or better accuracy with fewer parameters.

B. Object Detection

Efficient object detection on resource-limited devices primarily reduces backbone or detector complexity using lightweight architectures. Two-stage methods remain less efficient, while one-stage methods offer an accuracy–efficiency trade-off.

  • Detection approaches: Object detection identifies object categories and locations, using one-stage or two-stage methods.One-stage methods predict classification and localization in one forward pass, whereas two-stage methods first generate regions before further processing.
  • One-stage efficiency: Lightweight detection methods reduce backbone complexity or combine SSD with MobileNet, ShuffleNet, or SqueezeNet.Examples include Tiny-DSOD, Tiny-SSD, Tiny-YOLO, and SqueezeDet.
  • Two-stage efficiency: Two-stage efficiency efforts modify heavy detection heads or jointly configure input resolution, backbone, and detection head.ThunderNet uses an SNet variant of ShuffleNet, while EfficientDet combines EfficientNet with a bidirectional feature pyramid network.
  • Limitations: Two-stage methods provide high accuracy at the cost of efficiency and may fail to guarantee real-time constraints even on powerful GPUs.The passage gives 25fps for video as an example of a real-time requirement.
  • Compression context: Network compression removes redundancy through pruning, quantization, or knowledge distillation to reduce DNN complexity.These approaches remove weights or channels, use fewer bits, or train a compact student from an over-parameterized teacher.

A. Network Pruning

Network pruning reduces redundancy in over-parameterized DNNs through fine-grained weight removal or structured channel and filter removal. Its benefits depend on the pruning strategy and hardware: sparse reductions may not improve latency, whereas regular structures can run directly on conventional platforms.

  • Overview: Network pruning removes redundant weights or channels from over-parameterized DNNs to reduce computational complexity and storage requirements.The surveyed methods are divided into non-structure and structure pruning according to the removed structure.
  • Non-structure pruning: Non-structure pruning removes individual weights or neurons, significantly reducing parameter count and memory footprint.Deep Compression combines pruning with quantization and Huffman encoding for further model compression.
  • Non-structure pruning: Non-structure pruning methods include direct pruning of pretrained networks, variational drop-rate approaches, and grow-prune schemes such as NeST.These methods differ in how they identify and remove irrelevant weights.
  • Energy-aware pruning: Energy-aware pruning orders layers by energy consumption and prunes weights according to that order.This extends non-structure pruning beyond model size toward energy reduction.
  • Non-structure pruning: Non-structure pruning can reduce memory footprint and MACs without directly improving latency because sparse structures create irregular access patterns.Efficient execution may require compressed sparse formats and specialized hardware or software libraries.
  • Structure pruning: Structure pruning removes channels and filters while maintaining regularity, enabling inference acceleration on off-the-shelf hardware without specialized software support.Its common process defines a criterion, selects channels under a target, and fine-tunes the pruned model to retain accuracy.
  • Structure pruning: Structure pruning methods vary by training strategy, pruning scope, and selection mechanism, including training-based, inference-based, layer, global, rule-based, and learning-based approaches.Learning-based methods include reinforcement learning, evolutionary algorithms, and gradient-based optimization.
  • Hardware awareness: Hardware-aware pruning targets direct platform metrics such as latency because FLOPs and compression ratio do not necessarily predict performance improvement.NetAdapt uses a hardware-specific lookup table, while SIMD-aware pruning selects strategies according to hardware parallelism.

B. Quantization

Quantization compresses DNN weights and activations into lower-precision representations to reduce edge-device cost, using uniform, mixed-precision, binary, ternary, and post-training approaches. The section also describes accuracy, convergence, robustness, and hardware-execution considerations.

  • Quantization overview: Quantization encodes full-precision weights and activations with fewer bits, such as FP16, INT8, or binary representations.It can reduce computational cost and memory requirements while preserving accuracy in some settings.
  • Hardware-aware quantization: Mixed-precision quantization exploits hardware supporting multiple arithmetic widths, including 1-bit, 4-bit, 8-bit, and 16-bit operations.HAQ is described as a hardware-aware automated quantization approach.
  • Binary and ternary quantization: Binary and ternary methods constrain weights to {−1,+1} or {−1,0,+1}, enabling compact representations and cheap bit operations on dedicated hardware.Ternary quantization uses a threshold and scaling factor, while XOR-Net simplifies binary dot-product instructions.
  • Limitations: Binarization can make training harder to converge and substantially degrade model performance because the parameter space is highly discrete.The section motivates improved binarization and quantization methods to reduce this loss.
  • Post-training quantization: Post-training quantization converts FP32 activations and weights into low-precision formats such as INT8 using an encoding algorithm.PyTorch and TensorFlow support this prevalent quantization workflow.
  • Edge deployment: Quantization is presented as a standard way to compress memory-intensive DNNs for resource-constrained edge systems while supporting robust and efficient execution.The discussion mentions defensive quantization, energy-oriented quantization, and deployment tools such as TensorRT, TensorFlow Lite, and OpenVINO.

C. Knowledge Distillation

Knowledge distillation trains a compact student model with information from a larger teacher model, combining hard-label supervision with teacher-derived soft targets. The surveyed work highlights transferability to small models and potential usefulness for edge systems with limited data.

  • Core concept: Knowledge distillation trains a compact student model with assistance from a complicated, pretrained teacher model.The student uses conventional hard-target training while also learning from teacher probabilities.
  • Training objective: The distillation objective combines ground-truth cross-entropy with a teacher-student loss based on temperature-scaled outputs.The formulation uses balancing factors α and β and student and teacher logits z_s and z_t.
  • Extensions: Knowledge distillation can use multiple teachers, weighted teacher distributions, and intermediate representations rather than only final-layer outputs.FitNets transfers knowledge from a shallower, wider teacher to a deeper, thinner student.
  • Benefits for edge systems: For a given dataset, small models learned through knowledge distillation outperform small models trained from scratch.The paper identifies this transferability as useful when edge systems lack large amounts of high-quality training data.
  • Future integration: The survey anticipates combining knowledge distillation with other techniques, such as neural architecture search, to derive accurate compact networks.It also notes reported applications to robust-network design.

V. HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH

Hardware-aware NAS searches for DNN architectures tailored to target platforms rather than relying only on platform-independent complexity metrics. The surveyed methods incorporate latency, accuracy, energy, memory, and search-cost considerations, but the search process can remain resource-intensive.

  • Hardware-aware search: MnasNet incorporates measured target-device latency and accuracy into a reinforcement-learning reward for hardware-aware architecture search.MnasFPN extends this approach to object-detection networks.
  • Search strategies: Hardware-aware NAS methods use reinforcement learning, evolution, predictors, or gradient-based optimization to search large architecture spaces.ChamNet uses evolution with energy, accuracy, and latency predictors, while ProxylessNAS reduces search cost with gradients.
  • Search cost: RL- and evolution-based NAS can require thousands of GPU days, and MnasNet is estimated to need 40,000 GPU hours.This cost limits hardware-specific architecture search for users with constrained resources.
  • Broader design space: Some methods search beyond operators by jointly exploring network width and depth, while others target microcontrollers or multiple hardware platforms.DenseNAS searches width and depth; SpArSe targets small-memory microcontrollers; OFA samples different network sizes from a supernetwork.
  • Motivation: Conventional NAS can produce inefficient edge models because FLOPs and MACs do not reliably reflect performance across hardware platforms.Existing methods may also prioritize accuracy while overlooking energy and latency.

VI. ADAPTIVE MODELS

Adaptive DNN models change their computation according to input data and runtime resource conditions, addressing inference-time variability on edge systems. The section organizes adaptivity around selective blocks, channels, early exits, branches, attention, and execution paths.

  • Motivation: Runtime resource sharing can change DNN inference time, creating quality-of-service problems and risks for applications with strict timing requirements.Adaptive models respond to input data and system status such as low-power mode.
  • Adaptive principle: Adaptive networks selectively activate channels or layers per input because not every input requires the full model.This selective execution can improve efficiency compared with always activating the complete network.
  • Block and channel adaptivity: Block/layer-adaptive and channel-adaptive methods execute only portions of the network’s blocks, layers, or channels.Runtime Network Pruning uses reinforcement learning to select filters dynamically according to input data.
  • Early exit: Early-exit methods use intermediate representations from early layers to predict simple inputs and skip later computation.Adaptive Neural Network learns when an early prediction is sufficient.
  • Multi-branch and path adaptivity: Multi-branch and path-selection methods choose kernels, branches, or graph execution paths on a per-input basis.HydraNet selects branches, while D2NN selects an efficient path through a directed acyclic graph using reinforcement learning.
  • Attention: Attention-based methods reduce computation by applying expensive convolutions only to spatially important image areas.The figure description identifies the red line as highlighting the area of interest.

VII. DISCUSSION AND ENVISIONS

Edge intelligence brings AI closer to data and users, but its diverse scenarios and hardware constraints create a broad design space. The paper identifies directions for making edge intelligence more capable and accessible.

  • The real design space for edge intelligent systems contains diverse scenarios with different design concerns.
  • Edge intelligence combines edge computing and artificial intelligence to support intelligent services near data producers and end-users.
  • The paper uses observations to identify important topics expected to advance DNNs on edge systems.

A. Different metrics oriented DNN models

Edge DNN design must balance accuracy, latency, power, and security rather than relying on accuracy alone. FLOPs are an imperfect proxy for latency, motivating more careful or combined metrics.

  • Different metrics oriented DNN models: Edge DNN design must consider latency, power, and security alongside accuracy.These metrics matter when models are deployed on real systems, where adversarial vulnerability is also a concern.
  • Different metrics oriented DNN models: Models with the same latency can differ in FLOPs by up to 26% on Nvidia Jetson Xavier.At 120ms latency, the cited architectures range from 461 FLOPs to 623 FLOPs.
  • Different metrics oriented DNN models: Models with the same FLOPs can show latency ranging from below 80ms to 120ms.The result demonstrates that FLOPs do not directly determine latency across architectures and hardware.
  • Different metrics oriented DNN models: The paper argues that edge DNN design needs careful use of indirect metrics and potentially combined metrics.The large design space and multiple concerns make exhaustive search costly and may produce sub-optimal designs.

B. Hardware-software co-design

Hardware-software co-design is needed because DNNs and accelerators have interacting design spaces and diverse hardware characteristics. Hardware-aware NAS demonstrates that resource constraints can guide efficient model design.

  • Hardware-software co-design: Different hardware accelerators have diverse characteristics, while many DNNs are designed without considering the underlying hardware.Such designs may fail to fully utilize available hardware.
  • Hardware-software co-design: FNAS designs models for different FPGA resource constraints using loose, medium, and tight patterns.The cited platforms include low-end 7Z020 and high-end 7A50T FPGAs.
  • Hardware-software co-design: FNAS achieves the same accuracy level under different resource constraints while reducing search cost and inference latency.
  • Hardware-software co-design: The large design space of edge intelligent systems requires hardware-software co-design for effective and efficient implementation.
  • Hardware-software co-design: Effective DNN design needs constrained design spaces, hardware modeling, guiding metrics such as Roofline, and benchmarks such as MLPerf and ParaDNN.

C. Lightweight models for other applications

Edge intelligence research has focused mainly on computer vision, while complex models from NLP and other domains remain promising targets. The paper also highlights unresolved challenges in local learning, data scarcity, distribution shifts, and privacy-preserving collaboration.

  • Lightweight models for other applications: Most edge deep-learning research targets computer-vision tasks, including image classification and object detection.Examples include face recognition, UAV tracking, robot navigation, and video analytics.
  • Lightweight models for other applications: Complex models such as BERT indicate substantial potential for extending edge intelligence beyond computer vision.The passage identifies NLP and machine translation as other domains with successful DNN models.
  • Lightweight models for other applications: On-device learning remains at an early stage because edge training is more computation-intensive and memory-hungry than inference.Limited energy supply further complicates local training.
  • Lightweight models for other applications: Limited large-scale labeled data makes transfer learning important for many edge applications.The paper gives wildlife surveillance and manufacturing defect detection as examples where data collection and labeling are difficult or expensive.
  • Lightweight models for other applications: Incremental and federated learning can update edge models for distribution shifts, new classes, and privacy-preserving collaboration.Federated learning shares learned gradients rather than centralized raw data.
  • Lightweight models for other applications: The paper surveys lightweight models, network compression, hardware-aware NAS, and adaptive models to support efficient DNN deployment at the edge.
Loading 2011.14808v1…