PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Deep Learning for Resource-Constrained Environments: A Survey

Hou-I Liu, Marco Galindo, Hongxia Xie, Lai-Kuan Wong, Hong-Han Shuai, Yung-Hui Li, Wen-Huang Cheng

arXiv:2404.07236v2cs.CVcs.LG

TL;DR

Deploying increasingly accurate deep learning models on mobile phones, microcontrollers, and other constrained devices remains difficult because of limited resources. This survey synthesizes lightweight architectures, compression methods, and hardware acceleration, while examining TinyML and edge-device LLM deployment. It concludes that practical lightweight design requires balancing accuracy, efficiency, hardware compatibility, and the challenges of highly constrained deployment settings.

  • Problem

    Resource-constrained devices cannot readily deploy increasingly large and complex deep learning models, creating a need for accurate yet lightweight designs.

  • Method

    The survey reviews lightweight architectures, pruning, quantization, knowledge distillation, neural architecture search, hardware acceleration, and deployment techniques.

  • Results

    The survey finds no one-size-fits-all lightweight architecture: accuracy and efficiency trade off, and model selection must account for hardware compatibility.

  • Takeaways & Limitations

    Effective lightweight deep learning requires coordinated choices across model design, compression, and hardware, with quantization techniques selected according to the deployment environment.

  • Takeaways & Limitations

    TinyML remains constrained by limited resources, hardware and software heterogeneity, and a lack of datasets suited to edge-device sensor data.

Abstract

from arXiv · show

Over the past decade, the dominance of deep learning has prevailed across various domains of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. While there have been remarkable improvements in model accuracy, deploying these models on lightweight devices, such as mobile phones and microcontrollers, is constrained by limited resources. In this survey, we provide comprehensive design guidance tailored for these devices, detailing the meticulous design of lightweight models, compression methods, and hardware acceleration strategies. The principal goal of this work is to explore methods and concepts for getting around hardware constraints without compromising the model's accuracy. Additionally, we explore two notable paths for lightweight deep learning in the future: deployment techniques for TinyML and Large Language Models. Although these paths undoubtedly have potential, they also present significant challenges, encouraging research into unexplored areas.

1 INTRODUCTION

Lightweight deep learning addresses the rising resource demands of increasingly large models by integrating architecture design, compression, and hardware acceleration. The survey also examines emerging TinyML and edge-device LLM deployment challenges.

  • Motivation: Deep learning models have grown substantially in complexity and resource demands while achieving higher accuracy across AI applications.Examples include Model Soups with over 1,843 million parameters and GPT-4 with 1.76 trillion parameters.
  • Motivation: Green AI concerns highlight the environmental costs of models requiring substantial GPU resources and training time.Lightweight and edge devices are increasingly relevant because of portability, automation, and energy-efficiency needs.
  • Survey scope: The survey analyzes lightweight deep learning across neural architecture design, compression methods, and hardware acceleration.This integrated pipeline distinguishes the survey from prior work focused on narrower parts of the process.
  • Survey scope: The review covers lightweight architecture families, compression algorithms, hardware acceleration methods, and their interconnections.It organizes architectures chronologically and discusses techniques including quantization, pruning, knowledge distillation, and neural architecture search.
  • Future directions: The survey extends its scope to TinyML and the deployment of large language models on resource-constrained edge devices.These directions are presented alongside their current challenges and opportunities for future research.

2 LIGHTWEIGHT ARCHITECTURE DESIGN

The survey introduces lightweight architecture fundamentals and organizes CNN architectures into series to clarify their historical development and design advantages.

  • Architecture organization: The survey first presents fundamental lightweight architecture knowledge and groups CNN architectures into series arranged chronologically.This organization is intended to clarify the evolution of lightweight design and the advantages of different architectures.

2.1 Prior Knowledge of Lightweight Architecture

Lightweight architecture evaluation requires considering computation, memory, and inference speed together. The section reviews core convolutional building blocks and their efficiency trade-offs.

  • Evaluation metrics: FLOPs, MACs, and Memory Access Cost are commonly used to measure computational operations and memory footprint.FLOPs approximately equal twice MACs, while Memory Access Cost corresponds to RAM usage.
  • Evaluation metrics: The MAC equation combines feature-map and weight memory terms for a convolution layer.The first term represents feature-map memory, while the second represents weight memory.
  • Inference speed: Throughput measures inferences per second, whereas latency measures the time between input arrival and output generation.These metrics characterize inference speed from complementary perspectives.
  • Convolution operations: Pointwise convolution uses 1×1 kernels to adjust channel dimensions and can reduce FLOPs when inserted at architectural bottlenecks.Its computation time and information flow make it an important lightweight design component.
  • Convolution operations: Group convolution divides feature-map channels into groups and applies convolutions separately, reducing computational complexity by N times.Here, N denotes the number of groups.
  • Convolution operations: Group convolution can restrict information flow and add MAC-related inference costs, motivating feature sharing, learned grouping, and adaptive assignment.ShuffleNet, CondenseNet, and Dynamic Group Convolution address these limitations in different ways.
  • Convolution operations: Depthwise separable convolution applies depthwise convolution followed by pointwise convolution, saving computation while potentially increasing execution time.Later work aggregates multiple kernel sizes into one depthwise convolution and uses AutoML to search the design space.

2.2 Lightweight CNN Architecture

Lightweight CNN architectures reduce computation, parameters, or memory demands through specialized convolutional structures, pruning, shifting, and hardware-aware design. The survey emphasizes that efficiency gains involve trade-offs among accuracy, computation, memory access, and device compatibility.

  • SqueezeNet series: SqueezeNet uses pointwise convolutions and a fire module, while SqueezeNext factorizes 3×3 kernels into 3×1 and 1×3 kernels.The factorization reduces kernel parameters from k^2 to 2k.
  • SqueezeNet series: 50× and 112× parameter reductions let SqueezeNet and SqueezeNext retain AlexNet-level ImageNet accuracy.
  • ShuffleNet series: ShuffleNet improves group convolutions by shuffling channels to exchange information across groups, while ShuffleNetV2 provides practical guidelines targeting memory efficiency and MAC reduction.
  • CondenseNet series: CondenseNet prunes less important grouped connections using weight-based importance and structured sparsity, removing connections with small sparsity values.
  • CondenseNet series: CondenseNetV2 dynamically learns connections during training and reactivates previously pruned features, addressing fixed connectivity and improving feature reuse.Its blocks can reactivate earlier pruned connections while removing previously active ones.
  • Shift-based series: Shift convolution replaces spatial multiplications with feature-map shifts, achieving zero parameters and FLOPs, while later variants target learnability, reduced memory movement, and hardware efficiency.Shift-based networks offer greater hardware efficiency but lower expressive capacity than multiplication-based networks; ShiftAddnet combines bit-wise shifts with additive operations.

2.3 Transformer-based Series

Transformer-based models offer strong accuracy but impose substantial parameter, computation, and memory demands, especially for long or high-resolution inputs. The survey reviews efficient attention, lightweight architectures, token reduction, and hybrid or distillation-based designs to address these costs.

  • Transformer motivation: Transformers require many parameters and high MAC, making training and inference costly, particularly for long input sequences.Their internal computation and network structures are also more complex than those of CNNs.
  • Efficient attention: Efficient attention methods reduce self-attention complexity from N^2 to N or N log(N) using low-rank projection or locality-sensitive hashing.Linformer uses linear projection, while Reformer replaces dot products with locality-sensitive hashing.
  • Lightweight transformer blocks: Grouped linear transformations make transformer feed-forward networks more lightweight by applying grouped computation to their dense connections.Hierarchical group transformation further aims to improve information flow between groups.
  • Vision transformers: Image resolution strongly affects ViT token computation because images are split into patch embeddings, with ImageNet inputs requiring more computation than CIFAR inputs.T2T-ViT uses soft unfolding to preserve surrounding spatial relationships while enabling smaller MLP sizes and greater memory efficiency.
  • Token reduction: Token pruning methods reduce transformer computation by masking less significant tokens or retaining the top-K tokens according to predicted importance or attentiveness.DynamicViT, EViT, and A-ViT use token-importance mechanisms to adapt computation.
  • Hybrid and distillation models: Knowledge distillation and CNN-transformer hybrids improve the efficiency of transformer models by combining transformer representations with CNN inductive biases.DeiT-B achieves 84.5% Top-1 accuracy using a CNN teacher and transformer student, although transformer students may require large networks to maintain performance.

3 FUNDAMENTAL METHODS IN MODEL COMPRESSION

The survey organizes model compression as a group of methods for designing efficient deep-learning models and examines their characteristics and improvements over time.

  • Compression methods: The survey covers pruning, quantization, knowledge distillation, and neural architecture search as widely adopted model-compression methods.It provides a detailed exploration of each method based on its distinctive characteristics.

3.1 Pruning

Pruning compresses neural networks by removing redundant weights, channels, filters, or other components. The survey contrasts unstructured and structured pruning and emphasizes balancing accuracy, FLOPs reduction, and hardware compatibility.

  • 3.1 Pruning: Pruning methods compress and accelerate neural networks by removing redundant weights and are categorized as unstructured or structured.Unstructured methods remove individual weights, whereas structured methods remove components while preserving regularity.
  • 3.1.1 Unstructured pruning: Unstructured pruning sets selected weights to zero without physically removing their nodes, producing zero multiplications but potentially irregular network structures.Second-order pruning methods can assess weight importance but may require substantial computational power.
  • 3.1.1 Unstructured pruning: Pruning weights before network convergence can significantly reduce accuracy because deleted connections cannot be restored.Splicing can recover connections later identified as important, while single-shot pruning removes weights before training begins.
  • 3.1.2 Structured pruning: Structured pruning removes components from pretrained networks while preserving regular structure, including filters and channels.Methods select components using criteria such as norms, Hessian-modeled dependencies, layer grouping, or Fisher information.
  • 3.1.2 Structured pruning: The most useful structured-pruning method balances pruned FLOPs against accuracy loss rather than maximizing either quantity alone.GFP achieves the highest pruned accuracy with 50.6% FLOPs reduction, whereas ASTER removes the most FLOPs without the best pruned accuracy.

3.2 Quantization

Quantization compresses model weights and activations by mapping them to finite, lower-precision values, but calibration and precision choices determine the balance between resource savings and accuracy. The survey contrasts quantization representations, training strategies, and empirical trade-offs.

  • Quantization representations: Symmetric quantization uses a range where -α = β, whereas asymmetric quantization allows -α ≠ β.The representation also uses a real-valued scale and, for asymmetric quantization, an integer zero point.
  • Calibration: Dynamic calibration computes the range for each feature map and is more accurate but computationally demanding; static calibration is lighter but less accurate.Static calibration estimates ranges from typical values after several iterations.
  • Training strategies: Post-training quantization compresses pretrained models effectively, whereas quantization-aware training simulates inference-time errors during training to mitigate accuracy loss.QAT uses FakeQuant so the model can adjust to quantization errors before deployment.
  • Accuracy–compression trade-off: Binarized networks target 32x compression and speedup but incur significant accuracy drops, while most 4-bit approaches preserve accuracy with less loss.One approach reduced model size approximately 8-fold, from 42.56 MB to 5.37 MB; theoretical speedups may be reduced by quantization and dequantization operations.

3.3 Knowledge Distillation (KD)

Knowledge distillation transfers information from larger or differently trained models to smaller models, with offline, online, and self-distillation differing in teacher availability and training strategy. The survey compares their implementation constraints and reported outcomes.

  • KD foundations: Knowledge distillation transfers knowledge from a large teacher network to a smaller student using teacher predictions alongside ground-truth labels.In the conventional setup, the teacher is pretrained before student training.
  • KD categories: Offline distillation trains teacher and student sequentially, online distillation trains them concurrently, and self-distillation uses the student model as its own teacher.These categories differ primarily in teacher definition and training strategy.
  • Offline distillation: Offline distillation requires a pretrained teacher, creating unavoidable time and computational overhead before student training.This requirement can make offline methods infeasible when a sufficiently large teacher cannot be obtained.
  • Online distillation: Online distillation removes the need for a pretrained teacher by concurrently training a cohort of networks that incorporate one another’s predictions.Deep Mutual Learning allows every network to benefit from knowledge exchanged within the cohort.
  • Self-distillation: Self-distillation can improve results and reduce training time because it transfers knowledge within a single model without requiring additional networks.The surveyed methods use deeper layers, earlier epochs, or other internal representations as instruction sources.
  • Empirical comparison: On CIFAR-100, WRN-28-10 improved 0.27% from 78.69% to 78.96% after online distillation, while PS-KD yielded 3.36% gains for ResNet18.The survey notes that comparisons depend heavily on experimental details and recommends combining self-distillation with other compression or distillation methods when needed.

3.4 Neural Architecture Search (NAS)

Neural architecture search automates choices over model hyperparameters by combining a search space, search algorithm, and performance evaluation strategy. The survey covers reinforcement-learning, evolutionary, gradient-based, and hardware-aware approaches.

  • NAS components: NAS searches combinations of kernel size, channel size, stride, depth, and other hyperparameters to construct architectures suited to task-specific performance criteria.Evaluation may use Top-1 or Top-5 classification scores, or AP and F1 for object detection.
  • Search algorithms: NAS search algorithms include random search, reinforcement learning, evolutionary algorithms, and gradient optimization, each trading search efficiency against exploration scope.The survey organizes methods according to these search strategies.
  • RL-based NAS: RL-based NAS uses an RNN controller to generate candidate hyperparameters, receives child-network rewards such as accuracy or AP, and updates iteratively.The reward guides refinement of subsequent candidate architectures.
  • Evolutionary NAS: Evolutionary NAS maintains a population by evaluating models, removing weaker candidates, and mutating stronger parent architectures to produce new children.Repeated training and validation assessment progressively updates the population.
  • Gradient-based NAS: Gradient-based NAS converts a discrete search space into a continuous differentiable one, enabling gradient descent and reducing black-box search difficulties.DARTS follows this strategy, while FBNet extends differentiable NAS with hardware awareness through a latency lookup-table pipeline.
  • Hardware-aware NAS: Hardware-aware NAS incorporates device constraints such as latency, energy, and memory through lookup tables that estimate layer or architecture costs.NetAdaptV1 pre-measures layer latency and records it by layer structure to simplify search for pretrained networks.

3.5 Discussion and Summary

The survey’s practical guidance links compression choices to hardware capabilities and emphasizes matching methods to deployment requirements and computational resources. It also identifies algorithmic and hardware-accessibility challenges for deploying ViTs on constrained devices.

  • Pruning: Structure pruning is often preferable to unstructured pruning because irregular sparse structures can be incompatible with hardware accelerators.The discussion presents structure pruning as a practical response to accelerator compatibility constraints.
  • Quantization: Quantization should be selected according to deployment hardware, with full-integer quantization essential on MCUs or edge TPUs that support only integer operations.TensorFlow Lite can reduce model size by up to four times and accelerate inference by more than three times.
  • Practical guidance: The appropriate lightweight approach depends on task requirements and available computational resources.The survey frames method selection as deployment-specific rather than universally optimal.

4 HARDWARE ACCELERATION OF DEEP LEARNING MODELS

Hardware acceleration makes deep learning more practical on constrained devices by combining accelerator architectures, dataflow and locality optimization, and deployment libraries. The survey emphasizes that efficient deployment requires balancing computational performance, memory movement, energy use, and hardware–software co-design.

  • Hardware accelerators: GPUs, FPGAs, ASICs, and TPUs provide distinct hardware acceleration options based on temporal or spatial architectures.Temporal architectures centralize control and local memory, whereas spatial architectures distribute processing across processing elements.
  • Hardware accelerators: TPU-v4 achieves 2.7 times better performance/watt and tenfold higher speed than TPU-v3.TPU-v4 scales to 4096 chips and uses optical circuit switches to restructure interconnection topology.
  • Dataflow and locality: Dataflow optimization addresses CNN memory and energy constraints by organizing computation across processing elements and reducing redundant external-memory access.The survey categorizes pipeline-like, DaDianNao-like, systolic-array-like, and streaming-like dataflows.
  • Dataflow and locality: Data locality optimization uses loop unrolling, tiling, and interchange to increase reuse and parallelism under limited on-chip memory.These transformations can improve utilization and speed but may increase code size, memory use, or mapping complexity.
  • DL libraries: DL libraries provide high-level APIs that simplify implementation, design, and training while supporting GPU acceleration and automatic differentiation.The survey includes popular libraries as part of the deployment pipeline.
  • Co-Design of Hardware Architecture: Hardware–software co-design can evaluate CNN–accelerator pairs directly or optimize hardware around a candidate CNN, with supernets reducing repeated training.The survey presents these as two approaches for jointly searching neural architectures and hardware settings.

5 CHALLENGE AND FUTURE WORK

The survey identifies unresolved deployment challenges for TinyML, LLMs, diffusion models, and ViTs in resource-constrained environments. Future progress depends on addressing severe resource limits, platform heterogeneity, compression difficulties, and missing hardware support.

  • 5.1 TinyML: TinyML enables deep learning on IoT devices consuming less than 1mW, but extremely constrained hardware and platform-dependent libraries limit general-purpose deployment.TinyML commonly uses MCUs with very small SRAM and flash memory, while libraries such as CMSIS-NN and TinyEngine are often device-specific.
  • 5.1 TinyML: TinyML solutions remain difficult to scale because hardware and software heterogeneity requires tailoring implementations to individual devices.The survey also identifies a lack of datasets as an inherent constraint.
  • 5.2 Large Language Models: Large language models typically require GPU-level hardware and tens of gigabytes of memory for inference, complicating deployment on mobile devices.The survey notes that their varied weight distributions and homogeneous embeddings make quantization difficult.
  • 5.2 Large Language Models: Quantization and pruning are being used to construct lightweight LLMs, including one-step pruning of GPT models to at least 50% sparsity with minimal accuracy loss.The survey presents these techniques as active directions for reducing LLM resource demands.
  • 5.2 Diffusion Models: Diffusion-model deployment on edge devices is challenging because compression can simplify or remove components and severely degrade image quality.Post-training quantization offers a training-free route to improve denoising efficiency and reduce weight storage, but further work remains.
  • 5.2.4 Deployment of Vision Transformers (ViTs): ViT inference on mobile devices can have latency and energy consumption up to 40 times higher than CNN inference.The survey attributes the computational burden to quadratic growth in self-attention operations and calls for algorithm and hardware optimization.
  • 5.2.4 Deployment of Vision Transformers (ViTs): ViT deployment requires reducing computational bottlenecks and expanding hardware accessibility because important transformer operators lack support on some mobile accelerators.Examples include mobile-GPU concatenate constraints and missing LayerNorm support on Intel VPU.

6 CONCLUSION

The survey examines lightweight architectures, compression methods, and hardware acceleration for deploying deep learning in resource-constrained environments. It also considers hardware–software co-design and future deployment challenges for TinyML and large models.

  • 6 CONCLUSION: The survey examines lightweight models and optimization strategies including MobileNet, efficient transformers, pruning, and quantization.It frames these techniques within growing concerns about energy conservation, carbon footprint, and cost-effectiveness in edge AI.
  • 6 CONCLUSION: Hardware acceleration coverage includes architectures, dataflow types, data locality optimization, and libraries for training and inference.The survey emphasizes the interplay between hardware and software in expediting deep learning processes.
  • 6 CONCLUSION: Deploying deep learning models in resource-constrained environments remains challenging, motivating further investigation of TinyML and LLM techniques.These future directions are presented alongside unresolved deployment issues.
Loading 2404.07236v2…