PaperlayerView exact revision on arXiv

Source-linked AI summary

From Tiny Machine Learning to Tiny Deep Learning: A Survey

Shriyank Somvanshi, Md Monzurul Islam, Gaurab Chhetri, Rohit Chakraborty, Mahmuda Sultana Mimi, Sawgat Ahmed Shuvo, Kazi Sifatul Islam, Syed Aaqib Javed, Sharif Ahmed Rafat, Anandi Dutta, Subasish Das

arXiv:2506.18927v2cs.LG

TL;DR

Cloud-based inference can impose latency, privacy, and energy costs, while TinyDL must also address constrained hardware and changing data. This survey synthesizes TinyDL architectures, hardware, optimization, software, applications, and deployment strategies, concluding that edge intelligence is moving toward more autonomous and efficient systems while retaining important memory and compute limitations.

  • Problem

    Cloud inference introduces latency, privacy, and energy challenges, while continual adaptation on constrained devices remains limited by memory and compute constraints.

  • Method

    The survey integrates literature on TinyDL architectures, hardware platforms, compression, software toolchains, applications, on-device learning, and deployment strategies.

  • Results

    The survey identifies a transition toward autonomous, efficient edge intelligence supported by neuromorphic systems, federated learning, lightweight foundation models, and hardware–software co-design.

  • Takeaways & Limitations

    TinyDL provides a framework for sophisticated on-device intelligence in applications requiring low latency, energy efficiency, and data privacy.

  • Takeaways & Limitations

    Continual learning on constrained devices remains limited by memory limitations and requires algorithmic innovations that differ from traditional approaches.

Abstract

from arXiv · show

The rapid growth of edge devices has driven the demand for deploying artificial intelligence (AI) at the edge, giving rise to Tiny Machine Learning (TinyML) and its evolving counterpart, Tiny Deep Learning (TinyDL). While TinyML initially focused on enabling simple inference tasks on microcontrollers, the emergence of TinyDL marks a paradigm shift toward deploying deep learning models on severely resource-constrained hardware. This survey presents a comprehensive overview of the transition from TinyML to TinyDL, encompassing architectural innovations, hardware platforms, model optimization techniques, and software toolchains. We analyze state-of-the-art methods in quantization, pruning, and neural architecture search (NAS), and examine hardware trends from MCUs to dedicated neural accelerators. Furthermore, we categorize software deployment frameworks, compilers, and AutoML tools enabling practical on-device learning. Applications across domains such as computer vision, audio recognition, healthcare, and industrial monitoring are reviewed to illustrate the real-world impact of TinyDL. Finally, we identify emerging directions including neuromorphic computing, federated TinyDL, edge-native foundation models, and domain-specific co-design approaches. This survey aims to serve as a foundational resource for researchers and practitioners, offering a holistic view of the ecosystem and laying the groundwork for future advancements in edge AI.

1 Introduction

TinyML brings real-time machine learning inference to severely constrained edge devices, while TinyDL extends this paradigm to compressed deep models. This survey distinguishes the emerging TinyDL landscape and synthesizes its architectures, hardware, optimization methods, toolchains, and deployment challenges.

  • Motivation: TinyML enables on-device analytics on MCUs and embedded platforms that previously could not support conventional machine learning models.Hardware design, model compression, and embedded inference make real-time edge intelligence feasible.
  • Foundational constraints: Devices targeted by TinyML commonly operate below 1 mW with 32–512 kB of SRAM and constrained flash storage.They often lack operating systems and floating-point hardware accelerators.
  • Motivation: On-device inference addresses cloud round-trip latency, privacy concerns from sensor transmission, and the energy cost of streaming data remotely.These pressures are especially relevant to autonomous systems, wearables, healthcare, and industrial IoT.
  • From TinyML to TinyDL: TinyDL emerged as deeper architectures became compressible enough to operate under TinyML-like memory and compute constraints.Architectural advances include depthwise separable convolutions, inverted residuals, and attention mechanisms.
  • Survey scope: The survey addresses the limited distinction between classical TinyML and TinyDL by integrating developments through 2025 across models, toolchains, hardware, and deployment strategies.It also identifies gaps involving federated learning, secure over-the-air updates, and robust TinyDL benchmarks.
  • Contributions: The paper contributes a unified TinyDL taxonomy, a synthesis of over 200 sources, a benchmarking framework, an open repository, and a research roadmap.The proposed benchmark includes latency, memory usage, model size, and energy efficiency.

2 Background and Foundational Concepts

TinyML deploys machine learning on devices constrained by memory, power, and compute, whereas TinyDL targets compressed deep networks for more sophisticated real-time tasks. The section frames this transition through hardware, workflows, optimization, and expanding application domains.

  • TinyML: TinyML keeps data processing and inference on resource-constrained embedded devices, supporting low latency, low power consumption, and local data privacy.Its hardware ecosystem centers on low-power MCUs integrated with application-specific sensors.
  • TinyML constraints: TinyML devices face three critical constraints: scarce memory, ultra-low power budgets, and limited compute capability.Typical systems use kilobytes to hundreds of kilobytes of SRAM, often under 1 MB of flash, and MCUs that may lack FPUs.
  • TinyDL: TinyDL deploys CNNs and RNNs on extremely resource-constrained hardware to support tasks such as image classification, object detection, and real-time gesture recognition.Its models are commonly compressed to a few hundred kilobytes.
  • Optimization: TinyDL uses quantization and pruning to reduce model size and computation while preserving practical accuracy for on-device inference.Quantization converts floating-point parameters to lower-precision integers, while pruning removes redundant parameters or connections.
  • TinyDL distinction: Unlike conventional deep learning, TinyDL prioritizes on-device efficiency, low power consumption, and minimal memory footprints measured in kilobytes.Architectures such as MobileNet, SqueezeNet, and Tiny-YOLO reduce operations and parameter counts for MCU execution.
  • Workflows: TinyML workflows include ML-oriented, hardware-oriented, and co-design approaches that distribute design emphasis across models, platforms, and their integration.Hardware-oriented design targets computational bottlenecks to improve throughput and reduce power consumption.
  • Evolution: The TinyML-to-TinyDL progression expands representative use cases from simpler monitoring tasks toward speech recognition, vision inference, and autonomous systems.The comparison spans model size, hardware platforms, optimization techniques, and applications.

3 Hardware Platforms for TinyML and TinyDL

TinyML and TinyDL hardware spans constrained MCUs and specialized neural accelerators, with deployment shaped by tight memory, compute, and power budgets. Standardized benchmarks compare these platforms across latency, accuracy, throughput, and energy efficiency.

  • MCUs: MCUs provide low-cost, battery-oriented platforms for edge inference, typically combining tens of megahertz clock speeds with kilobytes to hundreds of kilobytes of RAM.Their constrained resources require highly efficient model design.
  • MCUs: Optimized models allow MCUs such as ESP32 and STM32 devices to run keyword spotting, gesture recognition, anomaly detection, and simple image classification on-device.Quantization and related optimizations enable useful inference under tight memory and energy constraints.
  • Specialized AI Hardware: The Edge TPU delivers 4 TOPS at about 2 W and can run MobileNet V2 at nearly 400 frames per second.It offloads matrix multiplications and convolutions from a host MCU or microprocessor.
  • Specialized AI Hardware: Syntiant NDP120 performs keyword-spotting inference in about 4.3 ms at 35 µJ per inference, while Himax WE-I Plus can run person detection under 5 mW.These accelerators target always-on audio and event-driven vision workloads, respectively.
  • Specialized AI Hardware: Specialized AI chips accelerate neural operations through dedicated logic, parallel multiply-accumulate units, on-chip memory, and streamlined dataflows.These designs target higher throughput per watt than software-only MCU execution.
  • Benchmarking and Evaluation: MLPerf Tiny and EEMBC MLMark provide complementary evaluation frameworks for comparing embedded systems using latency, accuracy, throughput, and energy-related measurements.MLPerf Tiny includes energy-aware metrics, whereas MLMark emphasizes standardized latency and accuracy evaluation.

4 Evolution from TinyML to TinyDL

TinyDL evolved from classical TinyML by replacing shallow, manually engineered pipelines with compressed deep models, hardware-aware optimization, and increasingly capable edge hardware. This transition improves representational capacity while preserving strict memory, energy, and compute constraints.

  • Limitations of Classical TinyML: TinyML models often have higher generalization error because limited capacity restricts complex feature learning and robustness to unseen or out-of-distribution data.In VWW, MobileNet and MCUNet reach 85–90% accuracy within 200–250 KB, whereas HOG-based classical pipelines typically achieve 70–75%.
  • Limitations of Classical TinyML: Manual feature engineering is time-consuming, domain-dependent, and often inadequate for capturing complex patterns in scalable real-time edge applications.These limitations are amplified by the resource constraints of TinyML devices.
  • Optimization Techniques: Model compression reduces neural-network memory and computation through pruning, low-rank factorization, and knowledge distillation, with reported memory reductions of 20% to 30%.Compression can involve a trade-off in quality.
  • Optimization Techniques: Quantization represents weights and activations in reduced precision, reducing memory footprint and multiply–accumulate operations for deployment on constrained MCUs.The hardware ecosystem includes MCUs, DSPs, NPUs, SRAM, Flash, and DRAM; Cortex-M4 remains widely adopted, while NPUs accelerate matrix operations efficiently.
  • Optimization Techniques: Post-Training Quantization converts float models to INT8 with an approximately 4× size reduction, while Quantization-Aware Training supports more aggressive 4- or 2-bit inference.Mixed-precision schemes select per-layer bit widths according to latency and energy targets.
  • Optimization Techniques: Neural architecture search defines candidate spaces, searches for models, and evaluates accuracy–efficiency trade-offs, with newer methods reducing search costs to a few GPU days.MNasNet, FBNet, and MONAS incorporate latency, power, and computational constraints into multi-objective search.

5 TinyDL Architectures and Techniques

TinyDL architectures redesign neural networks around severe MCU constraints, using lightweight convolutions, parameter reduction, hardware-aware NAS, and compressed transformers. These approaches target useful accuracy while reducing model size, memory, and computation.

  • Lightweight CNNs: Lightweight CNNs reduce computational complexity through hardware-oriented architectural principles while maintaining competitive accuracy on constrained edge devices.The section highlights MobileNet, SqueezeNet, and MCUNet as representative designs.
  • MobileNet: MobileNet factorizes convolution into depthwise and pointwise operations, while MobileNetV2 adds inverted residuals and linear bottlenecks.MobileNetV2 achieves 71.8% ImageNet top-1 accuracy with a 3.4 MB model on STM32H7 MCUs.
  • SqueezeNet: SqueezeNet uses squeeze and expand fire modules, while SqueezeNext reaches AlexNet-level accuracy with 112× fewer parameters and 1.2 MB storage.SqueezeNext maintains 57.5% ImageNet accuracy under flash constraints.
  • MCUNet: MCUNet jointly optimizes neural architecture and inference scheduling through hardware-aware NAS for MCU deployment.It reports 68.7% ImageNet accuracy with a 0.51 MB model.
  • Tiny Transformers: Transformer deployment on MCUs is constrained by quadratic attention memory complexity, intermediate attention storage, and frequent data marshaling.Fused-Weight Self-Attention and Depth-First Tiling address these bottlenecks, with reported memory-peak reductions up to 6.19×.
  • Tiny Transformers: TinyBERT uses two-stage transformer distillation, achieving 96.7% of BERT-Base performance on GLUE while being 7.5× smaller and 9.4× faster.DistilBERT offers a different distillation approach, reducing BERT by 40% while retaining 97% of its language-understanding capabilities.

5.3 Model Optimization Techniques

TinyDL optimization combines compression, hardware-aware co-design, and training-time supervision to fit deep models within constrained devices. The resulting trade-offs are evaluated across model size, inference speed, accuracy, and deployment hardware.

  • Overview: TinyDL deployment requires optimization techniques that compress models and accelerate inference while preserving accuracy.Table 3 summarizes model size, inference speed, task accuracy, and deployment contexts across representative models.
  • Quantization: Quantization converts floating-point models to reduced-precision representations, with PTQ typically providing a 4× model-size reduction and QAT enabling more aggressive precision reduction.QAT incorporates quantization effects during training to maintain model performance.
  • Joint Optimization: MCUNet jointly searches neural architectures and tunes inference scheduling instead of optimizing the model and runtime separately.TinyNAS and TinyEngine expand the design space across architecture and inference-engine choices.
  • Pruning: Pruning removes redundant parameters, with magnitude-based pruning targeting small weights and structured pruning removing filters or layers.Structured pruning enables larger architectural simplifications for severely resource-constrained environments.
  • Combined Compression: Combining quantization and pruning can produce greater computational efficiency than either technique alone, particularly for ultra-low-latency applications.The benefits depend on target compression ratios and application requirements.
  • Training-Time Optimization: Network augmentation trains a tiny model inside larger weight-sharing networks with auxiliary supervision, without increasing the tiny model’s inference-time footprint.Only the tiny network is used during inference.

6 Software Toolchains and Deployment Frameworks

TinyDL deployment toolchains span model conversion, optimized kernels, graph compilation, hardware-specific runtimes, and end-to-end AutoML platforms. The ecosystem increasingly integrates quantization, pruning, autotuning, and cross-platform deployment for MCU, DSP, NPU, and mobile targets.

  • Toolchain Ecosystem: Deployment toolchains bridge trained models and ultra-low-power hardware through lightweight frameworks, compilation techniques, and end-to-end platforms.The surveyed lifecycle includes model conversion, optimization, testing, and deployment.
  • Deployment Frameworks: Early converters such as uTensor generated C++ code, while TFLite Micro added 8-bit quantization and broader operator support for microcontroller deployment.TFLite Model Maker supports fine-tuning and export for EdgeTPUs and mobile hardware, whereas PyTorch Mobile targets higher-end mobile SoCs.
  • Compilers and Runtime Libraries: CMSIS-NN supplies hand-optimized ARM Cortex-M kernels, and MicroTVM adds graph optimization, autotuning, and memory planning for bare-metal targets.MicroTVM supports platforms including STM32 and ESP32, while Glow lowers graphs for hardware accelerators and NPUs.
  • Specialized Toolchains: Specialized tools target hardware-aware deployment through static C generation, vendor integration, FPGA workflows, and automated compilation, quantization, and precision tuning.Examples include DeepC, X-CUBE-AI, QKeras with HLS4ML, OctoML, and Nebullvm.
  • Optimization Platforms: Optimization-focused platforms such as LEIP and OctoML combine mixed-precision quantization, pruning, and tuning across CPUs, NPUs, and MCUs.OctoML builds on TVM, while Latent AI targets hybrid edge use cases requiring cross-platform support.
  • End-to-End Platforms: Common platforms occupy distinct points in the usability–optimization space, from TFLite Micro’s 8-bit baseline and Edge Impulse’s drag-and-drop AutoML to MicroTVM’s compiler autotuning.Neuton TinyML emphasizes sub-kilobyte models for highly constrained deployments.

7 Applications of TinyML and TinyDL

TinyML and TinyDL support real-time, energy-efficient intelligence across vision, audio, healthcare, industrial, and environmental applications. These systems bring analytics closer to data sources while addressing latency, privacy, and resource constraints.

  • Computer Vision: TinyML supports small-object detection and low-resolution vision tasks for applications including criminal investigation, autonomous driving, and embedded sensing.Research emphasizes input-resolution improvements, contextual information, and energy feasibility for constrained vision systems.
  • Audio and NLP: Lightweight edge speech and audio systems enable on-device speech recognition, keyword spotting, environmental sound monitoring, and wildlife sound recognition.Transformer-based models are adapted through quantization, pruning, and architectural simplification.
  • Healthcare: TinyML healthcare applications analyze ECGs, physiological signals, emotions, and human activity while prioritizing privacy, low latency, and energy efficiency.Examples include reservoir computing on low-power MCUs, wearable sensing, mmWave radar, and privacy-preserving wristbands.
  • Industrial Monitoring: Industrial TinyML enables on-device anomaly detection and predictive maintenance using machine sounds, vibrations, and continuous equipment monitoring.Blockchain integration is also studied for transparent and immutable anomaly records.
  • Environmental Applications: TinyML contributes to smart agriculture and wildlife monitoring by moving crop, soil, disease, and ecological analyses closer to data sources.These applications connect edge intelligence with industrial efficiency and environmental sustainability.

8 On-Device Learning and Reformability

On-device learning addresses the limitations of static TinyML models by enabling adaptation after deployment. Reformable TinyML extends this idea into a broader lifecycle framework for modifying systems as conditions change.

  • Continual Learning: Static TinyML models can degrade under concept drift because real-world data distributions change after deployment.Traditional workflows train models offline and deploy optimized, fixed inference versions on MCUs.
  • Continual Learning: Online and continual learning update MCU models incrementally as new data arrives, avoiding complete model redeployment.TinyOL is presented as a framework for learning from continuous data streams under MCU constraints.
  • Reformable TinyML: Reformable TinyML designs systems that can be modified, updated, or reformed after initial deployment.Its intended workflow includes detecting performance degradation and triggering adaptation to maintain accuracy and efficiency over time.
  • Privacy and Security: On-device processing keeps sensitive audio and image data local, reducing exposure associated with transmission and third-party servers.This privacy benefit is tied to performing inference directly on the MCU rather than sending data to the cloud.

9 Evaluation Metrics and Benchmarks

TinyML evaluation must jointly measure task performance, resource use, and deployment feasibility. Benchmarks therefore combine accuracy with latency, memory, model size, operations, and energy metrics across constrained platforms and tasks.

  • Evaluation Framework: TinyML evaluation combines accuracy with efficiency, scalability, deployment feasibility, and trade-offs between performance and resource utilization.This broader framework differs from conventional evaluation centered mainly on accuracy or F1-score.
  • Efficiency Metrics: Model size measures the stored footprint of weights, biases, and metadata, with common MCU deployments targeting 250–500 KB.Pruning, Huffman encoding, and post-training quantization reduce storage requirements.
  • Efficiency Metrics: Inference latency measures one forward pass in milliseconds and affects responsiveness in real-time systems such as gesture recognition and wake-word detection.TensorFlow Lite Micro’s Google KWS model executes in less than 20 ms on a Cortex-M4 at 80 MHz.
  • Efficiency Metrics: Memory usage includes static parameters and dynamic activations, constrained by MCU SRAM such as the STM32L475’s 128 KB.TensorFlow Lite Micro and TVM provide memory-profiling support.
  • Efficiency Metrics: Operations count records MACs or FLOPs and supports comparisons across accuracy, latency, and energy.MLPerf Tiny uses MACs as a primary standardized metric.
  • Efficiency Metrics: Energy per inference ranges from 0.1 mJ to 10 mJ in benchmark models, while specialized accelerators can support sub-mJ inference.Energy efficiency is especially important for battery-powered and energy-harvesting devices.
  • Accuracy and Compression: Quantization and pruning reduce model resource demands but can create accuracy trade-offs.Removing 50% of MobileNetV2 weights incurred < 1% top-1 drop on ImageNet while shrinking model size by 1.9 times.
  • Constraint-Aware Design: Constraint-aware NAS targets deployment limits such as 64 KB of RAM, 10 ms latency, and 1 mJ per inference.μNAS and Once-for-All generate models tailored to hardware and deployment constraints.

10 Challenges and Open Research Problems

TinyDL faces persistent challenges in balancing accuracy with hardware limits, securing deployed systems, supporting adaptation, and evaluating heterogeneous platforms fairly. Toolchain limitations and insufficient benchmarks remain major barriers to broader deployment.

  • Resource Constraints: TinyDL deployment requires balancing model accuracy against stringent memory and computational constraints, especially in accuracy-critical applications.MCU memory hierarchies also constrain intermediate activations and require network-level memory scheduling.
  • Resource Constraints: Transformer deployment remains difficult because quadratic self-attention conflicts with MCU resource scaling, while current approximations remain insufficient for complex applications.This limits deployable model complexity on edge devices.
  • Security: TinyML devices are vulnerable to transferable adversarial attacks, model extraction, and evasion because their defensive capabilities are limited.Attacks demonstrated on powerful hosts can transfer to devices such as ESP32 and Raspberry Pi.
  • Security: Physical access and limited compute make secure model-update transmission and verification difficult, while hardware security modules and trusted execution environments increase cost.These constraints complicate deployment in remote or hostile environments.
  • Adaptation: Few-shot adaptation is constrained because meta-learning algorithms can exceed TinyML computational and memory budgets.This is particularly relevant to personalized wearable health monitoring under sub-256 KB memory limits.
  • Adaptation: Continual learning must acquire new knowledge while preventing catastrophic forgetting under severe memory limitations.Effective approaches require algorithmic innovations that differ fundamentally from traditional methods.
  • Evaluation: TinyDL lacks comprehensive standardized benchmarks, while heterogeneous hardware complicates fair comparisons across memory hierarchies, instruction sets, and optimization opportunities.Existing MLPerf Tiny coverage emphasizes computer vision and simple audio processing rather than domains such as NLP and sensor fusion.
  • Toolchains: Transformer deployment on MCUs is limited by inadequate compiler and runtime support for attention and layer normalization.Toolchains such as TensorFlow Lite Micro and CMSIS-NN lack efficient attention-layer primitives for some lightweight transformer models.

11 Future Directions

Future TinyDL directions include neuromorphic and federated learning, tiny foundation models, Edge AutoML, and heterogeneous accelerator–software co-design. These approaches target ultra-low-power inference, privacy-preserving adaptation, cross-task reuse, and efficient deployment under hardware constraints.

  • Neuromorphic Computing: Neuromorphic spiking architectures target ultra-low-power, event-driven, always-on inference on MCU-scale devices.BrainChip’s Akida and Intel’s Loihi are identified as leading hardware platforms, while surrogate-gradient training and event encoding remain research needs.
  • Federated TinyDL: Federated TinyDL enables decentralized learning without sharing raw data, but communication overhead, device heterogeneity, and adversarial resilience remain challenges.TinyFedTL and TinyMetaFed support on-device aggregation of quantized updates.
  • Tiny Foundation Models: Tiny foundation models use distillation, structured pruning, and quantization to approach MCU-suitable sizes while preserving task performance.Modular backbones with lightweight task-specific heads are proposed for multi-task edge deployment and fine-tuning.
  • Edge AutoML: Hardware-aware Edge AutoML methods such as TinyNAS and Once-for-All Networks balance accuracy with memory and latency constraints.Integrating AutoML with quantization, pruning, and hardware feedback remains an open deployment challenge.
  • Hardware–Software Co-design: Domain-specific accelerators improve TinyDL inference speed, energy efficiency, and model scalability, while compilation must coordinate heterogeneous hardware.The survey highlights NPUs, ASICs, FPGAs, specialized RISC-V engines, TFLite Micro, and CMSIS-NN as parts of this deployment ecosystem.

12 Conclusions

The survey synthesizes how efficient architectures, optimization methods, software toolchains, and specialized hardware enable sophisticated TinyDL on severely constrained devices. It connects these developments to applications requiring low latency, energy efficiency, and privacy, and identifies edge-native intelligence as a future direction.

  • Survey Scope: The survey examines the evolution from TinyML to TinyDL through efficient architectures, hardware platforms, software toolchains, and model optimization.Covered optimization methods include quantization, pruning, joint compression, and neural architecture search for edge constraints.
  • Applications: TinyDL applications span vision, audio, healthcare, and industrial monitoring, addressing requirements for low latency, energy efficiency, and data privacy.These applications illustrate the survey’s cross-domain examination of practical edge intelligence.
  • Future Outlook: Future TinyDL research emphasizes neuromorphic architectures, federated personalization, lightweight foundation models, hardware–software co-design, and standardized energy-aware benchmarks.The conclusion frames these directions as part of a shift toward autonomous edge intelligence.
Loading 2506.18927v2…