Source-linked AI summary
Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
Xubin Wang, Zhiqing Tang, Jianxiong Guo, Tianhui Meng, Chenhao Wang, Tian Wang, Weijia Jia
TL;DR
On-device AI deployment is constrained by computational, storage, memory, scalability, energy, privacy, and security challenges. This survey synthesizes application scenarios, technical challenges, optimization strategies, and emerging trends, highlighting hardware-aware NAS and compression methods for efficient edge deployment.
Problem
Deploying AI models on edge devices remains difficult because substantial computational, storage, memory, scalability, energy, privacy, and security demands exceed constrained-device capabilities.
Method
The survey organizes on-device AI research around fundamental concepts, applications, edge-environment challenges, optimization strategies, implementation methods, and emerging technologies.
Results
Hardware-aware NAS and model compression support efficient edge deployment by balancing accuracy with latency, memory, energy, and computational requirements.
Takeaways & Limitations
On-device AI models offer broad application potential across smartphones, IoT, edge computing, autonomous driving, and medical devices, while requiring continued optimization and responsible development.
Takeaways & Limitations
On-device AI models may reproduce racial, gender, or socioeconomic biases from training data, requiring bias detection, fairness-aware methods, evaluation standards, and regulatory mechanisms.
Abstract
from arXiv · showhide
The rapid advancement of artificial intelligence (AI) technologies has led to an increasing deployment of AI models on edge and terminal devices, driven by the proliferation of the Internet of Things (IoT) and the need for real-time data processing. This survey comprehensively explores the current state, technical challenges, and future trends of on-device AI models. We define on-device AI models as those designed to perform local data processing and inference, emphasizing their characteristics such as real-time performance, resource constraints, and enhanced data privacy. The survey is structured around key themes, including the fundamental concepts of AI models, application scenarios across various domains, and the technical challenges faced in edge environments. We also discuss optimization and implementation strategies, such as data preprocessing, model compression, and hardware acceleration, which are essential for effective deployment. Furthermore, we examine the impact of emerging technologies, including edge computing and foundation models, on the evolution of on-device AI models. By providing a structured overview of the challenges, solutions, and future directions, this survey aims to facilitate further research and application of on-device AI, ultimately contributing to the advancement of intelligent systems in everyday life.
1 Introduction
This survey examines on-device AI as local processing and inference on edge or terminal devices, motivated by deployment constraints, real-time demands, and privacy concerns. It organizes existing knowledge around applications, technical challenges, optimization methods, and future directions.
- Motivation and applications: Local deployment supports faster processing, reduced network-bandwidth dependence, and stronger privacy protection, while serving applications from smartphones to medical devices.
- Technical challenges: Resource-constrained deployment creates trade-offs among model accuracy, scalability, computational efficiency, storage, memory, and energy consumption.
- Definition: On-device AI models process data and perform inference locally on edge or terminal devices without transmitting data to the cloud.
- Survey scope: The survey addresses applications, deployment challenges, optimization and implementation methods, and future trends through four research questions.
- Practical implications: The review is intended to provide researchers, engineers, and industry professionals with perspectives and practical insights for implementing AI solutions in edge environments.
- Contributions: Its contributions include synthesizing the field, identifying technical challenges and literature gaps, discussing optimization strategies, and outlining future research directions.
2 Understanding On-Device AI Models
On-device AI spans diverse edge hardware and model types, processing data locally through coordinated data, model, and system stages. Compared with cloud-based AI, it prioritizes lower latency but requires optimization for constrained devices.
- Edge devices range from high-performance edge servers to constrained IoT sensors, including smartphones, drones, vehicles, robots, and smart-home technologies.
- AI models deployed at the edge range from rule-based systems to sophisticated machine-learning algorithms, with foundation models expanding the model landscape.
- How On-Device AI Works: On-device AI deployment follows a general pipeline covering data processing, model development, and system integration, although not every step is required in practice.
- How On-Device AI Works: Data processing includes collection, filtering, feature extraction, aggregation, and quantization to produce data suited for efficient edge processing.
- How On-Device AI Works: Model development includes training, architecture and hyperparameter selection, and compression to reduce computational requirements while maintaining accuracy.
- How On-Device AI Works: System integration combines software and hardware optimization before deploying the model to edge devices for real-time processing.
3 Applications of On-Device AI Models
On-device AI models support applications across mobile, IoT, edge-computing, autonomous-driving, and healthcare domains. These applications use local or near-source processing for interaction, monitoring, recognition, control, navigation, and medical support.
- Smartphones and Mobile Devices: Smartphones use on-device AI for voice assistants, image recognition, and personalized recommendations.
- IoT Devices: IoT devices apply AI to smart-home automation and real-time environmental monitoring.
- Edge Computing: Edge-computing applications include real-time video analysis, intelligent traffic management, and smart manufacturing.
- Autonomous Driving: Autonomous-driving systems use AI for environmental perception, path planning, and decision-making from camera, radar, and LiDAR data.
- Healthcare: Medical devices and health-monitoring systems support disease diagnosis, personalized treatment, and remote monitoring.
4 Technical Challenges of AI Models on Devices
Edge deployment is constrained by limited processing power, storage, memory, and multi-device resource management. The survey emphasizes lightweight architectures and compression methods to reduce computational and storage burdens while preserving performance.
- Processing Power: Limited CPU and GPU capabilities can prevent complex models from meeting real-time processing demands on edge devices.
- Processing Power: Complex models increase execution latency, motivating lightweight architectures such as MobileNets and neural architecture search.
- Optimization: Pruning, quantization, and knowledge distillation reduce computational burdens without significantly compromising accuracy.
- Storage and Memory: State-of-the-art models may require hundreds of megabytes to over a gigabyte of storage, exceeding many edge devices' capacities.
- Storage and Memory: Memory limitations can prevent loading all inference data, while distillation and incremental learning are presented as memory-management approaches.
- Storage and Memory: Multi-user deployments must manage storage allocation and contention, with distributed storage across edge devices proposed to improve resilience and scalability.
4.3 Energy Consumption Management
On-device AI must manage energy use, communication bandwidth, and computational placement under edge constraints. The survey highlights energy-efficient algorithms, dynamic management, specialized hardware, preprocessing, caching, and local computation as complementary strategies.
- Energy Consumption: Battery-powered edge devices can deplete rapidly when AI models have high energy demands, making energy reduction important for battery life.
- Energy Consumption: Dynamic energy-management algorithms can adjust processing power according to real-time performance metrics, workloads, and environmental conditions.
- Hardware Acceleration: TPUs and FPGAs can accelerate AI computations while reducing power requirements compared with general-purpose processors.
- Communication Bandwidth: Restricted edge connectivity makes transmitting the large data volumes required by many AI models difficult.
- Communication Bandwidth: Data preprocessing reduces transmission by filtering and compressing data so that only relevant information is communicated.
- Edge Caching: Edge caching stores frequently accessed data and models locally, reducing cloud communication and improving response times.
- On-Device Computation: On-device computation enables real-time responses by analyzing local inputs and transmitting only relevant data, while less critical tasks may be offloaded to the cloud.
4.5 Data Privacy and Security
On-device AI systems face privacy and security challenges because they process sensitive personal data locally. The survey discusses anonymization, trusted execution environments, encryption, secure computation, regulatory compliance, and federated learning as protection approaches.
- Edge AI devices often process health and location data, making security during processing and storage essential for preventing breaches.
- Privacy-preserving techniques include data anonymization, trusted execution environments, homomorphic encryption, and secure multi-party computation.
- GDPR compliance requires data governance practices such as regular audits, consent management, and transparent data handling.
- Federated learning keeps training data on edge devices while sharing model updates across a distributed network.
4.6 Model Transferability and Adaptability
On-device AI models must transfer across heterogeneous devices and adapt to changing operating conditions, user behavior, and mobility. The survey links these requirements to model management, scheduling, online learning, and migration across edge nodes.
- Models must migrate from high-performance servers to resource-constrained mobile devices while preserving performance and accuracy.
- Model management and scheduling encompass placement, migration, and elastic scaling across edge environments.
- Optimal model placement must consider dependencies, deadlines, and bandwidth limitations.
- Layer dependency-aware scheduling can reduce cold-start times when meeting latency requirements.
- Models must adjust processing and inference strategies across changes in lighting, temperature, and network connectivity.
- Online learning and adaptive capabilities support changing user needs, while migration to suitable edge nodes helps maintain Quality of Service under mobility.
5.1 Data Optimization Techniques
The survey presents data preprocessing and edge frameworks as complementary ways to improve the quality and efficiency of data used by on-device AI models. Techniques include filtering, feature extraction, aggregation, quantization, and near-source processing, each involving practical trade-offs.
- 5.1 Data Optimization Techniques: Data optimization for on-device AI includes filtering, feature extraction, aggregation, quantization, and edge computing frameworks.
- Data Filtering: Data filtering removes irrelevant or noisy inputs, while active label cleaning targets visibly mislabeled data.
- Data Filtering: Filtering methods such as ensemble approaches can be computationally intensive in resource-constrained IoT environments.
- Feature Extraction: Feature extraction reduces dimensionality and model complexity by retaining relevant information, but overlooking significant features can lose critical information.
- Data Aggregation: Data aggregation combines information from multiple sources to reduce redundancy and improve coherence, although it may introduce latency.
- Data Quantization: Quantization lowers representation precision to reduce memory usage and increase processing speed, but quantization levels must preserve accuracy.
- Edge Computing Frameworks: AWS Greengrass and Azure IoT Edge process data near its source, reducing extensive transfer and latency in real-time applications.
5.2 Model Optimization Techniques
Model optimization techniques reduce the computational, memory, and energy demands of on-device AI while aiming to preserve accuracy. The survey covers compression, lightweight model design, pruning, low-rank factorization, and hardware-aware architecture search.
- Compression methods: Model compression reduces model size and resource requirements through parameter sharing, pruning, quantization, knowledge distillation, and low-rank factorization.These techniques target deployment on devices with stringent computational, memory, and power constraints.
- Parameter sharing: Parameter sharing reuses weights across layers to reduce computational and memory demands without substantial accuracy loss.It has been applied to CNNs and RNNs, including clustering-based weight sharing and Tensor Ring compression.
- Pruning: 90% reduction in network complexity can be achieved through pruning without compromising accuracy.Pruning removes redundant parameters, nodes, filters, or layers, reducing inference cost and memory consumption.
- Low-rank factorization: Low-rank factorization approximates weight matrices with lower-dimensional matrices to reduce memory and computation while retaining significant information.MicroNet-M1 achieves 61.1% top-1 accuracy on ImageNet using 12 MFLOPs.
- Hardware-aware Neural Architecture Search: Neural Architecture Search balances accuracy with latency, memory usage, energy consumption, and training cost for edge deployment.Hardware-aware methods adapt architectures to device constraints; reported examples include ProxylessNAS at 74.6% accuracy, 320 million MACs, and 200 GPU hours.
- Energy-efficient Model Design: Lightweight architectures improve efficiency through structural design choices such as channel shuffling, compound scaling, and hybrid CNN-transformer representations.Examples include ShuffleNetV2 at 72.6% top-1 accuracy with 299 million MACs and MobileViT above 78% top-1 accuracy with approximately 320 million MACs.
5.3 System Optimization Techniques
System optimization combines software frameworks for lightweight training and inference with hardware accelerators and low-power chips. The survey describes platform-specific frameworks and accelerator designs for improving computational efficiency on constrained edge devices.
- Overview: System optimization combines software and hardware approaches to improve computational efficiency for on-device AI deployment.Software methods address lightweight model training and inference, while hardware methods use accelerators and low-power chips.
- On-Device AI Learning Frameworks: On-device AI learning frameworks support model training, optimization, and deployment on edge devices.TensorFlow Lite and PyTorch Mobile are optimized versions of widely used learning frameworks for mobile applications.
- On-Device AI Inference Frameworks: On-device AI inference frameworks focus on efficient execution of pretrained models across different hardware platforms.Examples include NCNN, OpenVINO, and ONNX Runtime; NCNN emphasizes low memory use and ARM compatibility.
- Hardware-based Optimization: Hardware optimization uses CPUs, GPUs, FPGAs, ASICs, and NPUs to accelerate AI processing under limited power budgets.These approaches include specialized processors, memory-efficient architectures, and domain-specific hardware.
- Hardware-based Optimization: FPGA-based accelerators exploit pipelining, parallelism, and specialized compilation to improve CNN throughput and energy efficiency.A lightweight FPGA overlay processor for CNNs is reported to achieve 5.5x better performance.
- Hardware-based Optimization: ASIC designs provide hardware-level optimization for on-device AI, including up to 7x energy savings through early-exit inference.The edgeBERT architecture applies entropy-based early exits to multi-task NLP inference.
6 Future Development Trends
Future on-device AI development emphasizes stronger connectivity, edge-cloud cooperation, adaptable and intelligent models, sustainability, and responsible deployment. These trends aim to improve responsiveness and resource efficiency while addressing privacy, environmental, and social concerns.
- Connectivity and Edge Computing: 5G and edge computing are expected to improve on-device responsiveness by enabling low-latency connectivity and processing closer to data sources.These technologies support real-time video processing, instant feedback, autonomous driving, and smart surveillance while reducing reliance on centralized clouds.
- Foundation Models: Foundation models can be pre-trained broadly and fine-tuned for local application needs in edge environments.Their adaptability is presented as a basis for faster development and deployment across multiple domains.
- Adaptability and Intelligence: Future models are expected to adapt dynamically to real-time inputs, user feedback, multiple data sources, and contextual information.The survey associates these capabilities with personalized services, more accurate predictions, and autonomous performance improvement.
- Sustainability and Green Computing: Energy-efficient algorithms, low-power hardware, pruning, quantization, and knowledge distillation are identified as routes toward greener on-device AI.Collaborative cloud-edge architectures can also share processing tasks and reduce environmental impacts from over-provisioned resources.
- Ethics and Social Impact: Responsible deployment requires privacy protection, data security, transparency, explainability, encryption, and attention to bias and employment effects.The survey highlights regulatory compliance, fairness-aware methods, diverse stakeholder involvement, and workforce preparation as related priorities.
7 Conclusion
The survey synthesizes on-device AI applications, deployment challenges, optimization methods, and future research priorities. It presents broad applicability alongside continuing resource, security, adaptability, and sustainability requirements.
- Main Findings of the Survey: On-device AI has application potential across smartphones, IoT devices, edge computing, autonomous driving, and medical devices.The survey characterizes these domains as diverse application scenarios for device-deployed models.
- Main Findings of the Survey: Deployment challenges include limited computation, storage, memory, and energy, alongside privacy, security, transferability, and adaptability concerns.These constraints define the practical conditions under which on-device models must operate.
- Main Findings of the Survey: Model compression, pruning, hardware acceleration, quantization, low-precision computing, transfer learning, and federated learning improve operation in constrained environments.The survey identifies these techniques as major optimization and implementation advances.
- Future Research Directions: Future research should develop more efficient compression and pruning methods and co-design models with hardware such as FPGAs and TPUs.These recommendations target lower computational and storage demands while preserving accuracy and performance.
- Future Prospects: On-device AI is expected to affect societal, economic, and technological systems through smart homes, health monitoring, personalized services, and industrial transformation.The conclusion also calls for attention to sustainability and ethical implications during continued development.