Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things

Jing Zhang; Dacheng Tao

Source-linked AI summary

Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things

Jing Zhang, Dacheng Tao

arXiv:2011.08612v1Nov 17, 2020cs.AIcs.CVcs.LGcs.MM

TL;DR

IoT must process massive heterogeneous sensor data and make timely decisions, but centralized architectures face bandwidth and latency constraints and changing environments challenge learned models. This survey reviews AIoT architectures and deep-learning advances through perceiving, learning, reasoning, and behaving, alongside applications, challenges, and opportunities. It concludes that deep learning has rapidly advanced perception and AIoT applications, while edge intelligence and learning for dynamic environments require further effort.

Problem
Massive heterogeneous IoT data are difficult to transmit, interpret, and use for timely decisions, while dynamic environments can cause trained models to lose performance.
Method
The paper conducts a comprehensive AIoT survey covering cloud, fog, and edge architectures and organizing enabling AI technologies by perceiving, learning, reasoning, and behaving.
Results
The survey finds rapid progress in deep-learning-based perception and many AIoT applications, while identifying lightweight models, computation scheduling, and edge intelligence as continuing challenges.
Takeaways & Limitations
AIoT research spans architectures, AI capabilities, applications, and challenges, with further work needed to improve intelligence for dynamic and complex environments.

Abstract

from arXiv · show

In the Internet of Things (IoT) era, billions of sensors and devices collect and process data from the environment, transmit them to cloud centers, and receive feedback via the internet for connectivity and perception. However, transmitting massive amounts of heterogeneous data, perceiving complex environments from these data, and then making smart decisions in a timely manner are difficult. Artificial intelligence (AI), especially deep learning, is now a proven success in various areas including computer vision, speech recognition, and natural language processing. AI introduced into the IoT heralds the era of artificial intelligence of things (AIoT). This paper presents a comprehensive survey on AIoT to show how AI can empower the IoT to make it faster, smarter, greener, and safer. Specifically, we briefly present the AIoT architecture in the context of cloud computing, fog computing, and edge computing. Then, we present progress in AI research for IoT from four perspectives: perceiving, learning, reasoning, and behaving. Next, we summarize some promising applications of AIoT that are likely to profoundly reshape our world. Finally, we highlight the challenges facing AIoT and some potential research opportunities.

I. INTRODUCTION

The introduction frames AIoT as the integration of AI with IoT to address large-scale, heterogeneous sensing, processing, and decision-making challenges. It motivates a survey covering architectures, AI capabilities, applications, and research challenges.

IoT Architecture: IoT connects sensors, actuators, and devices through networks to collect data, exchange information, and deliver smart services.Its three-layer architecture comprises perception, network, and application layers.
IoT Architecture: 50 billion devices are projected to connect to IoT by 2025, generating 79.4 zettabytes of data and creating bandwidth and latency challenges for centralized cloud processing.Fog computing addresses these constraints by placing storage, computation, and networking closer to devices.
AI for IoT: Heterogeneous sensor data and handcrafted-feature pipelines are increasingly unsuitable for perceiving and understanding dynamic, complex IoT environments.The introduction contrasts classical methods based on handcrafted features with deep learning approaches that learn representations end to end.
AI for IoT: Deep neural networks can automatically learn discriminative representations from labeled or unlabeled data and support low-latency processing across cloud, fog, and edge platforms.The paper links this capability to improved perception in IoT systems with many heterogeneous sensors.
AI for IoT: IoT systems must address unseen edge cases and changing operating conditions because models trained on initial data may lose performance as environments or machine modes vary.The introduction connects these issues to few-shot, zero-shot, meta-, unsupervised, semi-supervised, transfer, and domain-adaptation learning.
Survey Scope: The survey covers AIoT architecture, AI progress organized around perceiving, learning, reasoning, and behaving, promising applications, and challenges with research opportunities.It aims to show how AI technologies empower IoT applications.

B. Relationship to Related Surveys

Related surveys commonly isolate particular IoT perspectives, architectures, networking issues, applications, or security topics. This survey instead organizes deep learning for IoT data processing around how AIoT systems perceive, learn, reason, and behave, while also covering applications and security-related methods.

Survey Scope: Existing IoT surveys often focus on selected topics such as architectures, networking, applications, security, or privacy rather than AIoT as a whole.The paper positions its review as a focused treatment of AIoT.
Survey Scope: This survey applies a taxonomy based on how deep learning improves AIoT systems’ perceiving, learning, reasoning, and behaving abilities.It focuses on deep learning for IoT data processing and covers recent progress across application domains.
Security and Privacy: Security and privacy remain related IoT concerns because connected sensors may collect biometric data vulnerable to cyberattacks.The survey reviews AI-based approaches such as federated learning alongside access control and trust management.
Applications: The review also summarizes promising AIoT applications across domains including smart cities, homes, healthcare, agriculture, industry, vehicles, and video things.It discusses how AI enables these applications to become faster, smarter, greener, and safer.

C. Organization

The paper proceeds from AIoT computing architecture to enabling AI technologies, applications, challenges, and opportunities. Its architecture discussion covers cloud, fog, and edge computing as components of AIoT systems.

The paper first discusses AIoT computing architecture, then surveys enabling AI technologies, summarizes applications, and addresses challenges, opportunities, and conclusions.
Cloud computing provides scalable computation, storage, and networking for integrating and processing data from distributed sensors.
The architecture discussion includes cloud computing, fog computing, and edge computing, with hybrid deployment across these tiers.
Fog computing places storage, computation, and networking near devices to provide lower latency than remote cloud processing.

3) Edge Computing Layer:

The section distinguishes fog and edge computing by their locations and describes hybrid AIoT architectures that distribute models across cloud, fog, and edge resources. It also frames the survey around how deep learning improves AIoT systems’ perceiving, learning, reasoning, and behaving abilities.

Edge Computing Layer:: Fog computing operates at the network side, whereas edge computing operates at the thing side near sensors and actuators.The paper treats the concepts as distinct for clarity despite overlapping usage in some literature.
Edge Computing Layer:: Edge computing can reduce latency and network bandwidth compared with fog and cloud computing.
Edge Computing Layer:: AIoT systems can deploy different models across cloud platforms, fog nodes, and edge devices to form hybrid computing architectures.Offloading computation from edge devices to fog nodes and cloud is expected to combine low latency with deep-learning capacity for massive data.
Edge Computing Layer:: GPU parallelism accelerates neural-network operations such as convolution, reducing training and inference time.
Edge Computing Layer:: TensorFlow and PyTorch support mobile deployment through model compression, quantization, and hardware acceleration, while ONNX enables model porting across frameworks.
Edge Computing Layer:: The survey organizes AI technologies for AIoT by their contributions to perceiving, learning, reasoning, and behaving.It selects topics closely related to and useful for AIoT applications rather than surveying deep learning broadly.

2) Object Detection:

The section reviews deep-learning progress in generic scene and human-centric perception, including detection, tracking, segmentation, text spotting, and recognition tasks. It emphasizes accuracy–speed trade-offs, specialized visual challenges, and applications in traffic, safety, and autonomous systems.

2) Object Detection:: Generic object detection recognizes both object categories and locations and supports downstream tasks including face recognition, re-identification, pose estimation, and behavior analysis.
2) Object Detection:: Two-stage and one-stage methods form the main categories of contemporary deep-learning object detectors.
2) Object Detection:: Object detection applications include pedestrian, car, traffic-sign, and license-plate detection for traffic surveillance, public safety, autonomous driving, and video structuring.
2) Object Detection:: Object tracking has progressed through multi-resolution deep features, end-to-end representation learning, and Siamese-network approaches.
2) Object Detection:: Fully convolutional encoder–decoder networks are the dominant paradigm for semantic segmentation because they learn multi-resolution features while preserving spatial correspondence.
2) Object Detection:: Text spotting jointly performs text detection and recognition despite variable text shapes, lengths, fonts, styles, and backgrounds.Sequence-to-sequence modeling and connectionist temporal classification have also been applied to the task.
2) Object Detection:: Human-centric perceiving covers biometric recognition, person re-identification, pose, gesture, and action estimation, and crowd-density estimation.

6) Biometric Recognition:

The survey covers biometric and human-centric perception methods for recognizing people, faces, poses, gestures, and crowds in AIoT settings. It emphasizes deep learning progress alongside practical constraints involving sensors, latency, power, occlusion, and multimodal sensing.

Biometric Recognition: Face recognition systems typically combine detection, alignment, representation, and classification or verification, with deep learning improving shared feature representations.Jointly modeling auxiliary tasks such as alignment, pose estimation, and gender classification can provide additional supervisory signals.
Biometric Recognition: Biometric solutions span face, fingerprint, iris, palmprint, and person re-identification, with method choice depending on sensors, usage scenarios, latency, and power consumption.Applications include access control, aviation security, e-banking, and mobile payment.
Human-Centric Perceiving: Human pose estimation detects body joints using top-down or bottom-up methods, while current advances target feature representation, training strategies, sub-pixel processing, and crowded-scene occlusion.Top-down methods separate person detection from keypoint detection; bottom-up methods detect and associate keypoints directly.
Human-Centric Perceiving: AIoT human-centric perception also includes hand gesture recognition and crowd counting using visual, radar, Wi-Fi, or Bluetooth sensing.Practical crowd-counting research seeks lightweight models, joint crowd-flow estimation, and multimodal sensor integration.
Perception Overview: Figure 6 illustrates generic scene, human-centric, and 3D perception through segmentation, detection, text spotting, parsing, pose, facial analysis, and depth estimation.The figure processes a frame from the video “Walking Next to People” using these complementary techniques.

10) Depth Estimation/Localization/SLAM:

The survey reviews depth estimation, localization, visual odometry, visual-inertial odometry, and SLAM as core 3D-perception capabilities for AIoT. Deep learning and geometric objectives support these tasks, but occlusions, dynamic objects, and scale ambiguity remain challenges.

Depth Estimation: Depth estimation in AIoT supports monocular, stereo, and multi-view camera configurations, with recent work focusing on unsupervised monocular video and camera-pose learning.Photometric reprojection losses derived from multi-view geometry provide an optimization signal.
Depth Estimation: Depth and pose estimation must address occlusions, dynamic objects, per-frame scale ambiguity, and temporal inconsistency.These conditions complicate reliable estimation even when convolutional networks provide strong representation capacity.
Localization and SLAM: Visual odometry and visual-inertial odometry calculate sequential camera poses from camera and IMU data and serve as front ends in SLAM systems.SLAM back ends optimize pose graphs to obtain globally consistent and drift-free pose estimates.
Applications: Depth estimation, pose estimation, VO/VIO, and SLAM provide 3D perception for smart transportation, industry, agriculture, cities, and homes.Multiple viewpoints can support depth estimation and 3D reconstruction of objects or scenes.
Perception Preprocessing: AIoT visual perception often requires preprocessing such as illumination enhancement, contrast improvement, and distortion rectification before downstream tasks.Low-light and hazy images are ill-posed enhancement problems addressed by prior-based and learning-based methods.

12) Image Rectification and Stitching:

The survey situates image rectification and stitching within the preprocessing needed for AIoT visual perception, alongside auditory, language, and multimodal analysis. It describes geometric calibration and learning-based methods for correcting distorted imagery and broader deep-learning approaches for speech and translation.

Image Rectification and Stitching: Wide-FOV cameras support surveillance and autonomous driving but introduce distortion because their imagery violates the perspective-transformation assumption.Rectification is therefore performed before downstream perception tasks.
Image Rectification and Stitching: Rectification methods include camera-calibration approaches and direct distortion-parameter estimation followed by mapping to a rectified image.Optimization-based and learning-based methods exploit geometric cues such as lines to formulate constraints or loss functions.
Auditory Perception: Deep neural networks revolutionized speech recognition by enabling end-to-end modeling without handcrafted feature engineering, explicit HMM design, or domain knowledge.Real-time ASR can run on devices or through cloud-assisted computing for interaction, transcription, and voice-assistant applications.
Natural Language Processing: Neural machine translation uses encoder-decoder architectures and attention mechanisms to model source context during decoding.Attention supports joint alignment and translation, with reported benefits for long sentences.

16) Multimedia and Multi-modal Analysis:

The survey examines multimedia and multimodal AIoT analysis, where deep models align or combine text, audio, images, video, and other sensor modalities. It also links these capabilities to efficient, adaptive deployment through compression, architecture search, and learning from limited annotation.

Multimedia Analysis: Cross-media matching and retrieval seek semantic alignment across modalities, while disentangled representation learning separates modality-exclusive information from shared embeddings.Related generative tasks include image or video captioning and text-to-image generation.
Multimodal Perceiving: RGB data can be combined with depth, LiDAR, or thermal infrared inputs for scene parsing, low-light detection and tracking, and action recognition.Fusion occurs at input, feature, or output level, with feature-level fusion described as most prevalent.
Applications: Multimedia generation and cross-modal analysis support retrieval, personalized item descriptions, education, chatbots, smart security, and rehabilitation monitoring.These applications span speech-, image-, video-, and text-based interaction and understanding.
Efficient Models: Neural architecture search automatically explores architectures within a predefined search space using evolutionary, reinforcement-learning, or gradient-based methods.NAS replaces manual architecture design with an automated search procedure.
Efficient Models: Network compression improves DNN efficiency under AIoT computational limits through pruning, quantization, low-rank factorization, and knowledge distillation.Pruning commonly trains a large network, removes components using a criterion, and retrains the reduced model.
Learning: AIoT learning research addresses changing environments through unsupervised, semi-supervised, transfer, domain-adaptation, few-shot, and zero-shot learning.Unlabeled and multimodal sensor data can support continual model improvement and cross-modal pretext tasks.

2) Transfer Learning and Domain Adaptation:

Transfer learning addresses insufficient target-task data by transferring knowledge from related source tasks, while domain adaptation targets the same task across differing domains. These methods support AIoT deployment across new scenarios, working modes, and rare cases.

Transfer Learning: Transfer learning addresses target tasks with insufficient training data by transferring knowledge from a related source task.Unlike semi-supervised learning, source and target data distributions need not be identical.
Transfer Learning: Transfer dependencies across related AIoT tasks can reduce labeled-data demands while maintaining performance and improving generalization.A computational taxonomic map across twenty-six visual tasks provides empirical guidance on transfer directions and supervision reuse.
Domain Adaptation: Domain adaptation learns models for a target domain from one or more source domains for the same task, including unsupervised settings without target annotations.Its core strategy is learning domain-invariant representations by matching source and target distributions.
Domain Adaptation: Domain adaptation and transfer learning are practically useful when AIoT models move to new scenarios or machine operating modes.These settings include related tasks such as object detection, tracking, and semantic segmentation in video surveillance.
Few-shot Learning: Few-shot learning uses prior knowledge to learn from only a few annotated samples, supporting rare-case recognition when large-scale collection and annotation are difficult.Examples include car collisions, cyber attacks, and machine faults.
Zero-shot Learning: Zero-shot learning maps data to semantic representations so models can recognize unseen classes, including future fault types or newly emerging categories.Auxiliary attributes or text descriptions represent both seen and unseen classes.

5) Federated Learning:

AIoT combines learning, reasoning, and control to support intelligent decisions and actions across distributed devices and dynamic environments. Federated learning protects distributed data during collaborative model training, while knowledge and causal reasoning support downstream tasks and explainability.

Federated Learning: Federated learning enables multiple data owners to collaboratively train a model without leaking their datasets to one another.Its architecture typically includes a central server and distributed client devices that compute gradients locally.
Reasoning: Reasoning abilities are important for AIoT systems that must make smart and explainable decisions from knowledge such as web information, medical records, and financial transactions.The paper reviews knowledge graphs and reasoning as mechanisms for using this information.
Reasoning: Knowledge graphs represent entities and relations as triples, enabling knowledge reasoning that infers new knowledge from existing facts.Knowledge reasoning supports graph refinement, classification, entity resolution, relation extraction, question answering, and recommendation.
Reasoning: Causal reasoning includes causal inference for estimating effects and causal discovery for finding causal relations.It is relevant to recommendations, fault analysis, driving safety, and interpretation of DNN decisions.
Behaving: AIoT behaving systems control sensors and actuators to transform a measured or inferred current state into a target state determined by rules or decision models.Control algorithms are task-oriented and designed for the physical structure and sensors of each system.

2) Interaction:

AIoT interaction spans communication with people and physical action in the environment, while applications combine perception, learning, reasoning, and behavior. These capabilities support security, transportation, healthcare, and other connected domains.

Interaction: AIoT systems interact through input/output devices, mechanical arms, or gearing for communication, manipulation, and transportation.Examples include touchscreens and microphones, industrial robot arms, and mobile-robot wheels.
Interaction: Human-machine interaction can accept images, video, keywords, text, sketches, lyrics, and scores to generate descriptions, poems, images, or music.Speech synthesis provides another communication channel between people and AIoT systems.
Interaction: Human-robot interaction includes service robots gripping and filling cups and industrial robots assembling products, with research addressing planning, safety, and imitation learning.These examples connect physical assistance and industrial production within smart-home and IIoT settings.
AIoT Applications: AIoT applications combine perceiving, learning, reasoning, and behaving abilities across security, transportation, healthcare, education, industry, energy, agriculture, homes, and cities.The survey frames these capabilities as enabling faster, smarter, greener, and safer systems.
Smart Transportation: Self-driving systems integrate environmental perception, driving-policy decisions, and interaction with traffic participants, while domain adaptation and transfer learning address deployment challenges.Deep reinforcement learning can learn driving policies from visual input, but real-world training is unaffordable.
Transportation and Healthcare: AIoT supports driver monitoring, human-activity recognition, explainable healthcare assistance, and multimodal medical services using cameras, wearable sensors, and robots.Applications span dangerous-driving detection, rehabilitation, information queries, and auxiliary diagnosis.

D. Smart Education

AIoT applications support education, digital-twin-based smart factories, PCB manufacturing, smart grids, precision agriculture, and connected homes. Across these settings, AI processes sensor data for recognition, monitoring, diagnosis, control, and interaction.

Smart Education: AIoT can help learners recognize species, learn languages, select personalized resources, and support visually impaired users through interaction.Few-shot learning enables edge devices to recognize new bird and animal species from only a few labeled samples.
Smart Education: Offline speech-to-speech translation on mobile devices combines speech recognition, language translation, and text-to-speech synthesis for language learning.The system can translate multiple languages and provide grammatical information.
Smart Industry: A digital twin is a mirror digital representation of a physical system that supports smart-factory monitoring, fault diagnosis, and downtime prevention.Connected sensors and actuators send real-time production-line data to the digital twin in the cloud.
Smart Industry: PCB manufacturing illustrates AIoT use in production, visual defect inspection, and machine fault diagnosis.Deep learning enables automatic real-time visual defect inspection, while AI improves industrial robot functions such as grasping.
Smart Grids: Smart grids apply AIoT to fault diagnosis, load monitoring and scheduling, cyber-attack detection, and UAV-based damage assessment.CNN processing of UAV images predicts damage extent, while edge algorithms support real-time sound-source localization.
Smart Agriculture: Precision agriculture uses sensors, autonomous machines, and geographic information systems to observe, measure, and respond to crop variability.AIoT supports crop counting, yield estimation, continuous monitoring, path planning, navigation, and greenhouse plant-growth control.
Smart Cities/Homes/Buildings: Smart homes and cities reuse AIoT techniques from other sectors for speaker authentication, gesture-based control, and related transportation and security functions.These applications connect home interaction with methods developed for broader smart-sector systems.

V. CHALLENGES AND OPPORTUNITIES

AIoT faces challenges in handling heterogeneous data, deploying and scheduling models across constrained and distributed resources, learning from limited labels, and governing data ownership, privacy, and energy use.

Multi-modal Heterogeneous Data Processing, Transmission, and Storage: Heterogeneous AIoT data streams challenge processing, transmission, and storage, motivating efficient coding and edge extraction of compact representations.Task-oriented representations should be computed at the network edge to reduce bandwidth and latency.
Deep Learning on Edge Devices: Edge deployment requires lightweight, computationally efficient, and hardware-friendly deep neural networks because edge devices have limited computational and storage resources.Network pruning, compression, and quantization remain open directions.
Computational Scheduling in the AIoT Architecture: Computation scheduling across cloud, fog, and edge resources must jointly consider data, bandwidth, latency, accuracy, energy, security, privacy, and changing demand.Dynamic adaptive scheduling is needed to handle unbalanced data flow and user demands over time.
Big and Small Data for Deep Learning in AIoT: Most AIoT data are unlabeled, while labeling is expensive; unsupervised, transfer, semi-supervised, and few-shot learning are proposed for multimodal data, new classes, rare cases, and state drift.The cited opportunities address both abundant unlabeled data and small labeled datasets.
Data Monopoly: Data collection and exploitation can create data monopolies that form de facto barriers to market entry and threaten free-market competition.The paper frames proprietary data protected by established interests as the basis of this risk.
Data Security and Privacy: Ubiquitous AIoT sensing raises questions about biometric-data ownership, retention, use, security, and privacy, requiring appropriate protective measures and legislation.The paper cites GDPR as an example giving individuals control over personal data.
Energy Efficiency: Communication technologies are projected to use 21% of global electricity, with data centers contributing more than one-third of that usage, underscoring energy-efficiency needs.Cooling, workload analysis, and task scheduling are discussed as sustainability-related responses.

B. Opportunities

The survey identifies opportunities to move intelligence toward the edge, reduce sensing and computation costs, integrate data with knowledge, and preserve privacy through distributed learning. It also highlights simulation, reasoning, and interaction as routes toward more capable AIoT systems.

Built-in Neural Processing Capacity for Edge Devices: Built-in neural processing on edge devices can reduce processing latency and network bandwidth consumption by processing sensing data locally.Local biometric storage and verification can also reduce data-leakage risk, while federated learning supports distributed training.
Event-Based and Neuromorphic Processing: Event-based cameras and neuromorphic processors reduce redundant data and dense computation by operating on changing pixels and sparse asynchronous event streams.The paper connects these designs with low-power, low-latency gesture and action recognition.
Virtual Training Environments: Virtual 3D platforms can train embodied-AI models cost-effectively, especially for deep reinforcement learning, but domain shift between virtual and physical environments remains critical.Examples include Voyage Deepdrive, OpenAI Gym, and Habitat.
Data and Knowledge Integration for Perceiving, Learning, Reasoning, and Behaving: Combining prior knowledge with deep learning supports data-efficient learning through zero-shot attribute transfer, knowledge graphs, and graph neural networks for reasoning tasks.The paper identifies question answering and fault or disease diagnosis as example application areas.
Privacy-Preserving Learning: Federated learning offers a privacy-preserving approach for training on distributed local sensor data without requiring all data to be transmitted to and stored in the cloud.The survey presents federated learning as a representative solution to AIoT privacy concerns.
Survey Scope and Research Opportunities: The survey covers AIoT architectures, AI-enabled perceiving, learning, reasoning, and behaving, applications, and the field’s challenges and opportunities.Its conclusion emphasizes lightweight models, edge intelligence, dynamic-environment learning, knowledge and causal reasoning, control, and multimodal interaction.