Source-linked AI summary
A Review on Deep Learning in UAV Remote Sensing
Lucas Prado Osco, José Marcato Junior, Ana Paula Marques Ramos, Lúcio André de Castro Jorge, Sarah Narges Fatholahi, Jonathan de Andrade Silva, Edson Takashi Matsubara, Hemerson Pistori, Wesley Nunes Gonçalves, Jonathan Li
TL;DR
UAV remote-sensing research lacked a combined review of deep learning applications. This paper surveys UAV imagery methods and finds object detection, RGB sensors, and CNNs dominate, while CNN-LSTMs and GANs gain attention.
Problem
Deep learning applications in UAV remote sensing lacked a dedicated literature review, despite UAVs’ detailed, flexible, and low-cost data collection.
Method
The paper synthesizes state-of-the-art deep learning techniques and perspectives for UAV-based image processing across environmental, urban, and agricultural applications.
Results
91.2% of reviewed UAV applications used CNNs; object detection and RGB sensors predominated, while CNN-LSTMs and GANs gained attention.
Takeaways & Limitations
The review identifies CNNs as the prevailing approach and highlights CNN-LSTMs, GANs, and multi/hyperspectral data as notable directions for UAV remote sensing.
Takeaways & Limitations
Applying conventional deep learning architectures to UAV imagery may underuse its high resolution, while preserving resolution and processing speed remains challenging.
Abstract
from arXiv · showhide
Deep Neural Networks (DNNs) learn representation from data with an impressive capability, and brought important breakthroughs for processing images, time-series, natural language, audio, video, and many others. In the remote sensing field, surveys and literature revisions specifically involving DNNs algorithms' applications have been conducted in an attempt to summarize the amount of information produced in its subfields. Recently, Unmanned Aerial Vehicles (UAV) based applications have dominated aerial sensing research. However, a literature revision that combines both "deep learning" and "UAV remote sensing" thematics has not yet been conducted. The motivation for our work was to present a comprehensive review of the fundamentals of Deep Learning (DL) applied in UAV-based imagery. We focused mainly on describing classification and regression techniques used in recent applications with UAV-acquired data. For that, a total of 232 papers published in international scientific journal databases was examined. We gathered the published material and evaluated their characteristics regarding application, sensor, and technique used. We relate how DL presents promising results and has the potential for processing tasks associated with UAV-based image data. Lastly, we project future perspectives, commentating on prominent DL paths to be explored in the UAV remote sensing field. Our revision consists of a friendly-approach to introduce, commentate, and summarize the state-of-the-art in UAV-based image applications with DNNs algorithms in diverse subfields of remote sensing, grouping it in the environmental, urban, and agricultural contexts.
1 Introduction
UAV remote sensing offers flexible, detailed, low-cost data collection, but manual inspection remains time-consuming, biased, and inaccurate. This review addresses the gap in literature combining deep learning with UAV imagery by examining methods, applications, datasets, challenges, and future perspectives.
- Motivation: Automatic, rapid, and accurate processing is needed because visual inspection of multiple remote-sensing objects remains time-consuming, biased, and inaccurate.Deep learning offers robust and intelligent methods for improving Earth-surface mapping.
- Research gap: Existing remote-sensing reviews did not specifically survey deep-learning algorithms applied to UAV image sets, despite the growing volume of UAV data.A word-cloud analysis found “UAV” underrepresented or absent in related literature revisions.
- Motivation: UAV platforms provide flexible, low-cost, high-detail data collection with RGB, multispectral, hyperspectral, thermal, and LiDAR sensors.They can also gather data from difficult-to-access places and support flights programmed according to user demand.
- Study purpose: The study reviews deep-learning fundamentals and applications for UAV-based classification, object detection, and semantic segmentation in remote sensing.It presents these approaches in the context of UAV-image mapping tasks.
- Study contributions: The review examines published sensors and applications across environmental, urban, and agricultural mapping contexts.It also organizes publicly available UAV-acquired datasets labeled for object detection and segmentation tasks.
- Study contributions: The study describes challenges and future perspectives for applying deep-learning methods to UAV-based image data.Its broader aim is to summarize the direction of UAV-imagery applications in the remote-sensing community.
2 Deep Neural Networks Overview
Deep neural networks transform input data into predictions by progressively learning higher-level features through hidden layers and activation functions. In UAV remote sensing, their architectures support classification, segmentation, object detection, and regression tasks, with specialized networks suited to different data and applications.
- DNN fundamentals: DNNs transform inputs such as UAV imagery into outputs such as land-use and land-cover maps by progressively learning higher-level features through hidden layers.Neurons use activations and parameters to transform data, while subsequent hidden layers learn feature characteristics before producing a prediction.
- DNN fundamentals: Activation functions compute weighted sums of inputs and biases to determine neuron activation and help networks learn intrinsic patterns.Dense layers commonly implement these decision functions, which govern how neurons learn from interactions with other neurons.
- DNN architectures: Common DNN architectures include CNNs, RNNs, DBNs, and GANs, with model complexity determined by layer structure and additional computational methods.CNN-LSTM networks combine convolutional feature extraction with LSTM processing and can serve multitemporal applications.
- Remote-sensing tasks: Remote-sensing DL tasks include scene-wise classification, semantic and instance segmentation, and object detection, which labels images, assigns pixel classes, individualizes objects, or draws bounding boxes.Instance segmentation adds object individualization to pixel-wise classification, while object detection assigns class labels to detected bounding boxes.
- Remote-sensing tasks: Deep regression replaces the final fully connected classification layer to estimate continuous values, although it is less commonly used and often relies on application-specific networks.Fine-tuned VGG-16 and ResNet-50 have produced interesting results, but their application-specific development limits general-purpose use.
3 Deep Learning in UAV Imagery
The review finds that deep learning in UAV imagery is dominated by CNNs, classification, object detection, RGB sensing, and environmental applications. It also highlights segmentation’s growing role, sensor-specific task patterns, and future directions including real-time processing, dimensionality reduction, domain adaptation, attention mechanisms, and few-shot learning.
- Future perspectives: The review identifies real-time processing, data dimensionality reduction, domain adaptation, attention-based mechanisms, and few-shot learning as prominent future directions.It presents DL as increasingly useful for replacing laborious human tasks and improving conventional measurements across diverse UAV remote-sensing applications.
- Architectures: 91.2% of UAV-based application papers implemented CNNs, mostly using established architectures and emphasizing comparison with state-of-the-art models.The review notes that only a small portion proposed new models and compared them against established networks.
- Task types: 97.7% of papers evaluated classification tasks, compared with 2.3% using regression to estimate continuous data.Regression applications were identified across environmental, urban, and agricultural contexts.
- Task types: 53.9% of applications addressed object detection, 40.7% image segmentation, and 5.4% scene-wise classification.The review associates object detection’s prevalence with UAV imagery’s high spatial detail, which helps reveal objects while creating challenges.
- Application contexts: 46.6% of applications belonged to the environmental context, while urban and agricultural categories were almost evenly distributed.Environmental studies included land-use and change analysis, hazards, erosion, wildlife, forest inventories, and monitoring difficult-to-access regions.
- Sensors: 52.4% of papers used RGB sensors, followed by multispectral at 24.3%, hyperspectral at 17.8%, and LiDAR at 5.5%.RGB preference is associated with low-cost UAV systems and object-recognition features such as pixel size, form, and texture; segmentation used all listed sensor types.
4 Publicly Available UAV-Based Datasets
The section addresses the scarcity of labeled UAV-acquired data, especially multispectral and hyperspectral imagery, by providing publicly available datasets spanning urban and rural scenarios for developing and comparing deep-learning methods.
- Dataset contribution: The authors provide UAV-based datasets from both urban and rural scenarios to support future development and performance comparison of novel DL-based methods.Table 1 summarizes dataset information and indicates recent publications using previously conducted approaches.
- Dataset contribution: The datasets are intended to address the scarce availability of labeled examples from UAV-acquired data, particularly multispectral and hyperspectral data.
- Dataset access: The labeled datasets are available through the Geomatics and Computer Vision/Datasets webpage, which is intended to be continually updated with new datasets.
5 Perspectives in Deep Learning with UAV Data
The section identifies promising directions for combining deep learning with UAV imagery, while emphasizing challenges in preserving high resolution, enabling real-time processing, and handling computational demands. It also highlights quantization, efficient architectures, embedded hardware, and hyperspectral imaging as important avenues for future UAV mapping research.
- Future directions: Combining emerging deep-learning and computer-vision topics with remote-sensing data could support novel approaches to UAV mapping.These topics may also be combined with one another, as demonstrated in some studies.
- High-resolution processing: Traditional deep-learning architectures often scale images to approximately 512 × 512 pixels, potentially failing to exploit UAVs’ higher-resolution imagery.Future processing should maintain high resolution in deeper network layers.
- Real-time processing: Real-time UAV applications remain constrained because deep-learning algorithms are computer-intensive and commonly require data centers or dedicated GPUs for post-processing.Deep learning accelerates information extraction after training, but computational bottlenecks still hinder real-time use.
- Model efficiency: Quantization reduces model memory by representing weights with 16, 8, or 1 bit instead of 32-bit floating points, but ResNet-18 accuracy falls from 89.2% to 73.2% top-5 accuracy.The reported values compare full-precision ResNet-18 with its XNOR-Net version on ImageNet.
- Model efficiency: Architectures such as MobileNets, CEM, and SAM reduce computational requirements while retaining predictive performance in specific tasks such as object detection.These approaches are proposed as strategies for developing faster deep-learning models.
- Embedded hardware: Embedded AI platforms such as NVIDIA Jetson can run deep-learning algorithms, but UAV remote-sensing research using such hardware remains limited.The identified gap can be addressed by future work involving GPU, ASIC, FPGA, and MCU platforms.
- Hyperspectral imaging: UAV hyperspectral imaging offers tens to hundreds of spectral bands for object classification, but band correlation and increased computational cost create high-dimensionality challenges.The high dimensionality can also produce the Hughes phenomenon.
5.3 Domain Adaptation and Transfer Learning
Domain adaptation and transfer learning address spectral shifts and limited generalization when UAV imagery changes across regions, dates, equipment, or object classes. The review highlights few-shot, attention-based, semi-supervised, and related learning strategies as promising ways to reduce labeling and retraining demands.
- Domain Adaptation: Spectral shifts between training and test images arise from differences in acquisition, geography, and atmospheric conditions, motivating domain adaptation from a source domain to a different target domain.GAN-based methods can translate source images to mimic target-domain characteristics, while newer approaches align source and target distributions.
- Attention Mechanisms: High-resolution remote sensing imagery contains abundant information and increasing intra-class variation, making attention mechanisms useful for emphasizing valuable features and regions.When objects are split into patches, attention can aggregate relevant structural and discriminative information by combining global context with low- and high-level features.
- Few-Shot Learning: Few-shot learning targets poor generalization across geographical areas and new object classes without requiring a robust labeled dataset for every new setting.In a UAV maize-detection study, transfer learning fine-tuned CenterNet with 150 images from a new area, compared with 600 images used previously.
- Unsupervised and Semi-Supervised Learning: Expensive, time-consuming labeling motivates unsupervised and semi-supervised learning that incorporate unlabeled remote sensing images.Future directions include contrastive-loss and clustering-based approaches; one contrastive-loss method surpassed its supervised pre-trained counterpart.
- Multitask Learning: Multitask learning can accelerate learning and reduce overfitting while improving UAV detection when related tasks are modeled together.Reported UAV applications jointly addressed semantic segmentation, height estimation, and boundary detection, or plant and plantation-line detection; incorporating line detection improved plant detection.
6 Conclusions
The review finds that deep learning has produced important discoveries in UAV remote sensing while remaining largely a black-box solution. It identifies sensor, dataset, architecture, processing, and emerging research priorities for UAV image applications.
- 6 Conclusions: Deep learning remains largely a “black-box” solution, although research is advancing methods to reduce this limitation.The review nevertheless reports important discoveries across remote sensing implementations.
- 6 Conclusions: Most UAV remote-sensing publications use object detection methods and RGB sensors, while precision-agriculture and forest applications benefit from multi/hyperspectral data.
- 6 Conclusions: Additional labeled public UAV datasets are needed for network training and benchmarking, and the review contributes agricultural and environmental UAV datasets to a repository.
- 6 Conclusions: CNNs are the most adopted architecture, while CNN-LSTMs and GANs are gaining attention and may benefit future UAV remote-sensing image applications.
- 6 Conclusions: GPU-assisted deep learning can provide fast inference, but embedded real-time processing on UAVs requires further investigation.
- 6 Conclusions: Open-set, attention-based, few-shot, and multitask learning can be combined into novel UAV remote-sensing approaches that significantly improve DNN generalization capacity.
Conflicts of Interest
The authors declare no conflict of interest, and the funders had no role in the study’s design, data collection, analysis, interpretation, writing, or publication decision.
- Conflicts of Interest: The authors declare no conflict of interest, and funders had no role in the study design, data collection, analyses, interpretation, manuscript writing, or decision to publish.
Abbreviations
This section defines abbreviations used throughout the manuscript, including terms for deep learning, neural networks, remote-sensing models, data products, and computing concepts.
- The manuscript abbreviates core concepts and models, including AI, ANN, CNN, DL, DNN, GAN, LSTM, and DCGAN.It also defines AdaGrad, CEM, DDCN, and KL.
- Remote-sensing data products are abbreviated as DEM and DSM, while FPS, GPU, and IoU denote processing, hardware, and evaluation terms.FPS means Frames per Second, GPU means Graphics Processing Unit, and IoU means Intersection over Union.