Source-linked AI summary
Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review
Jiaxin Li, Danfeng Hong, Lianru Gao, Jing Yao, Ke Zheng, Bing Zhang, Jocelyn Chanussot
TL;DR
Multimodal RS fusion must handle heterogeneous observations and limitations of traditional algorithms. This survey defines the topic, analyzes its literature, reviews major modality-based subfields, compiles resources, and discusses challenges and future directions. It finds that DL has been applied across multimodal RS data fusion and has achieved promising results in recent years.
Problem
Traditional multimodal RS fusion algorithms face a performance bottleneck because they cannot comprehensively analyze and interpret strongly heterogeneous data.
Method
The survey defines multimodal RS data fusion, analyzes literature trends, reviews major modality-based subfields, compiles resources, and discusses challenges and prospects.
Results
DL has been applied across multimodal RS data fusion and has achieved tremendous and promising achievements in recent years.
Takeaways & Limitations
The review provides a comprehensive introduction to a fast-growing multimodal RS data-fusion domain and supports further research through synthesized resources and analysis.
Takeaways & Limitations
Some reviewed solutions assume perfectly co-registered inputs, while another solution is limited to the spatiospectral fusion domain.
Abstract
from arXiv · showhide
With the extremely rapid advances in remote sensing (RS) technology, a great quantity of Earth observation (EO) data featuring considerable and complicated heterogeneity is readily available nowadays, which renders researchers an opportunity to tackle current geoscience applications in a fresh way. With the joint utilization of EO data, much research on multimodal RS data fusion has made tremendous progress in recent years, yet these developed traditional algorithms inevitably meet the performance bottleneck due to the lack of the ability to comprehensively analyse and interpret these strongly heterogeneous data. Hence, this non-negligible limitation further arouses an intense demand for an alternative tool with powerful processing competence. Deep learning (DL), as a cutting-edge technology, has witnessed remarkable breakthroughs in numerous computer vision tasks owing to its impressive ability in data representation and reconstruction. Naturally, it has been successfully applied to the field of multimodal RS data fusion, yielding great improvement compared with traditional methods. This survey aims to present a systematic overview in DL-based multimodal RS data fusion. More specifically, some essential knowledge about this topic is first given. Subsequently, a literature survey is conducted to analyse the trends of this field. Some prevalent sub-fields in the multimodal RS data fusion are then reviewed in terms of the to-be-fused data modalities, i.e., spatiospectral, spatiotemporal, light detection and ranging-optical, synthetic aperture radar-optical, and RS-Geospatial Big Data fusion. Furthermore, We collect and summarize some valuable resources for the sake of the development in multimodal RS data fusion. Finally, the remaining challenges and potential future directions are highlighted.
1. Introduction
Multimodal remote sensing fusion combines heterogeneous observations to overcome the limits of unimodal data, while deep learning adaptively learns representations and fusion strategies. This review defines modalities, surveys DL-based fusion subdomains, analyzes prior literature, and provides resources, challenges, and prospects.
- Motivation: Each RS modality captures only some scene properties, so multimodal fusion integrates complementary information for more robust and reliable decisions.The paper gives change detection and LULC classification as example tasks.
- Definition: RS modality is characterized by sensor technical specifications and acquisition conditions, including spatial, spectral, radiometric, temporal, and observational properties.The paper notes that multimodal RS data fusion includes both multisource and multitemporal fusion.
- Motivation: Traditional fusion methods rely on handcrafted, domain-specific features and rough strategies, which can impair performance for heterogeneous data.Deep learning instead models complex input–output relationships while automatically extracting and fusing features.
- Review scope: The review organizes DL-based multimodal RS fusion into a unified framework according to fused modalities and corresponding tasks.It focuses on methods, modalities, and related tasks across the covered fusion subdomains.
- Review scope: Existing reviews often emphasize DL less or cover only a few subareas, motivating this survey’s comprehensive and systematic treatment of popular DL-based fusion domains.The paper also compiles tutorials, datasets, and codes and discusses challenges and prospects.
2. Literature analysis
The literature analysis uses Web of Science retrieval and CiteSpace to characterize publication trends, geographic and journal distributions, and keyword patterns in DL-based multimodal RS data fusion. Results show steadily growing attention, broad use of CNNs, and concentration on several fusion tasks and modalities.
- Data retrieval and collection: The analysis retrieves 598 article papers published from 2015 to 2022 after filtering Web of Science results by document type.An example query initially returned 691 results.
- Publication trends: Publication counts and their proportions increased steadily from 2015 to 2022, indicating growing attention to DL-based multimodal RS data fusion.The paper interprets the upward trends as evidence of DL’s increasing importance in this field.
- Countries and journals: The top 10 countries account for about 90% of total outputs, with China contributing more than half and the USA about 10%.These proportions are reported from the country-distribution analysis.
- Countries and journals: Remote Sensing, IEEE Transactions on Geoscience and Remote Sensing, and IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing comprise about half of publications.Remote Sensing ranks first among the listed journals.
- Keyword analysis: Keyword analysis identifies CNN as widely used, with classification, cloud removal, and object detection as major tasks involving MS, HS, LiDAR, and SAR.Figure 4 uses larger font size to represent higher keyword frequency.
3. A review of DL-based multimodal remote sensing data fusion methods
The paper organizes DL-based multimodal RS data fusion into homogeneous and heterogeneous fusion, using distinct criteria for diverse subdomains. Figure 5 presents this taxonomy alongside representative literature.
- Taxonomy: Existing methods are divided into homogeneous fusion and heterogeneous fusion.Homogeneous fusion includes pansharpening, HS pansharpening, HS-MS, and spatiotemporal fusion; heterogeneous fusion includes HS-optical, SAR-optical, and RS-GBD fusion.
- Taxonomy: The review adopts different criteria to introduce each subdomain because these fusion areas have developed diversely.
- Taxonomy: Figure 5 summarizes the paper’s taxonomy of DL-based multimodal RS data fusion.The review also lists classic literature in each direction for reader convenience.
3.1. Homogeneous fusion
Homogeneous fusion addresses resolution trade-offs in optical imagery by combining related observations across spatial, spectral, and temporal dimensions. The review covers pansharpening, HS pansharpening, HS-MS fusion, and spatiotemporal fusion methods.
- Overview: Homogeneous fusion targets trade-offs between spatial-spectral and spatial-temporal resolutions caused by optical imaging mechanisms.
- Pansharpening: Supervised pansharpening commonly relies on simulated references, whereas unsupervised methods train at the original scale using loss functions that preserve spatial and spectral consistency.
- Pansharpening: Pansharpening methods progress from CNN architectures that concatenate Pan and up-sampled MS inputs toward multi-branch, hybrid, GAN, and Transformer designs.Multi-branch networks separately extract modality-specific features, while hybrid methods combine traditional-method concepts with learned representations.
- Pansharpening: Simply stacking pre-interpolated MS with Pan ignores individual features and increases computational burden.
- HS pansharpening: HS pansharpening extends pansharpening architectures with attention, residual, adversarial, and Transformer designs to combine HS spectral information with Pan spatial detail.Unsupervised HS pansharpening remains rarely studied because the spectral-range discrepancy between Pan and HS complicates spatial-information preservation.
- HS-MS fusion: Pansharpening methods can be transferred to HS-MS fusion with necessary modifications because both tasks fuse paired observations to produce a high-spatial-resolution hyperspectral product.
3.2. Heterogeneous fusion
Heterogeneous fusion integrates modalities with different imaging mechanisms, commonly at feature or decision levels. The review discusses LiDAR-optical, SAR-optical, and RS-Geospatial Big Data fusion across multiple applications.
- Overview: Heterogeneous fusion integrates LiDAR-optical, SAR-optical, and RS-GBD data whose imaging mechanisms differ substantially.Feature-level and decision-level fusion are widely adopted for these modalities.
- LiDAR-optical fusion: HS-LiDAR fusion supplements insufficient hyperspectral discrimination with LiDAR’s three-dimensional spatial geometry.
- LiDAR-optical fusion: LiDAR-optical methods increasingly improve feature extraction and fusion through separate branches, attention, cross-attention, reconstruction, graph, and feature-decision networks.The review focuses particularly on HS-LiDAR fusion and describes its use in LULC classification and related tasks.
- SAR-optical fusion: SAR-optical fusion supports pixel-, feature-, and decision-level applications including registration, cloud removal, classification, change detection, and biomass estimation.SAR’s weather independence supports cloud-free optical reconstruction, while speckle noise remains a practical constraint.
- SAR-optical fusion: SAR-optical networks use separate branches and specialized losses or sparse constraints to extract heterogeneous features, reduce speckle, preserve structure, and limit redundancy.
- RS-GBD fusion: RS-GBD fusion combines physical attributes from remote sensing with socioeconomic descriptions from sources such as POI, social media, and vehicle trajectories.Large structural differences between GBD and RS prevent direct use of standard dual-branch modality-specific networks.
4. List of resources
The paper assembles tutorials, datasets, and open-source code to support research and practical model selection in DL-based multimodal RS data fusion. These resources cover foundational learning, evaluation data, and implementation comparison.
- Resources: The review collects tutorials, multimodal RS datasets, and open-source codes to support development and communication in this field.
- Tutorials: Tutorials and references introduce RS modalities, DL models, and successful RS applications for beginners.
- Datasets: Available multimodal RS datasets are presented as necessary for evaluating algorithms and selecting models for practical applications.The review supplements datasets from the IEEE Data Fusion Contest with additional available datasets.
- Datasets and code: The authors summarize available datasets and open-source implementations to facilitate comparison between different approaches.
5. Problems and prospects
The review identifies unresolved challenges in DL-based multimodal RS data fusion, including registration, quality assessment, modality scalability, missing modalities, and interpretability. It outlines prospects involving integrated preprocessing, application-oriented evaluation, broader modality handling, crossmodal learning, and more transparent models.
- From well-registered to non-registered: High-precision alignment is especially difficult for HS-Pan and HS-MS data because paired acquisitions under identical conditions are harder to obtain than Pan-MS pairs.The review therefore recommends incorporating registration into the fusion process.
- From well-registered to non-registered: Registration remains a prerequisite for many fusion tasks, yet much work assumes perfectly co-registered inputs and overlooks this preprocessing step.Only a few studies jointly perform registration and fusion, including an end-to-end unsupervised network for unregistered HS-MS data.
- From image-oriented to application-oriented quality assessment: Existing quality indexes do not comprehensively and fairly compare fusion methods, while reference images may be unavailable in real-world applications.The review points toward reference-free and application-related indexes, including indicators tied to downstream tasks.
- From two-modality to multi-modality: Most DL-based fusion algorithms handle only two modalities, limiting their applicability as increasingly diverse sensor data become available.A proposed direction is a unified framework capable of processing an arbitrary number of modalities.
- From multimodal to crossmodal learning: Practical scenarios may lack one modality, motivating crossmodal learning that transfers information from multimodal regions to areas where a modality is missing.A representative approach uses limited HS-MS or HS-SAR pairs to support large-scale classification in MS- or SAR-only areas.
- From black-box to interpretable learning: DL fusion remains difficult to interpret because learned high-level features are hard to explain, and model-driven transparency is currently limited mainly to spatiospectral fusion.The review connects model-driven and data-driven methods as a route toward more transparent networks, while noting limits for high-level fusion.
6. Conclusion
The review surveys DL-based multimodal RS data fusion through literature analysis, systematic coverage of major sub-fields, resource compilation, and future prospects. It concludes that DL has been applied broadly across the field and has achieved promising results compared with traditional methods.
- Conclusion: Artificial intelligence-related technologies demonstrate advantages over traditional methods through their feature-extraction capability.The conclusion situates DL-based multimodal fusion within the growth of RS big data and cutting-edge tools.
- Conclusion: The review provides a comprehensive introduction through literature analysis, systematic summaries of prevalent RS fusion sub-fields, available resources, and future prospects.Its detailed analysis focuses on DL-based methods in multimodal RS data fusion.
- Conclusion: DL has been applied across multimodal RS data fusion and has achieved tremendous, promising results in recent years.The conclusion presents this broad adoption and progress as encouragement for further study.