PaperlayerView exact revision on arXiv

Source-linked AI summary

Multisource and Multitemporal Data Fusion in Remote Sensing

Pedram Ghamisi, Behnood Rasti, Naoto Yokoya, Qunming Wang, Bernhard Hofle, Lorenzo Bruzzone, Francesca Bovolo, Mingmin Chi, Katharina Anders, Richard Gloaguen, Peter M. Atkinson, Jon Atli Benediktsson

arXiv:1812.08287v1cs.LGeess.SPstat.ML

TL;DR

Heterogeneous remote-sensing and ancillary data pose processing challenges, while their combination offers opportunities for application-specific utility. This review organizes multisource and multitemporal fusion advances by research community and modality, outlining challenges and future directions. It aims to provide a detailed starting point for further investigation, while excluding some topics such as SAR and optical data fusion.

  • Problem

    The growing volume and heterogeneity of remote-sensing data create a challenge of how to combine multiple sources for a given application.

    Source: §I · p. 2

  • Method

    The review organizes multisource and multitemporal fusion advances by data modality and the research communities in which methods developed.

    Source: §I · p. 2, §VII · p. 20

  • Results

    The review details pansharpening and resolution enhancement, point-cloud fusion, hyperspectral and LiDAR fusion, multitemporal fusion, and big-data and social-media fusion, outlining challenges and future directions.

    Source: §VII · p. 20

  • Takeaways & Limitations

    The review aims to provide readers a detailed starting point for investigating remaining challenges in fusion approaches suited to applications.

    Source: §VII · p. 21

    The review is restricted in scope and does not address some topics, including SAR and optical data fusion.

    Source: §I · p. 2

Abstract

from arXiv · show

The sharp and recent increase in the availability of data captured by different sensors combined with their considerably heterogeneous natures poses a serious challenge for the effective and efficient processing of remotely sensed data. Such an increase in remote sensing and ancillary datasets, however, opens up the possibility of utilizing multimodal datasets in a joint manner to further improve the performance of the processing approaches with respect to the application at hand. Multisource data fusion has, therefore, received enormous attention from researchers worldwide for a wide variety of applications. Moreover, thanks to the revisit capability of several spaceborne sensors, the integration of the temporal information with the spatial and/or spectral/backscattering information of the remotely sensed data is possible and helps to move from a representation of 2D/3D data to 4D data structures, where the time variable adds new information as well as challenges for the information extraction algorithms. There are a huge number of research works dedicated to multisource and multitemporal data fusion, but the methods for the fusion of different modalities have expanded in different paths according to each research community. This paper brings together the advances of multisource and multitemporal data fusion approaches with respect to different research communities and provides a thorough and discipline-specific starting point for researchers at different levels (i.e., students, researchers, and senior researchers) willing to conduct novel investigations on this challenging topic by supplying sufficient detail and references.

I. INTRODUCTION

Remote sensing data are growing in volume and heterogeneity, creating processing challenges and opportunities for fusion. This review organizes fusion approaches by data modality and describes the measurement, sampling, and application considerations that shape them.

I. INTRODUCTION

  • Heterogeneous remote-sensing and ancillary datasets create a challenge of how best to combine sources for application-specific utility.Sensors differ in measured properties and spatial and spectral resolutions, including sensors on the same platform.Source: §I · pp. 1–2
  • The review groups fusion approaches by data modality because methods have developed along different paths in separate research communities.It does not aim to synthesize them into a single framework and excludes some topics, including SAR and optical data fusion.Source: §I · p. 2
  • Because measurement and sampling limit information transfer from a scene to any dataset, fusion methods operate on different parts of the scene-to-data sequence.The review focuses on fusion to predict measured properties x, rather than properties of interest y.Source: §I · pp. 2–3
  • Fusion methods combine observations that represent different views of the same real world, relying on expected coherence between the source and observations and among datasets.Spatial correlation and expected object sizes and shapes can also constrain possible solutions.Source: §I · p. 3
  • Fusion applications include combining elevation with optical data for classification and mapping, and combining spatially and spectrally detailed imagery to reduce mixed pixels.The introduction also identifies ancillary data as a source for smart-city, smart-environment, and hazard applications.Source: §I · pp. 3–4
  • Fine-resolution time series can support monitoring of seasonal or rapid changes, but large data volumes and urgent events create demands for case-specific processing and speed.The introduction cites phenology, deforestation, fires, and floods as examples of applications.Source: §I · p. 4

II. PANSHARPENING AND RESOLUTION ENHANCEMENT

This section reviews spatio-spectral and spatio-temporal fusion, which address trade-offs among optical imagery’s spatial, spectral, and temporal resolutions. It introduces spatio-spectral fusion approaches and explains component substitution methods and their trade-offs.

II. PANSHARPENING AND RESOLUTION ENHANCEMENT

  • Optical fusion approaches address either fine-spatial and fine-spectral resolution or fine-spatial and fine-temporal resolution.The section reviews these spatio-spectral and spatio-temporal technologies with recent advances.Source: §II · p. 4, Figure 2 · p. 4

A. Spatio-spectral fusion

  • Spatio-spectral fusion combines fine-spatial imagery with coarser-spatial imagery to create fine-spatial imagery across bands; a single fine-resolution PAN image defines pansharpening.When multiple fine-spatial bands are available, the task is called multiband image fusion.Source: A. Spatio-spectral fusion · p. 4
  • Five common spatio-spectral fusion approaches are component substitution, multiresolution analysis, geostatistical analysis, subspace representation, and sparse representation.The section’s figure presents representative literature for these categories, with circle size proportional to annual average citations.Source: A. Spatio-spectral fusion · p. 5, Figure 3 · p. 5

1) Component Substitution:

  • Component substitution transforms multispectral data, replaces a spatial-information component with intensity-adjusted PAN data, then applies the inverse transform.A general framework injects PAN-derived spatial detail using band-wise modulation coefficients and can mitigate spectral distortion through regression-based synthetic components.Source: 1) Component Substitution · p. 5
  • Component-substitution methods offer spatial-detail fidelity, low computational complexity, and robustness to misregistration, but limited spectral-response overlap can cause global spectral distortion.The limitation concerns overlap between the sensors’ spectral response functions.Source: 1) Component Substitution · p. 5

2) Multiresolution Analysis:

The review groups fusion approaches by method family, describing how they combine spatial details or model relationships between images, and outlining their trade-offs. It covers multiresolution, geostatistical, and subspace methods.

2) Multiresolution Analysis:

  • MRA pansharpening injects PAN-derived spatial details, multiplied by gain coefficients, into multispectral data; hypersharpening extends this approach to multiband fusion.Hypersharpening creates a fine-resolution synthetic image for each coarse-resolution band using linear combinations of fine-resolution bands.Source: 2) Multiresolution Analysis · p. 5
  • MRA methods preserve spectral consistency but have greater computational complexity than component-substitution techniques.Their spectral consistency means spatially degrading the fused image yields a spectrally consistent version of the coarse-spatial, fine-spectral input.Source: 2) Multiresolution Analysis · p. 5

3) Geostatistical Analysis:

  • DSCK fuses multispectral bands with PAN using cokriging, modeling auto- and cross-semivariograms; adaptive filtering can vary its weights by pixel.Extensions also downscale multispectral bands to finer resolution than any input image.Source: 3) Geostatistical Analysis · p. 5
  • KED simplifies semivariogram modeling by requiring only the observed coarse band's auto-semivariogram, but computes kriging weights locally for each fine pixel.Its computing time increases linearly with the number of fine pixels predicted.Source: 3) Geostatistical Analysis · pp. 5–6
  • ATPRK estimates an overall regression trend, downscales its residuals with area-to-point kriging, then adds them back; it requires only auto-semivariogram modeling.Compared with KED, ATPRK calculates kriging weights only once and is described as much faster; extensions include adaptive and approximate versions.Source: 3) Geostatistical Analysis · p. 6

4) Subspace Representation:

  • Subspace methods estimate fine-spectral basis vectors and fine-spatial coefficients, but iterative optimization can be costly and imperfect representations can create spectral artifacts.The family includes Bayesian, matrix-factorization, and spectral-unmixing methods; unmixing reconstructs the output from endmembers and their estimated fine-resolution abundances.Source: 4) Subspace Representation · p. 6

5) Sparse Representation:

The review describes sparse-representation pansharpening and spatio-temporal fusion approaches, including their assumptions and computational or prediction challenges. In a Hong Kong WorldView-2 comparison, J-SparseFI-HM further increased accuracy beyond the competitive baselines and ATPRK.

5) Sparse Representation:

  • Sparse-representation pansharpening learns patch correspondences from a dictionary and can reconstruct output patches from non-local fine-resolution patches, but may require supercomputers for acceptable processing times.Using the PAN image to construct a dictionary supports the setting where only one PAN–multispectral pair is available.Source: 5) Sparse Representation · p. 6
  • J-SparseFI-HM further increases accuracy beyond the competitive GSA, SFIM, and MTF-GLP-HPM baselines and ATPRK in the Hong Kong WorldView-2 assessment.The assessment compares five pansharpening methods; the cited table caption identifies its scope but supplies no individual metric values here.Source: 5) Sparse Representation · p. 6
  • In the visual comparison, GSA, SFIM, MTF-GLP-HPM, and ATPRK show sharper edges but boundary artifacts, while J-SparseFI-HM produces visually natural results.The reported artifacts occur along boundaries such as water and vegetation where PAN brightness is reversed relative to each band.Source: 5) Sparse Representation · p. 7, Figure 4 · p. 7

B. Spatio-temporal fusion

  • Spatio-temporal fusion blends fine-spatial, coarse-temporal data with coarse-spatial, fine-temporal data; methods are generally image-pair-based or spatial-unmixing-based.The motivation includes the trade-off between spatial resolution and revisit frequency.Source: B. Spatio-temporal fusion · p. 7
  • STARFM assumes consistent temporal change within each coarse pixel, favoring homogeneous landscapes; ESTARFM uses two image pairs and assumes class-specific change rates remain stable.Other image-pair-based methods learn relationships from available coarse–fine pairs to guide predictions on other days.Source: B. Spatio-temporal fusion · p. 7
  • Spatial-unmixing methods can use one fine-resolution land-cover map, but assume no land-cover or land-use change during the period of interest.Extensions also use one coarse–fine pair or two pairs to estimate changes in class endmember spectra.Source: B. Spatio-temporal fusion · pp. 7–8
  • Spatio-temporal prediction is ill-posed, with uncertainty especially for abrupt changes and heterogeneous landscapes; adding 250 m MODIS data can provide more information than 500 m data.The cited review states that this additional information can increase prediction accuracy.Source: B. Spatio-temporal fusion · p. 8
  • Fit-FC fuses Sentinel-2 and Sentinel-3 to create nearly daily Sentinel-2 images through regression fitting, spatial filtering, and residual compensation, using only one image pair.The method is described as particularly relevant when temporal changes are strong.Source: B. Spatio-temporal fusion · p. 8

C. Challenges and trends of downscaling

Fair comparison remains a major challenge in spatio-spectral fusion, while point-cloud fusion must contend with heterogeneous, unstructured 3D data and the need for explicit, task-dependent neighborhood derivation.

C. Challenges and trends of downscaling

  • Spatio-spectral fusion lacks fair comparisons because datasets may be insufficiently diverse or unavailable, and many methods' source code is unreleased.The paper calls for accessible benchmark datasets covering varied scenes and release of method code.Source: C. Challenges and trends of downscaling · p. 8
  • The GRSS DASE website is proposed as a platform for benchmark datasets enabling fair algorithm competition.The cited passage describes this as an urgent community need.Source: C. Challenges and trends of downscaling · p. 8

III. POINT CLOUD DATA FUSION

  • Point clouds span applications from country-wide topographic mapping to near-real-time autonomous driving and are derived mainly from LiDAR or photogrammetry.Tomographic SAR and low-cost depth cameras are also emerging sources,.Source: §III · p. 9

A. Point cloud data model

  • A georeferenced point is a 3D coordinate vector that may carry variable additional features, but its unstructured spatial distribution complicates fusion.Neighborhoods must be derived explicitly, at computational cost, and their definition depends on the application and processing task,.Source: A. Point cloud data model · p. 9

B. Concepts of point cloud fusion

Point-cloud fusion strategies are distinguished by their target product, from enriching points to creating image or voxel layers and assigning point-cloud information to objects. Point-level methods include image texturing, classification, and registration, each with different uses and constraints.

B. Concepts of point cloud fusion

  • Fusion targets fall into three product levels: add features to points, derive image or voxel layers, or fuse information at segment/object level.The target data model also shapes the processing methods used for classification.Source: B. Concepts of point cloud fusion · p. 9

1) Point cloud level - Pixel to point and point to point:

  • Texturing adds image pixel values such as RGB to 3D points when image-to-point transformations are known or reconstructed from multiple images.These colored points can support classification and object detection.Source: 1) Point cloud level - Pixel to point… · p. 9
  • Joint LiDAR and image classification increased LiDAR-only accuracy by 2.3%, while image-domain workflows limit classification to single points without 3D neighborhoods.Vo et al. used RGB and laser intensity in supervised classification and then region growing to delineate roads; image-domain processing uses fast, established methods,.Source: 1) Point cloud level - Pixel to point… · pp. 9–10
  • Point-cloud co-registration aligns datasets from different sensors, accuracies, coverages, scales, or timestamps using point-, keypoint-, or surface-based methods.An image-based registration of airborne and terrestrial LiDAR was applied to the point clouds to assess tree-crown structure.Source: 1) Point cloud level - Pixel to point… · p. 10
  • Combining LiDAR and photogrammetry point clouds from three platforms can improve accuracy over using an individual source.The merged datasets were used to estimate bare-earth elevation accuracy and grass and shrub heights.Source: 1) Point cloud level - Pixel to point… · p. 10
  • Rasterizing point-cloud geometry and backscatter into image bands enables image-based classification, while CNN pixel labels can be back-projected to terrestrial LiDAR or photogrammetric points.Derived bands can encode elevation, point distributions, intensity, and features such as echo ratio and plane-fitting residuals,.Source: 2) Image/Voxel level -… · p. 10, 1) Point cloud level - Pixel to point… · p. 10

2) Image/Voxel level - Point-to-pixel/voxel:

Image/voxel fusion converts 3D point-cloud information into image layers or voxel features for classification, while feature/object approaches combine segment-level geometry and spectral information. Reported studies show improved classification, with performance depending on the fusion strategy and scene conditions.

2) Image/Voxel level - Point-to-pixel/voxel:

  • 88% overall accuracy for forest vegetation classification was reported with UAV LiDAR and hyperspectral data, compared with 76% using HSI alone.LiDAR produced the largest gains for vegetation classes that separate well in height.Source: 2) Image/Voxel level -… · p. 11
  • Voxel classification combined 3D geometry and spectral features, but shadow areas remained problematic and color-sensitive segmentation sometimes missed planes.The method used LiDAR and multispectral data for buildings, trees, vegetated ground, and sealed ground.Source: 2) Image/Voxel level -… · p. 11
  • 92.61% overall accuracy was achieved by fusing waveform LiDAR and HSI with SVM, versus 71.30% for LiDAR and 85.82% for HSI alone.The study classified nine land-cover classes using voxel-derived LiDAR features alongside principal components from the 72 HSI bands.Source: 2) Image/Voxel level -… · p. 11

3) Feature/Object level:

  • Object-level workflows derive segments and combine their 3D geometric or waveform features with spectral features for tree classification and biodiversity indicators.Examples include tree-crown segmentation followed by feature extraction and estimation of tree variables and forest indicators –.Source: 3) Feature/Object level · pp. 11–12
  • A combined pixel- and object-based HSI/LiDAR method reached 94.7% overall accuracy, a 7.1% increase over HSI classification alone.The workflow classified pixels and objects separately before merging their outputs.Source: 3) Feature/Object level · p. 11
  • A two-stage neural-network fusion of HSI and LiDAR achieved 98.5% overall accuracy, compared with 74.5% using HSI alone.The workflow segmented HSI and LiDAR separately before classifying the resulting segments and producing a 3D map.Source: 3) Feature/Object level · p. 12

C. Challenges and trends of point cloud fusion

Point-cloud fusion increasingly uses machine learning and rasterized 3D data, while heterogeneous sources and point-cloud time series remain areas for further work. The review highlights complementary HSI/LiDAR fusion and describes classification approaches and their evaluation on Houston data.

C. Challenges and trends of point cloud fusion

  • Most current point-cloud fusion approaches convert 3D information into images or voxels for analysis, then transfer classification labels back to points.The review also identifies increasing use of machine learning and joint spectral/3D data from single-source photogrammetry [104].Source: C. Challenges and trends of point… · p. 12
  • Future point-cloud fusion needs to address sources with heterogeneous point density and 3D accuracy; point-cloud time series create scope to investigate temporal fusion.The review notes that current fusion often uses one point-cloud source, such as HSI with LiDAR.Source: C. Challenges and trends of point… · p. 12

IV. HYPERSPECTRAL AND LIDAR

  • The review devotes a section to HSI and LiDAR raster fusion as contributions grow, reflecting the sensors' complementary information.LiDAR provides 3D structure but lacks spectral information, while HSI supplies detailed spectra but not elevation or material size.Source: C. Challenges and trends of point… · p. 12, §IV · p. 12

A. Houston University

  • The Houston section groups LiDAR/HSI classification approaches into filtering, low-rank, composite-kernel, and deep-learning methods and reports comparisons on a shared dataset.The data comprise a LiDAR-derived DSM and HSI captured over the University of Houston campus and nearby urban area; the accuracies are reported in Table II.Source: §IV · p. 12, A. Houston University · p. 12, Table II · p. 13
  • Feature fusion is reported to raise overall classification accuracy to 92.45%, versus 77.47% for HSI and 31.83% for LiDAR alone [138].The figure presents the result schematically as an HSI/LiDAR fusion comparison.Source: Figure 7 · p. 13

B. Filtering

Filtering extracts contextual and spatial features from HSI and LiDAR, while low-rank models reduce feature dimensionality before classification. The reviewed methods include dimensionality reduction, graph-based fusion, total-variation regularization, and sparse wavelet representations.

B. Filtering

  • Filtering with morphological, attribute, and extinction profiles extracts contextual and spatial features while attenuating redundant detail and preserving geometric characteristics.For HSI–LiDAR fusion, AP or EP features can be stacked and classified, with reported precise classification results and fast processing.Source: B. Filtering · p. 13
  • Filtering profiles can increase feature dimensionality without increasing training samples, creating a dimensionality and subsequent classification-time burden.Composite-kernel and low-rank approaches are presented as ways to address these shortcomings.Source: B. Filtering · pp. 13–14

C. Low-rank models

  • Low-rank models target the dimensionality and processing-time costs of filtering by representing extracted HSI and LiDAR features in a lower-dimensional space.The framework may use DR2 to equalize spectral, spatial, and elevation feature counts, reduce computational cost, and reduce noise; the figure caption says DR2 is optional and cited studies recommend it for more accurate maps.Source: C. Low-rank models · p. 14, B. Filtering · p. 14, Figure 9 · p. 14
  • The low-rank framework extracts base images, filters HSI and LiDAR for spatial and elevation features, optionally reduces dimensions, then fuses features for classification.Reviewed variants include graph-based feature fusion with SVM classification and concatenated spectral, spatial, and elevation features with GBFF and a 2D CNN.Source: C. Low-rank models · p. 14
  • OTVCA estimates a low-rank fusion of concatenated HSI, EPHSI, and EPLiDAR features, with spatial total variation promoting piecewise-smooth fused features.The reported result is listed as OTVCA in Table II.Source: C. Low-rank models · p. 14
  • The SLRCA approach represents fused features in a 2D wavelet basis and applies an ℓ1 penalty to wavelet coefficients to promote sparsity.The review says sparsity improves SNR and final classification accuracies; the results are listed as SLRCA in Table II.Source: C. Low-rank models · p. 14

D. Composite kernel

Composite-kernel approaches combine spectral, spatial, and elevation information in feature space. The reviewed methods differ in sensitivity to regularization and in how they learn or weight feature sets.

D. Composite kernel

  • Composite kernels handle spectral, spatial, and elevation features in feature space, partially overcoming shortcomings of filtering approaches.The review describes this as a partial remedy, not a complete resolution of filtering's limitations.Source: D. Composite kernel · p. 14
  • FFCK combines a local-region filter with composite kernels, but its classification accuracy is strongly influenced by μ, which trades off spectral against spatial-elevation kernels.The review contrasts it with MLRsub, which integrates feature sets without regularization parameters.Source: D. Composite kernel · p. 15
  • MLRsub uses attribute profiles and multiple-feature learning with a subspace multinomial logistic-regression classifier, avoiding regularization parameters for integrating feature sets.Its result is listed as MLRsub in Table II.Source: D. Composite kernel · p. 15
  • ALWMJ-KSRC weights multisource data for sparse representation, assigns labels by minimum ℓ2 distance to class-specific subdictionary representations, and also uses kernel tricks.The composite-kernel version's results are listed as ALWMJ-KSRC in Table II.Source: D. Composite kernel · p. 15

E. Deep learning

Deep architectures are discussed as a way to extract hierarchical features from nonlinear HSI data and fuse LiDAR information. The review also identifies transferability, limited training data, high dimensionality, and elevation estimation from optical imagery as research directions.

E. Deep learning

  • Deep architectures can extract hierarchical, abstract features that are usually invariant to nonlinearities in HSI inputs, which are magnified in multisensor data.The review attributes HSI nonlinearity to scattering, atmospheric and geometric distortions, and intraclass variability.Source: E. Deep learning · p. 15
  • A two-stream CNN extracts spatial and elevation features for fusion, followed by a fully connected deep network for classification.Other reviewed designs use a 2D CNN with GBFF or a three-stream CNN with multisensor composite kernels.Source: E. Deep learning · p. 15

F. Trends of hyperspectral and LiDAR fusion

V. MULTITEMPORAL DATA FUSION

Multitemporal fusion adds time to spatial and spectral or backscatter information, supporting analyses of phenomena from gradual evolution to abrupt change. The review distinguishes classification goals and surveys methods, while noting that deep learning faces computational and training-data constraints.

V. MULTITEMPORAL DATA FUSION

  • Time-series data support analysis of both slow, smooth phenomena and steep, abrupt changes, and the review presents a taxonomy of temporal phenomena observable in remote sensing.Fine-resolution, short-revisit satellite data make temporal information a source for scene interpretation and expand data representations from 3D to 4D.Source: §V · p. 15
  • Beyond time-series classification and change detection, dense fine-resolution multisensor series enable local vegetation phenology and field-level trend studies for precision farming.The review focuses its subsequent discussion on classification, including problem definitions, methods, and labeled-training-data challenges.Source: §V · pp. 15–16

A. Multitemporal Information in Classification

  • Multitemporal classification can produce a map for the latest acquisition, a map for every time point, or a seasonal or annual map of temporal-signature behavior.These objectives imply different classification approaches, and failure to identify and model the objective can yield suboptimal results.Source: A. Multitemporal Information in… · p. 16, Figure 12 · p. 17
  • Direct multidate classification stacks image feature vectors, but latest-date mapping assumes no land-cover changes, while mapping every date requires labeled examples for possible class combinations and transitions.The review notes that such comprehensive labeled data are seldom available in real applications.Source: A. Multitemporal Information in… · p. 16
  • Cascade classification links class probabilities across dates; later work extended this Bayesian framework to multisensor data using neural-network estimates from multispectral and SAR images.The literature also includes neural and kernel classifiers, multiple-classifier systems, and Markov Random Fields for spatiotemporal context,,,,.Source: A. Multitemporal Information in… · pp. 16–17
  • Deep learning for multitemporal classification remains challenging because 4D inputs increase model complexity and require training data that are currently far from available.The review identifies theoretically sound and computationally affordable architectures as an unresolved challenge,.Source: A. Multitemporal Information in… · p. 18

B. Challenges in Multitemporal Classification

Multitemporal classification is constrained by the difficulty of assembling labeled samples that represent class combinations, transitions, and variability across time and space. The review also discusses remote-sensing and social-media data fusion, whose heterogeneous formats call for tailored methods and remain an open challenge for deep learning.

B. Challenges in Multitemporal Classification

  • Training multitemporal classifiers requires samples representing all temporal classes, their transitions, and high spatial and temporal variability, which is difficult in real applications.The passage identifies the need for multitemporal ground references or reliably annotated multitemporal images.Source: B. Challenges in Multitemporal… · p. 18
  • Semi-supervised methods jointly use available labels and observed-image distributions to model spatio-temporal properties, with approaches including domain adaptation, active learning, and transfer learning.Transfer methods can propagate labels to unchanged pixels across acquisitions, while active learning targets portions of images with high multitemporal label uncertainty.Source: B. Challenges in Multitemporal… · p. 18
  • Deep-learning architectures still face an open challenge: learning spatio-temporal relations with feasible training-sample requirements without significantly degrading semantic extraction.The required samples must represent multitemporal classes, their transitions, and high temporal and spatial variability, constraints described as difficult to satisfy in real applications.Source: B. Challenges in Multitemporal… · p. 18

VI. BIG DATA AND SOCIAL MEDIA

  • Remote-sensing and social-media data offer global imagery and local, timely information for smart-city, environmental, hazard, and disaster applications.Social-media sources include geolocated posts, photos, video, and audio, and can contribute ground reference through social or crowd sensing.Source: §VI · pp. 18–19
  • Because remote-sensing and social-media data differ in properties and formats, conventional fusion methods cannot be applied directly; deep neural networks and accelerated computing are discussed as approaches to processing them.Remote-sensing data include multisource, multitemporal, and multiresolution images, whereas social media also includes text, video, and audio.Source: §VI · p. 19
  • Deep neural networks can fuse social-media photos with remote-sensing images at feature, decision, or combined feature-decision levels, with unified model design remaining a challenge.Feature fusion can integrate extracted features within or after separate networks; decision fusion combines classifications produced from the respective data.Source: §VI · p. 19, Figure 14 · p. 20
  • The paper describes a classification comparison using volunteer training labels informed by social-media photos, with overall accuracies and classification maps compared for SVM, CNN, and FCN models.The setup uses fine-resolution Jilin-1 imagery of Frankfurt; the supplied passages do not state the comparison's outcomes.Source: §VI · p. 20, Table III · p. 20

VII. CONCLUSIONS

The review addresses multisource and multitemporal remote-sensing data fusion across research communities, whose modality-specific methods have developed along different paths. It surveys prominent topics and their challenges, while noting that theoretical and application questions remain open.

VII. CONCLUSIONS

  • The growing availability of heterogeneous complementary datasets motivates fusion to improve the efficacy and efficiency of remote-sensing processing for the problem at hand.The breadth of multisensor and multitemporal fusion makes it challenging to cover the field comprehensively in one review.Source: §VII · p. 20
  • The review details pansharpening and resolution enhancement, point-cloud fusion, hyperspectral and LiDAR fusion, multitemporal fusion, and big data with social media, outlining challenges and future directions.It focuses on advances across research communities because fusion methods for different modalities have developed along different paths.Source: §VII · p. 20
  • Although remote-sensing data fusion is a mature field, theoretical and application research questions remain open.The review presents further investigation as relevant to developing fusion approaches suited to applications at hand.Source: §VII · pp. 20–21
Loading 1812.08287v1…