PaperlayerView exact revision on arXiv

Source-linked AI summary

Machine Learning Information Fusion in Earth Observation: A Comprehensive Review of Methods, Applications and Data Sources

S. Salcedo-Sanz, P. Ghamisi, M. Piles, M. Werner, L. Cuadra, A. Moreno-Martínez, E. Izquierdo-Verdiguier, J. Muñoz-Marí, Amirhosein Mosavi, G. Camps-Valls

arXiv:2012.05795v1cs.CVcs.LG

TL;DR

The paper addresses how to exploit an expanding, heterogeneous Earth-observation data deluge for analysis. It reviews ML information-fusion methods, applications, data sources, models, and case studies, concluding that fusion is central to successful EO applications while identifying future directions and unresolved challenges.

  • Problem

    EO data are rapidly growing in volume and heterogeneity, creating challenges for integrating multiple remote-sensing and ancillary datasets and for understanding underlying causal relations.

  • Method

    The paper provides a practical review of ML information fusion for EO, covering literature, applications, data sources, models, case studies, and future methodological directions.

  • Results

    The review finds that information fusion is a key step for successful EO applications and surveys case studies where ML fusion achieved excellent performance in real problems.

  • Takeaways & Limitations

    The growing availability, diversity, and improving resolution of EO sensors is expected to give ML information fusion a major impact in the coming years.

  • Takeaways & Limitations

    Current ML information-fusion methods have strong predictive capabilities but provide limited learning of underlying causal relations.

Abstract

from arXiv · show

This paper reviews the most important information fusion data-driven algorithms based on Machine Learning (ML) techniques for problems in Earth observation. Nowadays we observe and model the Earth with a wealth of observations, from a plethora of different sensors, measuring states, fluxes, processes and variables, at unprecedented spatial and temporal resolutions. Earth observation is well equipped with remote sensing systems, mounted on satellites and airborne platforms, but it also involves in-situ observations, numerical models and social media data streams, among other data sources. Data-driven approaches, and ML techniques in particular, are the natural choice to extract significant information from this data deluge. This paper produces a thorough review of the latest work on information fusion for Earth observation, with a practical intention, not only focusing on describing the most relevant previous works in the field, but also the most important Earth observation applications where ML information fusion has obtained significant results. We also review some of the most currently used data sets, models and sources for Earth observation problems, describing their importance and how to obtain the data when needed. Finally, we illustrate the application of ML data fusion with a representative set of case studies, as well as we discuss and outlook the near future of the field.

1. Introduction

Earth observation combines increasingly diverse observations, models, and computational methods to study a complex Earth system. The paper frames ML-based information fusion as a practical way to exploit these complementary sources for Earth-system analysis.

  • Earth system science integrates disciplines to describe interacting processes across the Earth’s spheres and support sustainable development.
  • Earth-system models encode physical knowledge through first-principles mechanistic modeling for understanding, forecasting, and simulation.
  • Earth observation combines station, sensor, and ancillary-model data to monitor Earth processes with unprecedented spatial and temporal resolution.
  • Big-data growth has produced heterogeneous observations from satellites, aircraft, drones, in-situ sensors, numerical models, reanalysis, and other sources.These data vary in volume, speed, sampling frequency, spectral range, and spatiotemporal scale.
  • ML methods extract patterns from large Earth-science datasets and support tasks including land-cover classification, exchange modeling, anomaly detection, extreme-event detection, and causal discovery.
  • The paper reviews ML information-fusion algorithms, EO applications, data sources, datasets, models, case studies, and future directions with a practical orientation.Its structure covers literature, data sources and models, illustrative case studies, and discussion of the field’s near future.

2. Machine Learning information fusion in Earth observation: a comprehensive literature review

Because EO spans many problems and applications, the paper narrows its review to ML information fusion and organizes prior work by applications and problems.

  • The review focuses on ML information fusion for EO rather than attempting to summarize the entire Earth-observation research area.It uses application- and problem-based organization to make the literature tractable.

2.1. Previous reviews and overviews in EO

Earlier reviews cover broad information fusion, computational intelligence, remote-sensing modalities, big-data methods, environmental data science, and deep learning, but generally address narrower aspects.

  • The first EO information-fusion review presented the field’s main concepts using a general, coarse-grained treatment.
  • A subsequent Earth-science review broadly described computational-intelligence methods while emphasizing data fusion.
  • Several reviews specialize in remote-sensing fusion, including spectroscopy and laser systems, spatiotemporal fusion, multimodal classification, and data-fusion algorithms.
  • Other reviews address image fusion, super-resolution, hyperspectral and multispectral fusion, and multisource or multitemporal data fusion.
  • Recent reviews also examine satellite big-data techniques, environmental data science, and deep learning in Earth-science or remote-sensing applications.

2.2. A taxonomy of ML information fusion approaches

EO information-fusion methods combine disparate inputs through preprocessing, fusion mechanisms, and decision support, with fusion commonly performed at sub-feature, feature, or decision level.

  • EO information fusion combines multisource data to support decisions, but its problem-specific methods make a complete taxonomy difficult.The main building blocks are disparate inputs, preprocessing, fusion mechanisms, and outputs.
  • Sub-feature level: At the sub-feature level, transformations harmonize data with different spatiotemporal scales into common multidimensional grids.
  • Feature level: At the feature level, datasets or learned feature representations are stacked and supplied to an ML algorithm.Feature combinations and source transformations may themselves be learned from data.
  • Decision level: At the decision level, separate modality-specific processing paths produce outputs that are fused to improve accuracy.Methods can operate on combinations of output activation functions.
  • Real literature examples provide the clearest way to distinguish these fusion levels.

2.3. Literature review

The literature applies ML information fusion across diverse Earth observation tasks, including classification, regression, detection, prediction, gap filling, and multisensor blending. These studies combine heterogeneous satellite, airborne, ground-based, model, and ancillary data at different fusion levels.

  • ML fusion methods: EO fusion studies mainly use classifiers, anomaly or change detectors, and regression methods for land-cover labeling, target screening, and variable estimation.These methods operate on data from satellites, aircraft, drones, and other sensors.
  • Environmental estimation: Surface-temperature, drought, and water-quality studies fuse ground, satellite, and reanalysis data through feature-level ML methods.Examples include Random Forest downscaling of soil-moisture drought indices to 1 Km and Genetic Programming for daily lake-water-quality estimates.
  • Cloud detection and classification: Cloud studies use CNN architectures to fuse visual, multimodal, and multiscale features for cloud classification and cloud-shadow masking.One method was validated on optical satellite images spanning spatial resolutions from 0.5 to 50 m.
  • Image and land-use classification: Land-use and hyperspectral image studies combine multispectral, radar, and hyperspectral information using feature-, decision-, and sub-feature-level fusion.Reported approaches include SVM, fuzzy C-means, deep CNNs, and cellular automata–Markov models.
  • Renewable energy: Renewable-energy prediction explores heterogeneous inputs, including in-situ measurements and the GFS model, with temporal Gaussian Processes outperforming alternative ML algorithms for solar radiation prediction.The application is linked to cloud prediction and the intermittency of solar, wind, and ocean energy resources.
  • Gap filling and blending: Optical remote-sensing fusion addresses noise, cloud contamination, missing pixels, and limited temporal, spectral, or spatial resolution through temporal, spatial, spatiotemporal, and blending approaches.Landsat and Sentinel 2 revisit cycles are cited as examples of temporal limitations, at 16 and 8 days respectively.

3. Data sources and models for Earth observation

Earth observation draws on heterogeneous observational, model-based, and ancillary data sources spanning satellite, ground, atmospheric, marine, and socio-economic domains. The reviewed resources include established climate datasets, measurement networks, harmonization platforms, and autonomous observing systems.

  • Data-source diversity: EO data sources include atmospheric and climate models, simulations, social media, socio-economic data, and observations from diverse sensors and platforms.The sources support information fusion for studying the Earth system and related applications.
  • Ground-based observations: Ground-based datasets provide meteorological and climatological measurements, including station series, gridded fields, global temperature anomalies, precipitation, and surface-temperature records.Examples include ECA&D, CRU, GPCC, GPCP, GHCN-M, GISS, NOAA ESRL, and DATA.GOV resources.
  • Land-atmosphere observations: FLUXNET compiles eddy-covariance measurements of carbon dioxide, water vapor, and energy exchanges across biomes and climates.Its observations support machine-learning and information-fusion approaches that upscale point estimates into spatially explicit gridded flux fields.
  • Data harmonization: The Earth System Data Lab curates more than 40 variables across land-surface, atmospheric-forcing, and socio-economic data streams and supports running algorithms online.It is presented as an initiative for land and atmosphere data harmonization.
  • Atmospheric and marine observations: Atmospheric soundings provide instantaneous vertical profiles at particular locations, while marine databases supply freely available oceanographic and meteorological observations.Marine sources include the U.S. National Data Buoy Center, European infrastructure, and the Australian Ocean Data Network.
  • Autonomous observations: The Argo program uses more than 3500 profiling floats to measure ocean temperature and salinity from the surface to 2000 m every 10 days.The program illustrates how autonomous observations address data scarcity.

3.3. Numerical weather models

Numerical weather models compute atmospheric evolution from physical equations on global grids and assimilate observations from radiosondes, satellites, and other systems. The review describes operational global and regional models, reanalyses, and comparisons with emerging ML forecasting approaches.

  • Numerical weather models: Numerical weather models solve Navier–Stokes, energy, mass, and momentum equations to calculate atmospheric variables across grid nodes.They represent fluid motion, thermodynamics, radiative transfer, and atmospheric chemistry.
  • Data assimilation: Global forecasting models assimilate current observations and require supercomputers to solve atmospheric equations across the global grid.Inputs include radiosonde, weather-satellite, and other observing-system data.
  • Forecast limitations: Current numerical weather models have forecast skill extending to only about six days because of the atmosphere’s nonlinear and chaotic nature.This is stated as a limitation of current global forecasting systems.
  • Operational models: Operational global models discussed include GFS, GEM, NAVGEM, IFS, UM, and ARPEGE.They are produced or maintained by meteorological agencies in the United States, Canada, Europe, the United Kingdom, and France.
  • Model comparison and ML: Studies compare numerical models and reanalyses for meteorological applications, while ML has also been discussed as an alternative for global weather forecasting.The passages describe comparisons but do not establish a general performance advantage for ML.
  • Reanalysis: Reanalysis combines past observations with modern numerical weather models through data assimilation to create long, spatially comprehensive records of the Earth system.The review distinguishes global projects such as NCEP/NCAR and CFSR from regional systems such as NARR and COSMO-REA6.

4. Case studies on ML information fusion in Earth observation

Four case studies provide empirical evidence for ML information fusion in practical Earth observation problems across gap filling, satellite blending, natural-hazard prediction, and EO–social-media fusion.

  • Case-study scope: The case studies examine ML fusion for soil-moisture gap filling, heterogeneous optical-satellite blending, natural-hazard prediction, and EO–social-media integration.They span multiple fusion levels, spatial and temporal resolutions, and structured and unstructured data.

4.1. Multitemporal and multisensor gap-filling in remote sensing

EO data gaps limit applications requiring continuous records, while conventional interpolation cannot reliably reconstruct sharp transitions, long gaps, or information from collocated sensors. The reviewed multi-output LMC-GP approach learns cross-sensor relationships to reconstruct missing soil-moisture observations and preserve coverage.

  • Standard interpolation and autoregressive methods fail on sharp transitions, long gaps, and information available from collocated sensors.These limitations complicate harmonizing multiple satellites into consistent climate records and fusing microwave with optical observations.
  • Multi-output LMC-GP learns relationships among sensors to fill spatiotemporal gaps in collocated satellite observations.The approach uses an across-domain kernel function based on the Linear Model of Corregionalization.
  • The case study integrates six years of SMOS, AMSR2, and ASCAT soil-moisture products, each containing different observational gaps.The products are part of the ESA Climate Change Initiative soil-moisture product.
  • Reconstructed time series closely follow original observations, capture wetting and drying events, and reproduce an October 2014 peak observed only by SMOS.Predictions also express higher uncertainty when sensor-specific training data are unavailable and can back-propagate AMSR2 estimates before launch.
  • Predictions are available at every timestamp with at least one satellite measurement, maximizing the datasets’ spatiotemporal coverage.Comparisons with in-situ data show Pearson’s R, mean error, and unbiased RMSE remain within reasonable bounds after reconstruction.

4.2. Modern data assimilation and hybrid modeling in geosciences

Image blending combines Landsat’s fine spatial resolution with MODIS’s frequent observations to address sensor limitations and produce gap-free reflectance estimates. HISTARFM implements this strategy as a scalable, bias-aware Bayesian assimilation method in Google Earth Engine, with validation showing satisfactory accuracy and useful uncertainty estimates.

  • Blending Landsat and MODIS combines 30 m spatial resolution with MODIS’s higher temporal resolution for gap-free surface-reflectance prediction.MODIS helps track rapid land-cover changes and increases the likelihood of cloud-free observations.
  • HISTARFM uses two simultaneous Kalman estimators to reduce noise and dynamically correct possible biases in predicted Landsat reflectances.The first combines Landsat climatology with blended Landsat–MODIS information; the second corrects biases from that estimate.
  • Figure 3 compares original and gap-filled Landsat images for cropland in Texas in May 2010.
  • HISTARFM’s Google Earth Engine implementation processes huge datasets faster than other approaches and was validated across 1050 conterminous United States sites.The validation assessed feasibility and produced satisfactory results.
  • Relative mean errors remained below 2% in all spectral bands, while relative mean absolute and root-mean-squared errors ranged from 10–20% depending on band.Predicted uncertainties also showed high agreement with validation errors, supporting their use for error propagation.

4.3. Natural hazard prediction and data fusion

Natural-hazard information fusion combines heterogeneous observations to improve hazard detection, prediction, and avoidance. A flood case study integrates Sentinel-2, weather-station, and land-survey information with random forests for susceptibility mapping.

  • 700,000 people were killed and 1.7 billion affected by disasters between 2005 and 2014, motivating accurate hazard mapping and prediction.
  • Fusing satellite, radar, laser, UAV, weather-station, crowdsourcing, social-media, and GIS data can enhance hazard-system robustness and performance.
  • Flood and drought modeling is complex because multiple causes operate across different spatial and temporal scales.
  • Flood prediction by integrating remote sensing and weather station data: The flood case study integrates weather-station, land-survey, and satellite data to improve flood susceptibility mapping.The study uses Sentinel-2 imagery to identify inundated regions and flooded or non-flooded points in Iran’s Gorganroud Basin.
  • Flood prediction by integrating remote sensing and weather station data: 368 flash-flood locations were sampled from inundated points, with 70% used for training, 30% for testing, and 10-fold cross-validation for random-forest modeling.

4.4. Fusion of Earth data and Social media

Earth-observation data from satellites and airborne platforms provide broad coverage but leave semantic ambiguities that ancillary and social-media data can help address. Social-media fusion uses metadata, text, and images, while noisy data and sparse labels remain important limitations.

  • Satellite and airborne observations offer a bird’s-eye perspective but cannot reliably distinguish some semantic concepts, such as land use from land cover.Additional covariate data can help resolve these ambiguities.
  • Location-based social media is fused with Earth-system data through metadata analysis, text mining, and image analysis.
  • Spatial Statistics of Metadata: Spatial summaries of message activity can provide features related to Earth-system parameters and human activity patterns.
  • Spatial Statistics of Metadata: Social-media metadata significantly improved the convergence speed and final quality of a deep-learning model for urban land-use prediction.
  • Text Mining: Text mining converts language into numeric semantic representations, including TF-IDF, topic models, LDA, LSTMs, and transformer-based embeddings.BERT and GPT2 are identified as promising directions for Earth-system applications.
  • Text Mining: Positive-sentiment tweets around New York were distributed unevenly, appearing skewed toward commercial centers.
  • Image Mining and Multimedia Analysis: Social-media images can reveal social, spatial, and economic information, but their content may not represent the physical context of the post.
  • Summary: Learning from noisy social-media streams with sparse labels remains in its infancy and requires advances in unsupervised ML, natural-language processing, and spatial data science.

5. Conclusions, discussion and outlook

The review finds ML information fusion increasingly central to Earth observation because data sources are growing in amount, diversity, resolution, and heterogeneity. It highlights scalability, causal understanding, and physics-aware hybrid modeling as key challenges and future directions.

  • ML information fusion has obtained excellent results across many Earth observation problems and applications.
  • Scalability remains an important challenge as Earth observation algorithms confront the big data era.
  • Increasingly diverse, heterogeneous, and non-stationary datasets make effective and efficient integration a central Earth observation challenge.Sources differ in sensed properties, spatial, spectral, and temporal resolutions, and platform configurations.
  • Hybrid physics-aware ML is proposed to improve modeling consistency and interpretability by combining data-driven methods with domain knowledge and physical models.Probabilistic programming can represent uncertainties, priors, and constraints, while differentiable programming supports optimization of complex nonlinear models.
  • Causal inference is identified as a route beyond correlation toward understanding causal structures in heterogeneous, nonlinear, and non-stationary multivariate data.The paper distinguishes observational causal discovery from conventional correlation-based analysis when interventions are unavailable.
Loading 2012.05795v1…