Source-linked AI summary
Prolongation of SMAP to Spatio-temporally Seamless Coverage of Continental US Using a Deep Learning Neural Network
Kuai Fang, Chaopeng Shen, Daniel Kifer, Xiao Yang
TL;DR
SMAP soil-moisture observations are valuable but limited by their short record and irregular revisits, while land-surface models provide longer coverage with substantial biases. This paper trains an LSTM using atmospheric, simulated-moisture, and physiographic inputs to predict SMAP L3 data. The resulting system reduces model bias, improves climatology, and achieves high-fidelity performance across temporal, spatial, and regional tests.
Problem
SMAP’s short time span and irregular revisit schedule limit continuous soil-moisture observation, while long-duration land-surface-model simulations contain substantial spatially varying bias.
Method
The study trains a CONUS-scale LSTM to predict SMAP L3 soil moisture from atmospheric forcing, land-surface-model moisture, static physiographic attributes, and product flags.
Results
LSTM reduces model bias and improves soil-moisture climatology, with RMSE smaller than SMAP’s design measurement accuracy in many parts of CONUS.
Takeaways & Limitations
LSTM shows potential for long-range soil-moisture hindcasting or forecasting, weather modeling, and data assimilation, including across regions with differing climates and physiography.
Takeaways & Limitations
The study is an initial LSTM application in hydrology; drought hindcast performance, comparisons with regionally trained simpler models, and low RMSEs in forested regions require further investigation.
Abstract
from arXiv · showhide
The Soil Moisture Active Passive (SMAP) mission has delivered valuable sensing of surface soil moisture since 2015. However, it has a short time span and irregular revisit schedule. Utilizing a state-of-the-art time-series deep learning neural network, Long Short-Term Memory (LSTM), we created a system that predicts SMAP level-3 soil moisture data with atmospheric forcing, model-simulated moisture, and static physiographic attributes as inputs. The system removes most of the bias with model simulations and improves predicted moisture climatology, achieving small test root-mean-squared error (<0.035) and high correlation coefficient >0.87 for over 75\% of Continental United States, including the forested Southeast. As the first application of LSTM in hydrology, we show the proposed network avoids overfitting and is robust for both temporal and spatial extrapolation tests. LSTM generalizes well across regions with distinct climates and physiography. With high fidelity to SMAP, LSTM shows great potential for hindcasting, data assimilation, and weather forecasting.
1 Introduction
SMAP provides valuable soil-moisture observations, but its short record and irregular revisit schedule limit continuous monitoring. The study therefore investigates LSTM deep learning to extend SMAP information and correct land-surface-model errors while testing robustness against overfitting.
- Motivation: SMAP measures top-5-cm global soil moisture but has observations only since April 2015 and revisits every 2–3 days.These constraints hinder observing soil-moisture responses immediately after storms or snowmelt.
- Motivation: Land-surface models provide seamless, long-duration soil-moisture simulations but exhibit substantial biases that vary by region, model, season, and depth.For example, moisture is generally overestimated in arid western CONUS and underestimated in the wetter eastern United States.
- Approach: Deep learning automatically extracts abstract features through hidden layers, offering an approach for correcting systematic model errors.This contrasts with earlier machine-learning methods that depend on human-engineered features.
- Approach: Large deep-network parameter spaces raise overfitting concerns, although regularization methods such as Dropout can reduce them.Because LSTM had not previously been applied to hydrology, its robustness required examination against conventional statistical methods.
- Objectives: The study hypothesizes that LSTM can learn soil-moisture dynamics and land-surface-model errors from two years of SMAP data to create a seamless CONUS product.Its objectives include correcting process-based model errors and comparing spatial and temporal extrapolation against conventional methods.
2 Methods and Datasets
The study trains a CONUS-scale LSTM to predict daily SMAP L3 soil moisture from atmospheric, model-simulated, physiographic, and quality-related inputs. It evaluates temporal, spatial, and regional extrapolation against conventional alternatives, including a long-term hindcast test.
- Data and inputs: The LSTM predicts daily SMAP L3 soil moisture using atmospheric forcing, land-surface-model moisture, static physiography, and SMAP product flags.Inputs are aggregated and interpolated to the 36-km SMAP grid, with each pixel treated as an instance.
- LSTM setup: LSTM updates hidden and cell states from current inputs and previous network states, allowing sequential information to influence predictions.Its inputs include forcings and static attributes, while training uses back-propagation through time and Dropout regularization.
- LSTM setup: At each time step, the network outputs one scalar prediction compared with SMAP observations, and training minimizes time-series mean-squared error.Only time steps with SMAP observations contribute to the masked loss, which is averaged over randomly sampled pixel mini-batches.
- Evaluation: The evaluation tests temporal generalization, regular spatial generalization, and regional holdout across distinct CONUS climates and physiographic conditions.The regional test trains on four of 18 HUC2 regions and evaluates performance on held-out regions.
- Baselines: LSTM is compared with regularized linear regression, autoregressive models, and a single-layer feedforward neural network using the same inputs.The feedforward network can model nonlinearities but cannot retain time dependence.
- Hindcast test: A proof-of-concept hindcast trains LSTM and ARp on two years of Noah-simulated soil moisture to reconstruct conditions 10 years earlier.This test targets long-term hindcasting without continuously available observations.
3 Results
Across temporal, spatial, and regional tests, LSTM corrected model bias and improved climatology while outperforming conventional alternatives in RMSE, bias, and correlation.
- The LSTM correction reduced Noah’s spatially varying bias by an order of magnitude in the temporal generalization test.
- LSTM improved correlation, with R mostly above 0.8 and 50% of pixels above 0.9, while most CONUS pixels had RMSE below 0.035.
- LSTM captured SMAP temporal dynamics across representative pixels, including frequent rain events and most peaks, whereas ARp performed poorly for one low-percentile cell.
- Across CONUS, LSTM achieved the lowest test RMSE and bias and highest R among the compared methods; 75% of R values exceeded 0.80 and 50% exceeded 0.87.
- LSTM maintained robust 10-year hindcast performance and the smallest RMSE and bias in regular spatial generalization tests.
- Across HUC2 regional holdouts, LSTM greatly reduced Noah bias and improved climatology relative to both the one-layer NN and Noah.
4 Discussion
The discussion interprets LSTM’s performance as useful for long-range hindcasting and bias correction, while emphasizing unresolved retrieval and model-representation questions.
- LSTM RMSE was smaller than SMAP’s design measurement accuracy in many CONUS regions, including forested Southeast areas with RMSE of 0.02-0.035.
- Because longer-term hindcasts lack continuous observations, pattern-based LSTM methods appear more suitable than recurrent alternatives that can drift without updates.
- LSTM appears to compensate for inadequate land-surface-model representations by integrating raw data uniformly across CONUS.
- LSTM’s generalization capability is attributed to learning from heterogeneous data and constructing internal models in attribute space without requiring every climate–physiography combination during training.
5 Conclusion
The study concludes that a CONUS-scale LSTM can create a seamless, SMAP-consistent soil-moisture product and generalize across space and time.
- The trained LSTM corrected spatially heterogeneous Noah bias and climatological errors, producing a seamless CONUS-scale moisture product with high fidelity to SMAP.
- LSTM generalized better in spatial and temporal tests than linear regression, autoregressive models, and a one-layer neural network.
- The authors identify long-range hindcasting, weather modeling, and data assimilation as potential uses for LSTM’s SMAP-consistent predictions.
6 Limitations and Future Work
The authors characterize the study as an initial investigation and identify several priorities for validating and extending the method.
- The study is not a thorough investigation because it is the first paper applying LSTM in hydrology.
- Future work should examine drought hindcast performance using in-situ data and compare LSTM with regionally trained simpler models.
- The accuracy of SMAP data and the implications of low LSTM RMSEs in forested regions require further investigation.
Acronyms
The paper uses SMAP, DL, LSTM, and LR as abbreviations for Soil Moisture Active Passive, deep learning, Long Short-Term Memory, and linear regression.
- SMAP means Soil Moisture Active Passive.
- DL means deep learning.
- LSTM means Long Short-Term Memory.
- LR means linear regression.
7. Text S2. Proof-of-concept test for the potential of LSTM for long-term hindcast.
The supplementary material identifies Figure S6 as a proof-of-concept comparison of LSTM and AR for synthetic long-term hindcasting.
- Figure S6 evaluates the performance of LSTM and AR in a synthetic long-term hindcast experiment.
- The experiment uses Noah-simulated soil moisture as the hindcast target.
Text S1. Technical Details about Conventional Methods
The paper compares LSTM with regularized regression, autoregressive models, and a feedforward neural network, while supplementary figures examine data quality, interpolation, and alternative model inputs.
- Conventional methods: LSTM is compared with lasso, autoregressive models, and a single-layer feedforward neural network using the same inputs.
- Conventional methods: LR uses regularization to penalize large coefficients, with atmospheric and Noah-simulated hydrologic variables as exogenous inputs.
- Conventional methods: The autoregressive model is trained point-by-point and recursively forecasts without observations during long-term hindcasts.
- Conventional methods: The feedforward neural network uses a nonlinear transformation and L2 regularization, with hidden sizes of 100 and 30 for NN and NNp.
- Supplementary comparisons: Supplementary figures assess LSTM-versus-RNN design, SMAP data quality, Noah performance, soil-moisture interpolation, and Noah-versus-MOS inputs.
- Caveat: The authors state that LSTM robustness and its hydrologic application require thorough evaluation.
Text S2. Proof-of-concept test for the potential of LSTM for long-term hindcast
A synthetic hindcast experiment tests whether LSTM can extend soil-moisture dynamics over multiple years using two years of Noah-based training data. LSTM maintains lower and more stable errors than the point-by-point autoregressive model.
- Experiment: LSTM and ARp are trained on two years of Noah-simulated soil moisture and used to hindcast ten years backward.
- Experiment: The synthetic target combines Noah’s top-10-cm soil moisture with a Gaussian relative error term ϵ ∼N(0, 0.07).
- Results: 0.043 is the LSTM median error in 2005-2006 for the white-noise case, compared with 0.049 for ARp.
- Results: LSTM hindcast error does not increase with hindcast length, while its 85th-percentile error is below ARp’s 25th-percentile error in 2005-2006.
- Interpretation: LSTM can use static attributes to distinguish locations with different soil textures and land covers, unlike ARp.
- Inputs: The predictor inventory includes land-surface, atmospheric, vegetation, soil, snow, geographic, and surface-water variables.