Source-linked AI summary
Super-resolution of Sentinel-2 images: Learning a globally applicable deep neural network
Charis Lanaras, José Bioucas-Dias, Silvano Galliani, Emmanuel Baltsavias, Konrad Schindler
TL;DR
The paper asks whether Sentinel-2’s 20 m and 60 m bands can be computationally super-resolved to a complete 10 m data cube. It trains globally sampled CNNs on synthetically downsampled Sentinel-2 data and reports substantially improved accuracy, spectral preservation, and practical full-resolution results.
Problem
The paper addresses the need to super-resolve Sentinel-2’s lower-resolution bands to 10 m for more detailed and accurate information extraction.
Method
DSen2 uses two end-to-end deep CNNs that jointly map all input bands to 10 m outputs, trained with reduced-resolution data synthesized from real Sentinel-2 images.
Results
50% lower RMSE than the best competing methods and almost 6 dB higher SRE were reported, with qualitative performance confirmed across land-cover types, biomes, and climate zones.
Takeaways & Limitations
A single globally trained network generalizes to unseen locations and can super-resolve Sentinel-2 data without retraining in the reported tests.
Takeaways & Limitations
The released weights are trained specifically on real Sentinel-2 images and are not suitable for other sensors or other Sentinel-2 processing levels.
Abstract
from arXiv · showhide
The Sentinel-2 satellite mission delivers multi-spectral imagery with 13 spectral bands, acquired at three different spatial resolutions. The aim of this research is to super-resolve the lower-resolution (20 m and 60 m Ground Sampling Distance - GSD) bands to 10 m GSD, so as to obtain a complete data cube at the maximal sensor resolution. We employ a state-of-the-art convolutional neural network (CNN) to perform end-to-end upsampling, which is trained with data at lower resolution, i.e., from 40->20 m, respectively 360->60 m GSD. In this way, one has access to a virtually infinite amount of training data, by downsampling real Sentinel-2 images. We use data sampled globally over a wide range of geographical locations, to obtain a network that generalises across different climate zones and land-cover types, and can super-resolve arbitrary Sentinel-2 images without the need of retraining. In quantitative evaluations (at lower scale, where ground truth is available), our network, which we call DSen2, outperforms the best competing approach by almost 50% in RMSE, while better preserving the spectral characteristics. It also delivers visually convincing results at the full 10 m GSD. The code is available at https://github.com/lanha/DSen2
1. Introduction
Sentinel-2 provides globally distributed, multi-spectral imagery but its bands have different spatial resolutions. DSen2 uses CNN-based multi-spectral super-resolution to produce 10 m bands, with strong accuracy and global applicability in the reported tests.
- Super-resolving lower-resolution bands can support more detailed and accurate information extraction.
- Sentinel-2 acquires 13 spectral bands at 10 m, 20 m, and 60 m resolutions.
- DSen2 jointly super-resolves 20 m and 60 m bands to 10 m using convolutional neural networks informed by correlations across all bands.
- Training uses globally sampled Sentinel-2 data spanning climate zones, biomes, land-cover types, and geographical areas.
- A single model performed better than prior state-of-the-art methods on globally sampled independent test sites and was reported as applicable worldwide.
- Almost 50% lower RMSE than the best competing methods was achieved, alongside higher signal-to-reconstruction-error ratio and improved spectral angle mapping.The reported improvements were >5 dB in signal-to-reconstruction-error ratio and >30% in spectral angle mapping.
2. Related work
Prior multi-resolution super-resolution methods include per-band pan-sharpening, explicit imaging-model inversion, and supervised machine learning. These approaches differ in how they use spectral information, physical assumptions, regularization, and training data.
- Pan-sharpening methods independently increase target-band resolution by blending information from spectrally overlapping high-resolution bands.
- Sentinel-2 lacks a broad-spectrum panchromatic band, so pan-sharpening studies often synthesize or heuristically select a substitute from 10 m bands.
- Model-based methods formulate super-resolution as an ill-posed inverse imaging problem with explicit blur, downsampling, noise, and regularization assumptions.
- Lanaras et al. use per-band point-spread functions, band correlation, dimensionality reduction, and learned spatially varying gradient penalties.
- SMUSH uses patch-based self-similarity regularization and hierarchically sharpens 20 m bands before the coarser 60 m bands.
- Supervised machine-learning methods learn the lower-resolution-to-higher-resolution relation from examples rather than specifying it explicitly.
- Deep learning can capture complex relations but requires massive training data, substantial computation, and high-dimensional optimization.
3. Input data
The study uses globally distributed Sentinel-2 observations from the 2A and 2B satellites, whose multi-resolution bands and accurate co-registration support super-resolution. Tiles were selected to cover varied geographical, climatic, biome, and land-cover conditions.
- Sentinel-2A and 2B are identical satellites in the same orbit with a 180-degree phase difference, reducing equatorial revisit time from 10 to 5 days.
- Sentinel-2 systematically covers all land masses except Antarctica.
- The 10 m and 20 m bands support land-cover mapping, agriculture, forestry, biophysical-variable retrieval, water monitoring, and disaster mapping.
- The three 60 m bands are mainly intended for water-vapour, aerosol-correction, and cirrus-cloud estimation, although they are captured at 20 m and downsampled to 60 m.
- Low Sentinel-2 band-to-band co-registration errors are sufficiently small to ignore without correcting band offsets.
- The study uses Level-1C top-of-atmosphere reflectance products that are radiometrically and geometrically corrected, geocoded, and orthorectified.
- Tiles acquired from both satellites were randomly selected for broad global distribution and variety in climate zone, land-cover, and biome type.
4. Method
DSen2 learns Sentinel-2 super-resolution from globally sampled, synthetically downsampled data, using separate networks for the 20 m and 60 m bands. Its architecture combines high-resolution guidance, residual blocks, and additive corrections to bilinear upsampling.
- Learning strategy: The method uses CNNs to learn a mapping from multi-resolution Sentinel-2 patches to high-resolution patches for the bands requiring upsampling.Two separate networks are trained for the 20 m and 60 m bands because the 60 m bands do not contribute information to 20→10 m upsampling.
- Simulation process: Training data are created by Gaussian blurring and window averaging, then used for 40→20 m and 120→60 m proxy super-resolution tasks.The 2× dataset uses 20 m and 40 m data, while the 6× dataset uses 60 m, 120 m, and 360 m data; evaluation is therefore performed at reduced resolution.
- Multi-resolution inputs: The networks group Sentinel-2 bands into 10 m set A, 20 m set B, and 60 m set C, then use separate inputs for 2× and 6× upsampling.The 2× network uses A and B to predict B, whereas the 6× network uses A, B, and C to predict C at 10 m GSD.
- Residual design: The architecture uses EDSR-inspired residual blocks with skip connections and residual scaling, including a fixed λ = 0.1 layer.Residual scaling is reported as crucial for training very deep networks, while batch normalization did not improve accuracy or training time.
- Basic architecture: Low-resolution bands are bilinearly upsampled before being combined with 10 m bands, allowing the network to operate at target resolution from the beginning.The inputs are concatenated and processed through convolutional layers and ReLU activations before the residual-block stack.
- Output formulation: A long additive skip connection makes the network predict a correction to the bilinearly upsampled image rather than the final image directly.The authors report that this strategy helps preserve input radiometry.
- Training details: Training uses 32×32-pixel patches for T2× and 96×96-pixel patches for S6×, providing local context rather than kilometer-scale context.The corresponding receptive fields cover several hundred metres and are intended to capture local texture and small structures, but not large-scale topography.
5. Experimental results
Experiments evaluate DSen2 and VDSen2 against established baselines at reduced scales where Sentinel-2 ground truth is available. The networks achieve substantially better reconstruction and spectral fidelity, with advantages especially evident for 60 m and SWIR bands.
- Evaluation setup: Quantitative evaluation is performed only at reduced scales: 40→20 m for T2× and 360→60 m for S6×, where ground truth is available.The datasets are synthetically degraded from Sentinel-2 imagery for these evaluations.
- T2× — 20 m bands: 48% lower RMSE, more than 5 dB higher SRE, and 24% lower SAM distinguish DSen2 from the previous state of the art for 2× upsampling.These aggregate results average the six 20 m bands over 15 test images at the 40→20 m evaluation scale.
- T2× — 20 m bands: An auxiliary network trained on 80→40 m still outperforms all baselines when evaluated at 20 m, supporting sufficient scale invariance of the learned mapping.The authors use this result to justify 40→20 m synthetic training as a proxy for 20→10 m upsampling.
- T2× — 20 m bands: The learned network’s advantage is consistent across bands and particularly pronounced for the challenging extrapolation to B11 and B12.B11 and B12 are SWIR bands outside the spectral range of the high-resolution inputs; baselines show a marked accuracy drop there, while DSen2 remains comparatively stable.
- T2× — 20 m bands: Residual images show that baselines commonly produce edge errors resembling high-pass filtering, whereas DSen2 has visibly lower residuals across all spectral bands.The baseline residuals indicate blurred edges or exaggerated contrast in the displayed test scene.
B1 B9 Average SAM RMSE SRE UIQ RMSE SRE UIQ RMSE SRE UIQ
At reduced scale, DSen2 and VDSen2 outperform competing super-resolution and pan-sharpening methods, especially for challenging 6× upsampling and SWIR bands. On original Sentinel-2 data, the networks produce sharper images with visually convincing detail, although quantitative ground truth is unavailable.
- 6× super-resolution: VDSen2 reduces RMSE by approximately 3× versus the previous state of the art and achieves about 20% lower RMSE than DSen2 at the larger upsampling factor.VDSen2 also reaches 2.3 dB higher SRE than DSen2.
- 6× super-resolution: 65% lower error than SupReME makes VDSen2 the strongest method in the 6× comparison.SupReME has better overall baseline numbers than Superres because of its performance on B9.
- 6× super-resolution: B9 is the most challenging band: it is the only band where the proposed method falls below 33 dB SRE and 0.9 UIQ.All methods perform better relative to average radiance on B1 than on B9.
- Qualitative evaluation: DSen2 and VDSen2 produce much lower and less structured residuals than competing methods in qualitative comparisons with ground truth.The comparison uses absolute residuals at reduced scale.
- Original-scale evaluation: On original-scale images, super-resolved bands are sharper and reveal more detail than low-resolution inputs, with perceptual quality visually matching native 10 m RGB bands.The reported 2× and 6× examples cover varied locations and land-cover settings, but no quantitative ground truth is available at the original scale.
- Pan-sharpening comparison: Pan-sharpening benefits some visible and near-infrared bands but performs poorly for SWIR bands B11 and B12, sometimes falling below bicubic interpolation.Its performance depends sensitively on the selected panchromatic band and can fail on individual images.
- Pan-sharpening comparison: The authors conclude that pan-sharpening cannot substitute for qualified super-resolution and is unsuitable for Sentinel-2.This conclusion follows the observed difficulties with spectrally distant SWIR bands.
6. Discussion
The discussion weighs network depth, computational cost, runtime, and deployment options. VDSen2 offers stronger performance when hardware permits, while pretrained models and public implementations reduce the cost of applying or adapting the method.
- Network depth: There is no single right network depth because the choice depends on application variability, computational resources, and update frequency.The authors report no disadvantage to using VDSen2 when adequate hardware is available.
- Computational cost: Training is computationally demanding and can take several days, but a trained network super-resolves a new image with a fast single forward pass.Adapting an existing network with additional data is less costly than training from scratch.
- Runtime: Runtime comparisons for a complete Sentinel-2 tile are indicative because baselines use CPU code while CNNs are generally run on GPUs.The comparison therefore does not claim a universally hardware-independent speed ranking.
- Deployment and adaptation: The released Keras source, training procedure, and pretrained weights support out-of-the-box Sentinel-2 super-resolution and region-specific refinement.The authors expect better results in studies focused on particular geographic regions, biomes, or land-cover types after targeted training.
- Scope: The released weights are trained only on real Sentinel-2 images and are not suitable for other sensors or other processing conditions.Their performance is tied substantially to the image statistics of Sentinel-2 inputs.
7. Conclusions
The paper presents DSen2 and VDSen2 as end-to-end networks that learn cross-band mappings for producing a uniform 10 m Sentinel-2 cube. Global experiments show strong accuracy and practical full-resolution performance, while the generic architecture and open-source release support adaptation beyond Sentinel-2.
- 7. Conclusions: DSen2 and VDSen2 jointly learn mappings from all input bands to 2× and 6× super-resolved outputs, producing a uniform 10 m Sentinel-2 data cube.Training assumes that cross-band high-frequency transfer is invariant across scales.
- 7. Conclusions: A globally sampled dataset yields a tool that generalises to unseen locations across different parts of the world.The conclusion is based on the authors’ experiments with varied global data.
- 7. Conclusions: The network reduces RMSE by 50% versus the best competing methods and increases SRE by almost 6 dB.Qualitative tests across land-cover types, biomes, and climate zones support performance on full-resolution Sentinel-2 images.
- 7. Conclusions: A complete 120 MPix Sentinel-2 tile can be processed in a few minutes, supporting practical large-scale applications.The reported computation time concerns full-resolution super-resolution.
- 7. Conclusions: Because the networks are learned end-to-end from image data, the authors consider them generic enough to retrain for other multi-resolution multispectral sensors.Software and models are released as open-source tools.