Source-linked AI summary
DiffusionSat: A Generative Foundation Model for Satellite Imagery
Samar Khanna, Patrick Liu, Linqi Zhou, Chenlin Meng, Robin Rombach, Marshall Burke, David Lobell, Stefano Ermon
TL;DR
Existing generative models are not tailored to satellite imagery's spectral, temporal, and metadata requirements, despite satellite data's importance for environmental and agricultural applications. DiffusionSat addresses this gap with a latent-diffusion foundation model conditioned on metadata and 3D control signals, achieving strong results across generation and inverse-problem tasks. The authors report that it outperforms prior methods, while noting occasional hallucinated details and the need for broader data and more efficient sampling.
Problem
Satellite imagery differs from natural images and existing models do not support its metadata-conditioned inverse problems, including super-resolution, temporal in-painting, and cloud removal.
Method
DiffusionSat uses latent diffusion with satellite metadata conditioning and a 3D control-signal module for super-resolution, temporal generation, prediction, and in-painting.
Results
DiffusionSat outperforms prior methods and achieves state-of-the-art LPIPS with near-optimal SSIM and PSNR across conditional satellite-generation tasks.
Takeaways & Limitations
The model provides a generative foundation for high-resolution satellite imagery and multiple conditional remote-sensing generation tasks.
Takeaways & Limitations
Generated inverse-problem outputs can sometimes hallucinate details, motivating reduced sample variance and broader datasets.
Abstract
from arXiv · showhide
Diffusion models have achieved state-of-the-art results on many modalities including images, speech, and video. However, existing models are not tailored to support remote sensing data, which is widely used in important applications including environmental monitoring and crop-yield prediction. Satellite images are significantly different from natural images -- they can be multi-spectral, irregularly sampled across time -- and existing diffusion models trained on images from the Web do not support them. Furthermore, remote sensing data is inherently spatio-temporal, requiring conditional generation tasks not supported by traditional methods based on captions or images. In this paper, we present DiffusionSat, to date the largest generative foundation model trained on a collection of publicly available large, high-resolution remote sensing datasets. As text-based captions are sparsely available for satellite images, we incorporate the associated metadata such as geolocation as conditioning information. Our method produces realistic samples and can be used to solve multiple generative tasks including temporal generation, superresolution given multi-spectral inputs and in-painting. Our method outperforms previous state-of-the-art methods for satellite image generation and is the first large-scale generative foundation model for satellite imagery. The project website can be found here: https://samar-khanna.github.io/DiffusionSat/
1 INTRODUCTION
Satellite imagery supports important analytical and inverse-problem applications but differs from natural imagery in ways existing web-trained and discriminative foundation models do not address. DiffusionSat fills this gap with metadata-conditioned generative modeling and task-specific conditioning for satellite imagery.
- Satellite-image analysis supports disaster response, environmental monitoring, poverty prediction, crop-yield estimation, and urban planning.
- Satellite imagery differs from natural images in perspective, resolution, spectral bands, and temporal regularity, limiting existing foundation models for inverse problems.
- DiffusionSat trains a generative foundation model on publicly available satellite imagery using metadata such as latitude, longitude, timestamp, and GSD.
- ControlNets extend DiffusionSat to super-resolution, in-painting, and temporal generation tasks.
- The contributions include numerical-metadata and text conditioning, a 3D conditioning extension, and a global pre-training dataset compiled from public satellite datasets.
2 BACKGROUND
Diffusion models learn to denoise progressively corrupted data, while latent diffusion performs this process in a compressed representation to reduce computational and memory costs. Side information can condition denoising through inputs such as class embeddings, text, or images.
- Diffusion models learn a data distribution by adding Gaussian noise to samples and training a denoiser to recover the corrupted target.
- The denoising target may be the input noise, the original image, or the velocity, and the model can use side information as conditioning.
- Latent diffusion encodes images into a lower-dimensional representation, applies diffusion there, and decodes the denoised latent back to an image.
3 METHOD
DiffusionSat combines latent-diffusion image generation with numerical metadata and 3D control-signal conditioning for satellite imagery. Its design supports single-image generation, multispectral super-resolution, temporal prediction, and in-painting while handling irregular satellite-image sequences.
- Single Image Generation: DiffusionSat pre-trains a single-image model for satellite images conditioned on text captions and/or metadata.
- Single Image Generation: Because satellite captions are sparse, the model learns p(x|τ, k) using cheaply available metadata alongside text, allowing generation when captions are poor or missing.
- Single Image Generation: Each numerical metadata value is normalized, sinusoidally encoded, mapped through a metadata-specific MLP, and summed with the embedded diffusion timestep.
- Single Image Generation: The single-image pipeline encodes images into SD latents, adds Gaussian noise, predicts it using text and metadata conditioning, and decodes the denoised latents.
- Control Signal Conditional Generation: A 3D control-signal module conditions generation on image sequences, associated metadata, a caption, and target metadata for inverse problems.
- Control Signal Conditional Generation: Satellite sequences require handling irregular time gaps and global time, so the framework predicts or interpolates images from existing observations rather than focusing on long generated sequences.
- Control Signal Conditional Generation: Per-image metadata determines temporal position, making the control sequence invariant to image ordering and supporting past, future, and interpolated predictions.
4 EXPERIMENTS
Experiments evaluate DiffusionSat on single-image generation and several conditional satellite-image tasks, including superresolution, temporal prediction, and in-painting. Across these tasks, the model shows improved image quality and performance over relevant baselines.
- Experimental scope: Experiments cover single-image generation, multispectral superresolution, temporal prediction, and in-painting.The conditional tasks use Texas-housing, fMoW-Sentinel/fMoW-RGB, fMoW-temporal, and xBD datasets.
- Single-image generation: Metadata improves single-image generation quality and control beyond using metadata only inside text captions.The comparison includes pretrained SD 2.1, finetuned SD without metadata, and finetuned SD with metadata in captions.
- Conditional generation: DiffusionSat uses single-image pretraining as a prior for superresolution, temporal generation, and in-painting.The conditional-generation approach uses 3D conditioning for these inverse problems.
- Quantitative evaluation: DiffusionSat achieves state-of-the-art LPIPS and near-optimal SSIM and PSNR on conditional generation tasks.Conditional-generation metrics are reported on 10,000 images and include SSIM, PSNR, and LPIPS with VGG features.
- Superresolution: In fMoW-Sentinel superresolution, DiffusionSat reconstructs high-resolution RGB images from low-resolution multispectral inputs and beats other superresolution models.The input contains all 13 Sentinel-2 multispectral bands, while most corresponding fMoW-RGB images have less than 1m GSD.
- Ablation: DiffusionSat pretraining and 3D ControlNet each improve sample quality across all metrics over direct SD finetuning and 2D ControlNet.The 2D comparison stacks images in the channel dimension, whereas the 3D approach is designed for conditioning image sequences.
- Temporal generation: DiffusionSat outperforms STSR, MCVD, and regular SD with 3D ControlNet on fMoW temporal prediction.The model better captures target-date seasonal changes and the development of roads and buildings.
- In-painting: On xBD, DiffusionSat reconstructs damaged roads and houses and can add disaster damage for forecasting or evacuation preparation.The qualitative in-painting experiments cover floods, wind, fires, earthquakes, and other natural disasters.
5 RELATED WORK
Related work places DiffusionSat within diffusion-based generative modeling, controllable diffusion, and remote-sensing superresolution. These areas provide the foundation for its use of pretrained diffusion priors and control signals.
- Diffusion models: Diffusion models have become prominent across generative applications including speech, 3D geometry, and graphics.Large-scale latent-space variants also support high-resolution image generation.
- Controllable diffusion: ControlNet, T2IAdapter, and InstructPix2Pix add trainable parameters to pretrained diffusion networks to incorporate control signals.These methods are presented as successful approaches for controllable generation after diffusion-model pretraining.
- Positioning: DiffusionSat combines pretrained diffusion modeling and controllable-generation ideas for satellite-image inverse problems.Its experiments apply these ideas to superresolution, temporal generation, and in-painting.
- Remote-sensing generation: Remote-sensing superresolution includes GAN-based methods such as SR-GAN alongside other natural- and satellite-image approaches.The cited literature spans established natural-image superresolution and remote-sensing applications.
6 CONCLUSION
DiffusionSat is presented as a generative foundation model for remote sensing, combining single-image generation with 3D-conditioned inverse-problem capabilities. The authors identify broader datasets, synthetic-data generation, reduced sample variance, faster sampling, and efficient architectures as future directions.
- 6 CONCLUSION: DiffusionSat combines high-resolution satellite-image generation from metadata and text with 3D conditioning for super-resolution, temporal prediction, and in-painting.It is based on the latent-diffusion architecture of Stable Diffusion.
- 6 CONCLUSION: Future work includes expanding training to larger and more diverse satellite-imagery datasets.
- 6 CONCLUSION: The authors propose testing synthetic-data generation to augment discriminative methods and scale to larger datasets.
- 6 CONCLUSION: Reducing output variance is identified as important because inverse-problem outputs can sometimes hallucinate details.
- 6 CONCLUSION: Faster sampling and more efficient architectures are proposed to support deployment in resource-constrained settings.
A.1 TRAINING DETAILS
The experiments use multi-GPU training, half precision, gradient checkpointing, and task-specific DiffusionSat variants and ControlNets. Sampling uses a common DDIM configuration, while training schedules and conditioning inputs vary by task.
- A.1 TRAINING DETAILS: All models use half precision and gradient checkpointing with the Diffusers library.
- A.1 TRAINING DETAILS: Single-image models train on 8 NVIDIA A100 GPUs for 100000 iterations at a 2e-6 learning rate, with 512×512 and 256×256 variants.
- A.1 TRAINING DETAILS: Single-image sampling uses DDIM with 100 steps and guidance scale 1.0, generating 10000 fMoW-RGB validation samples.
- A.1 TRAINING DETAILS: Super-resolution uses the 512-pixel single-image model as a prior and trains a ControlNet on upsampled multispectral inputs for 50000 iterations.The conditional image is upsampled to 256×256 pixels, and Sentinel bands B1, B9, and B10 are dropped.
- A.1 TRAINING DETAILS: Texas Housing trains a 3D ControlNet on high-resolution images and two low-resolution Sentinel-2 images, using the 256-pixel model as prior.
- A.1 TRAINING DETAILS: fMoW temporal generation trains a 3D ControlNet with at most three conditioning images, padding shorter sequences by copying an input image.
A.2.1 CAPTIONS AND METADATA
Captions are built from dataset-specific label and metadata fields, while seven numerical metadata fields are normalized and incorporated separately. Figures provide examples of the normalization ranges and dataset-specific captions and metadata.
- A.2.1 CAPTIONS AND METADATA: Captions use available dataset labels and metadata, with optional caption sections independently dropped at a 10% rate.
- A.2.1 CAPTIONS AND METADATA: The method incorporates numerical metadata from 7 fields in addition to captions.
- A.2.1 CAPTIONS AND METADATA: Metadata normalization maps each field from its reference range to a scaled value using mnorm = m/(high−low)×scale.The low and high reference values map to 0 and the chosen scale, respectively.
- A.2.1 CAPTIONS AND METADATA: The metadata fields include longitude, latitude, GSD, cloud cover, year, month, and day, each with specified normalization ranges.
- A.2.1 CAPTIONS AND METADATA: Figure 7 summarizes numerical metadata fields and their value ranges used for normalization.
- A.2.1 CAPTIONS AND METADATA: Figure 8 shows sample captions and pre-normalization metadata for the fMoW, Satlas, and SpaceNet datasets.
A.3 TEMPORAL GENERATION
DiffusionSat generates satellite-image sequences autoregressively using captions, metadata, and a 3D-conditioning ControlNet. The evaluation examines temporal behavior and geographical variation, while noting important metric, sampling, data-coverage, and bias-axis limitations.
- A.3 TEMPORAL GENERATION: The temporal pipeline first generates an image from a caption and metadata, then autoregressively generates subsequent images using the 3D-conditioning ControlNet.
- A.3 TEMPORAL GENERATION: The conditioning mechanism supports both conditional and unconditional generation of arbitrarily long satellite-image sequences.
- A.3 TEMPORAL GENERATION: Generated sequences reflect seasonal changes and development trends, including fewer structures in past images and more detail in future images.
- A.4 GEOGRAPHICAL BIAS: Geographical evaluation uses FID and LPIPS across latitude and longitude, with figures reporting results for single-image and super-resolution models.
- A.4 GEOGRAPHICAL BIAS: The reported results show no particular location favoritism, including despite an expectation of better quality in North America and Europe.
- A.4 GEOGRAPHICAL BIAS: FID and LPIPS may not be sufficiently informative for estimating geographical bias in generative remote-sensing models.They are used because better alternatives are lacking for this problem.
- A.4 GEOGRAPHICAL BIAS: FID depends on sample size, and the datasets contain many more samples from developed regions than from large areas such as Africa.
- A.4 GEOGRAPHICAL BIAS: The analysis estimates only one bias angle, leaving possible bias across buildings, roads, trees, crops, and seasons for future work.