Source-linked AI summary
TerraMind: Large-Scale Generative Multimodality for Earth Observation
Johannes Jakubik, Felix Yang, Benedikt Blumenstiel, Erik Scheurer, Rocco Sedona, Stefano Maurogiovanni, Jente Bosmans, Nikolaos Dionelis, Valerio Marsocci, Niklas Kopp, Rahul Ramachandran, Paolo Fraccaro, Thomas Brunschwiler, Gabriele Cavallaro, Juan Bernabe-Moreno, Nicolas Longépé
TL;DR
Earth observation needs models that integrate complementary, heterogeneous modalities while preserving both contextual and fine-grained spatial information. TerraMind addresses this with dual-scale any-to-any generative pretraining across nine geospatial modalities, and reports beyond-state-of-the-art benchmark performance alongside gains from generated data during finetuning.
Problem
Earth observation is spatiotemporally sparse and contains heterogeneous modalities whose complementary information benefits multimodal integration.
Method
TerraMind uses dual-scale early fusion of pixel-level and token-level representations, with masked token reconstruction over nine geospatial modalities in the TerraMesh dataset.
Results
TerraMind outperforms other geospatial foundation models in unimodal and multimodal settings and achieves beyond-state-of-the-art performance on community-standard EO benchmarks.
Takeaways & Limitations
Thinking-in-Modalities uses generated data during finetuning to improve model performance and supports generative multimodal capabilities for EO.
Abstract
from arXiv · showhide
We present TerraMind, the first any-to-any generative, multimodal foundation model for Earth observation (EO). Unlike other multimodal models, TerraMind is pretrained on dual-scale representations combining both token-level and pixel-level data across modalities. On a token level, TerraMind encodes high-level contextual information to learn cross-modal relationships, while on a pixel level, TerraMind leverages fine-grained representations to capture critical spatial nuances. We pretrained TerraMind on nine geospatial modalities of a global, large-scale dataset. In this paper, we demonstrate that (i) TerraMind's dual-scale early fusion approach unlocks a range of zero-shot and few-shot applications for Earth observation, (ii) TerraMind introduces "Thinking-in-Modalities" (TiM) -- the capability of generating additional artificial data during finetuning and inference to improve the model output -- and (iii) TerraMind achieves beyond state-of-the-art performance in community-standard benchmarks for EO like PANGAEA. The pretraining dataset, the model weights, and our code are open-sourced under a permissive license.
1. Introduction
TerraMind addresses the need to integrate complementary, heterogeneous EO data by introducing an any-to-any generative multimodal model with dual-scale representations. Its pretraining and modality-thinking capabilities support broad EO applications and improved downstream performance.
- Motivation: Earth observation benefits from multimodality because complementary data sources help address sparse observations caused by revisit gaps and cloud coverage.Available modalities include satellite sensors, digital elevation, and other complementary geospatial information.
- Contribution: TerraMind is introduced as the first any-to-any generative multimodal model for Earth observation.
- Contribution: Dual-scale pretraining combines pixel-level and token-level data, encoding contextual relationships while retaining fine-grained spatial representations.The approach is presented as improving over training primarily on tokens.
- Contribution: TerraMind’s dual-scale approach outperforms other fusion approaches and enhances embedding-space structures.
- Results: Generated data during finetuning improves model performance, while TerraMind outperforms other geospatial foundation models in unimodal and multimodal settings.
2. Related Work
Related work spans computer-vision methods and multimodal integration across natural and Earth-observation data. EO multimodality commonly combines radar, optical, vector, text, or multiple views, while TerraMind is illustrated through benchmark performance and chained generation.
- Computer vision in Earth observation: Computer vision methods in Earth observation support classification, semantic segmentation, change detection, object detection, and regression.CNNs and Vision Transformers are among the architectures adapted to EO.
- Multimodality in computer vision: Multimodal computer vision integrates diverse streams such as images, text, video, and weather within foundation models.
- TerraMind examples: Figure 2 compares finetuned geospatial foundation models on PANGAEA using dataset-wise min-max-scaled mIoU.The caption states that TerraMind outperforms the other models.
- TerraMind examples: Figure 3 depicts chained generation from optical, radar, or digital elevation inputs, with generated data compared against ground truths and tokenizer reconstructions.
- Multimodality in Earth observation: EO multimodality traditionally integrates SAR and optical data or combines optical imagery with vector data.Other studies also consider text-image pairs, alternative image sources, and multiple optical sensors as modalities.
3. Dataset
TerraMesh is the multimodal pretraining dataset for TerraMind, assembled from aligned remote-sensing sources and expanded with task-specific, metadata, and language modalities. It contains approximately nine million globally distributed samples.
- Dataset construction: TerraMesh is created for TerraMind and its tokenizers and is intended to be open-sourced to the community.It builds on existing datasets and adds modalities from external sources or through pseudo-labeling.
- Model context: Table 1 positions TerraMind as a first-of-its-kind generative, multimodal model relative to other model architectures.
- Base datasets: The base datasets provide co-aligned Sentinel-1 radar and Sentinel-2 optical imagery, with complementary geographic and seasonal coverage.SSL4EO-S12 is multi-seasonal with lower coverage, while MajorTOM-Core covers most land at one timestamp.
- Dataset scale: Approximately 9 million globally distributed TerraMesh samples come from Sentinel-1 and Sentinel-2 data.Each sample measures 264×264 pixels at 10m resolution.
- Additional modalities: Additional modalities include yearly nine-class LULC maps, cloud and ice annotations, NDVI, globally covered DEM data, and synthetic captions.NDVI is computed from Sentinel-2 bands, while DEM comes from the Copernicus DEM 30m dataset.
4. Methods
TerraMind uses modality-specific tokenizers and a dual-scale encoder-decoder that combines discrete token and pixel-level inputs for multimodal generation. Its masked pretraining reconstructs target tokens from sampled inputs, while Thinking-in-Modalities recursively augments inputs with generated modalities.
- Tokenization: TerraMind first trains modality-specific tokenizers, then pretrains an encoder-decoder using token-level and pixel-level inputs.Image-like modalities are discretized into patch tokens, while captions and geolocations use a text tokenizer.
- Tokenization: Image-like tokenizers quantize 16×16-patch representations into codewords and use diffusion decoders to reconstruct images.Most tokenizers use a 16K vocabulary; LULC uses 4K.
- Architecture: Dual-scale processing links cross-modal token relationships with pixel-level spatial nuances and structures pixel data in the embedding space.The model accepts both discrete tokens and pixel-level satellite imagery or digital elevation maps.
- Pre-training: Masked pretraining samples input and target tokens from multimodal data and trains the decoder to reconstruct masked targets using cross entropy.This strategy prepares the model to operate when entire modalities are absent and reduces computation through compressed token representations.
- Generation: After pretraining, TerraMind generates any modality from any subset of input modalities, decoding image-like outputs through modality-specific tokenizers.Sequence-like outputs are generated autoregressively, while image-like outputs are generated from predicted tokens.
- Thinking-in-Modalities: Thinking-in-Modalities recursively generates synthetic modalities, concatenates them with observed inputs, and jointly processes the expanded input for downstream prediction.Multiple modalities can be generated in a chain, particularly when some modalities are missing.
5. Experiments
Experiments show that TerraMind’s multimodal fusion supports stronger water-body mapping, any-to-any generation, zero-shot applications, and competitive fine-tuning, while TiM improves downstream performance.
- Foundational experiments: Combining modalities consistently outperforms unimodal water-body mapping, while full multimodality improves individual performance by 7.5pp IoU.The comparison uses Sen1Floods11 water-class IoU with comparable model sizes and architectures.
- Foundational experiments: Token-level fusion outperforms late fusion, with especially large gains when LULC data is unavailable.The authors attribute this to early fusion capturing an internal multimodal-state representation that benefits fine-tuning.
- Generation experiments: TerraMind generates image-like modalities including S-1, LULC, and DEM from optical S-2 L2A inputs, with quality evaluated on unseen validation data.The paper also illustrates S-1 generations from inputs covering Singapore and Northern Spain.
- Zero-shot experiments: 45.4% zero-shot IoU is achieved by TerraMindv1-B on water-body mapping, rising to 69.8% in a DynamicWorld-trained ablation.The reported 82.2% DeCUR fine-tuning result provides the comparison point; benchmark models cannot be applied zero-shot.
- Zero-shot experiments: TerraMind’s geo-localization distribution for bare land highlights the Sahara, Middle East, Mexico, and Southern California.The distribution is obtained through Monte Carlo sampling and is described as fitting expected bare-land locations.
- Fine-tuning experiments: TerraMindv1-B outperforms other GeoFMs by at least 3pp average mIoU on PANGAEA, while TiM tuning improves water mapping by up to 2pp mIoU.TerraMindv1-L adds approximately 2pp average mIoU and peaks at 5pp on multimodal datasets; both optical and radar inputs work best across multimodal tasks.
6. Conclusion
TerraMind combines token-level and pixel-level data to provide generative multimodal capabilities and strong benchmark performance in Earth observation.
- Conclusion: Combining token-level and pixel-level data unlocks new model capabilities in EO.The conclusion describes TerraMind as the first fully generative multimodal model in the domain and reports beyond-state-of-the-art community-benchmark performance.
8. TerraMesh Dataset
TerraMesh is a large-scale, globally distributed EO pretraining dataset with nine aligned modalities and preprocessing designed for geographic, thematic, and temporal quality.
- Dataset composition: TerraMesh contains over 9 million globally distributed, spatiotemporally aligned samples across nine core modalities.Modalities are co-registered at 10-meter resolution, primarily on Sentinel-2 grids.
- Dataset composition: TerraMesh integrates Sentinel-1 SAR, Sentinel-2 optical products, DEM, LULC, NDVI, cloud annotations, and synthetic captions.Sentinel-2 coverage includes L1C top-of-atmosphere and L2A bottom-of-atmosphere reflectance.
- Sampling and preprocessing: Subsampling preserves representative global ecoregions and land-cover classes while downsampling highly homogeneous regions such as deserts and tundra.The design targets broad geographic and thematic diversity.
- Sampling and preprocessing: Preprocessing includes reprojection, temporal alignment, and filtering to reduce missing data and artifacts.These steps produce high-quality, analysis-ready samples.
- Model-specific subset: TerraMindv1-B-single uses a one-million-sample TerraMesh subset with four image modalities and additional image captions.The captions were generated using LLaVA-Next and Overture Maps data; expert evaluation found 69% without hallucinations and average completeness of 3.87/5.
9. Pretraining details
TerraMind uses ViT-based tokenizers and large-scale distributed pretraining, with experiments comparing model scaling and tokenizer design choices.
- Tokenizer pretraining: Tokenizer models use ViT encoders, patched UNet decoders, and modality-image sizes from 224x224 to 256x256.The encoder uses 16x16 patches, while the decoder uses 4x4 patches and a tanh MLP before quantization.
- Tokenizer pretraining: Finite-Scalar Quantization uses an 8-8-8-6-5 codebook and latent dimension 5 to learn consistent abstract patch representations.Codebook entries are normalized to the unit sphere during training.
- Diffusion pretraining: Diffusion pretraining uses 1000 timesteps, a linear beta schedule, MSE loss, fp16, AdamW, and model EMA.Training also applies warmup, gradient clipping, random horizontal flips, and batch size 1 per GPU.
- Training setup: Tokenizer pretraining runs for 100 epochs on four GPUs, with a global batch size of 256 and base learning rate 1e-4.GPU utilization reaches 99% for single-channel modalities and exceeds 80% for multichannel modalities.
- Training setup: TerraMindv1-B and v1-L are pretrained on 32 GPUs with a 2e-4 learning rate and cosine schedule over 500B tokens.TerraMindv1-B training took 12 days on 32 A100 GPUs, totaling 9’216 GPU hours.
- Scaling behavior: Scaling from TerraMindv1-B to v1-L produces a significant validation-loss gap across modalities, with v1-L outperforming v1-B after approximately 10% of its training schedule.The comparison uses identical hyperparameters and compute for the displayed scaling analysis.
10. Tokenizer performance and general learnings
TerraMind’s tokenizer experiments show that reconstruction quality is sensitive to compression, learning rate, quantization, and codebook choices. FSQ offers more stable training than VQ without improving reconstruction loss, while tokenizer performance remains broadly consistent across regions and modalities.
- General learnings: Up to 3000x compression creates substantial tokenizer sensitivity, with high learning rates causing NaN losses and low rates producing blurry reconstructions.The models are also sensitive to the learning rate because of non-differentiability at excessive rates.
- Codebook size: Codebook sizes above 16K introduce significant reconstruction artefacts for modalities with more than three input channels.Reconstructions with a 32K codebook showed more artefacts than those with 16K.
- Tokenizer updates: EMA updates smooth fine-grained features and can produce hallucination-like reconstructions, including removing bridges from images.The tokenizer models therefore omitted exponential moving average updates.
- Quantization: FSQ and VQ reach the same reconstruction performance, but FSQ converges more smoothly and requires less tuning.FSQ avoids the additional codebook loss required by VQ, making training more stable and less learning-rate-sensitive.
- Performance: Tokenizer validation shows low MSE and low geographic deviation overall, with minor optical reconstruction difficulties in Northern Asia.The assessment covers S-1 GRD, S-2 L2A, and DEM tokenizers on the SSL4EO-S12 validation subset.
- Performance: Optical reconstructions show structural and perceptual fidelity, while radar reconstructions remain visually plausible but have moderate structural deviations.S-1 GRD and S-1 RTC achieve comparable PSNR values, but their SSIM scores are lower.
11. Additional experiments
Additional experiments examine TerraMind’s geolocation, few-shot classification, fine-tuning comparisons, and Thinking-in-Modalities. TerraMind performs strongly across several settings, while generated auxiliary data provides a measurable benefit for crop mapping.
- Geolocation prediction: TerraMind’s predicted geolocations usually fall near training locations with similar optical samples, enabling geolocation of unseen areas through image similarity.The model can overpredict urban areas in North Africa and the Middle East, likely confusing bare land with urban areas.
- Few-shot experiments: TerraMind performs best on both EuroSAT and METER-ML few-shot datasets, outperforming other geospatial foundation models and CLIP.The base model performs better overall than the large model in these experiments.
- Finetuning comparisons with baseline models: TerraMindv1-B outperforms task-specific U-Net and ViT models across nine PANGAEA datasets by 1pp average mIoU in unimodal settings and 4pp in multimodal settings.Its multimodal advantage over task-specific U-Nets peaks at 4.5pp average mIoU.
- Finetuning comparisons with baseline models: TerraMindv1-B generally outperforms TerraMindv1-B-single, supporting the benefit of combining pixel-level and token-level representations.The comparison evaluates most datasets in the benchmark.
- Thinking-in-Modalities: TerraMind’s generated LULC maps outperform DynamicWorld but remain inferior to a specialized fine-tuned model on Sen1Floods11.The comparison evaluates out-of-the-box TerraMind generations against pseudo-label-derived flood maps.
- Thinking-in-Modalities: TiM tuning improves South Africa crop-type mapping performance by around 1pp by adding generated agricultural land-use information.The generated data does not specify crop locations or shapes but indicates where cropland is generally expected.
12. Any-to-any generation
TerraMind supports any-to-any generation across image-like and sequence-like modalities, with strongest quality from information-rich inputs. Tokenized inputs preserve generation performance with only minor reductions despite substantial compression.
- Any-to-any generation: Fine-grained, information-rich modalities such as optical and radar produce particularly good any-to-any generations.The model can also generate consistent artificial data from modalities with less information content.
- Chained generation: TerraMind can generate contextually appropriate modalities from geolocation alone, including desert-like optical imagery for a Middle Eastern location.The generated image matches the broad context, although its structure may differ from the ground-truth scene.
- Tokenized generation: Tokenized inputs cause only minor performance reductions compared with pixel-level inputs, despite up to 3000x compression for S-2 L2A.This suggests tokenized inputs can substitute for pixel-level data when generating artificial modalities.
- Large-tile generation: Large-tile generation iteratively produces overlapping 224x224-pixel outputs for S-1 RTC, S-1 GRD, NDVI, and LULC from a 35.5km × 69.5km optical tile.Averaging overlapping generations improves spatial consistency, and TerraMind consistently removes clouds in S-1 generations.