PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Geospatial Foundation Models via Continual Pretraining

Matias Mendieta, Boran Han, Xingjian Shi, Yi Zhu, Chen Chen

arXiv:2302.04476v3cs.CV

TL;DR

Geospatial foundation modeling must balance downstream performance against the cost of training domain-specific models and the limited gains of direct ImageNet adaptation. This paper constructs GeoPile and uses multi-objective continual pretraining to combine ImageNet representations with geospatial self-supervision, reporting favorable performance across seven downstream datasets and diverse tasks.

  • Problem

    Geospatial foundation modeling requires approaches that improve downstream performance without the prohibitive resource and carbon costs of training large models from scratch.

  • Method

    The paper constructs the compact, diverse GeoPile dataset and applies multi-objective continual pretraining with ImageNet representations, distillation, and self-supervised masked image modeling.

  • Results

    GFM performs favorably over previous geospatial pretraining methods across seven downstream datasets covering change detection, classification, multi-label classification, semantic segmentation, and super-resolution.

  • Takeaways & Limitations

    The approach provides an efficient paradigm for developing geospatial foundation models by combining pretrained ImageNet knowledge with learned in-domain features.

  • Takeaways & Limitations

    GFM may remain constrained by the performance of the ImageNet-22k model, while training from scratch on extremely large remote-sensing corpora could improve performance at higher time and CO2 cost.

Abstract

from arXiv · show

Geospatial technologies are becoming increasingly essential in our world for a wide range of applications, including agriculture, urban planning, and disaster response. To help improve the applicability and performance of deep learning models on these geospatial tasks, various works have begun investigating foundation models for this domain. Researchers have explored two prominent approaches for introducing such models in geospatial applications, but both have drawbacks in terms of limited performance benefit or prohibitive training cost. Therefore, in this work, we propose a novel paradigm for building highly effective geospatial foundation models with minimal resource cost and carbon impact. We first construct a compact yet diverse dataset from multiple sources to promote feature diversity, which we term GeoPile. Then, we investigate the potential of continual pretraining from large-scale ImageNet-22k models and propose a multi-objective continual pretraining paradigm, which leverages the strong representations of ImageNet while simultaneously providing the freedom to learn valuable in-domain features. Our approach outperforms previous state-of-the-art geospatial pretraining methods in an extensive evaluation on seven downstream datasets covering various tasks such as change detection, classification, multi-label classification, semantic segmentation, and super-resolution.

1. Introduction

Geospatial foundation models face a trade-off between limited gains from directly adapting ImageNet models and the substantial cost of training domain-specific models from scratch. The paper proposes GeoPile and multi-objective continual pretraining to combine ImageNet representations with learned geospatial features, achieving favorable performance across diverse downstream tasks.

  • Directly fine-tuning ImageNet models is straightforward but may leave performance gains unrealized.
  • Training geospatial models from scratch can require substantial data, time, computation, and environmental cost.SatMAE requires 768 hours on a V100 GPU to train a vision transformer.
  • GeoPile is a compact, diverse dataset assembled from multiple sources to promote feature diversity and effective pretraining.
  • The proposed multi-objective continual pretraining combines distillation with self-supervised masked image modeling.It uses ImageNet representations to guide learning while allowing valuable in-domain features to be learned from geospatial data.
  • GFM is evaluated against baseline and state-of-the-art methods on 7 datasets spanning change detection, classification, multi-label classification, semantic segmentation, and super-resolution.The paper reports favorable overall performance across these downstream tasks.

2. Related Work

Prior geospatial pretraining work includes supervised, self-supervised, and masked-image-modeling approaches, while continual pretraining has mainly been developed in natural language. The paper builds on these directions while addressing the challenges of augmentation design and the cost of learning large geospatial models from scratch.

  • Geospatial Pretraining: Geospatial pretraining has used supervised learning, self-supervised objectives, contrastive procedures, colorization, and masked image modeling.
  • Geospatial Pretraining: SatMAE trains a large vision transformer with masked image modeling, but often does not outperform an off-the-shelf ImageNet-22k pretrained ViT.
  • Masked Image Modeling: Masked image modeling learns representations by predicting pixel values or tokens for intentionally withheld image regions.
  • Masked Image Modeling: Contrastive geospatial methods can make augmentation selection difficult because transformations may alter meaningful remote-sensing information.
  • Continual Pretraining: Continual pretraining has primarily been used in natural language for domain-adaptive and task-adaptive training stages.

3. Methodology

The methodology combines a compact, diverse GeoPile dataset with multi-objective continual pretraining that transfers ImageNet-22k representations while learning geospatial features. The resulting GFM improves downstream performance with lower computational and carbon costs than comparable approaches.

  • 3.1. Pre-training Data Selection: The evaluation uses the ARP metric across seven tasks, including classification, multi-label classification, semantic segmentation, change detection, and super-resolution.Training time and carbon impact are also reported for each method.
  • 3.1. Pre-training Data Selection: GeoPile contains approximately 600k labeled and unlabeled satellite samples spanning varied ground sample distances and scenes.The dataset is designed to increase feature diversity beyond Sentinel-2 imagery.
  • 3.3. GFM Pretraining: GFM uses a frozen ImageNet-22k teacher branch to guide a randomly initialized student while masked image modeling learns features directly from geospatial data.The teacher receives the unmasked image, and its intermediate features guide the student through an auxiliary objective.
  • 3.2. Vanilla Continual Pretraining: Vanilla continual pretraining initializes the model with ImageNet-22k weights before MIM training on GeoPile, improving ARP by approximately 0.3% over training from scratch and 1.24% over the baseline.The baseline is a Swin-B model pretrained on ImageNet-22k.
  • 3.3. GFM Pretraining: The student optimizes the masked-image-modeling and feature-guidance objectives together, combining transferred representations with adaptation to in-domain data.The final loss is the sum of the two objectives.
  • 3.3. GFM Pretraining: GFM achieves better overall performance than vanilla continual pretraining with substantially less computation and emissions, reducing training time and carbon impact by more than 8× versus SatMAE.SatMAE is reported to require 768 V100-GPU hours and 109.44 kg CO2 equivalent.

4. Experiments

GFM is evaluated across seven geospatial datasets spanning change detection, classification, segmentation, and super-resolution. It generally improves on ImageNet-22k and prior geospatial pretraining methods across these tasks.

  • GFM is evaluated on seven datasets covering change detection, classification, multi-label classification, semantic segmentation, and super-resolution.
  • Change Detection: GFM consistently improves over the ImageNet-22k baseline on OSCD and DSIFN change detection.SatMAE improves over ImageNet-22k on OSCD but lags behind it on DSIFN.
  • Classification: GFM provides robust performance on UC Merced and BigEarthNet while retaining strong BigEarthNet performance with only 1% of the training data.The evaluation reports accuracy on UC Merced and mean average precision on BigEarthNet.
  • Segmentation: GFM improves segmentation over strong ImageNet-22k representations on both WHU Aerial and Vaihingen.The segmentation evaluation reports IoU results for both datasets.
  • Super-Resolution: GFM substantially improves over its Swin ImageNet-22k baseline in both PSNR and SSIM on SpaceNet2 super-resolution.

5. Ablation Studies

Ablations identify the effects of distillation placement, initialization, GeoPile composition, temporal pairs, and training objectives. The strongest configuration uses Stage 3 distillation and integrates frozen-teacher distillation with masked image modeling.

  • Distillation Stage: Distillation after Stage 3 yields the highest ARP and is used for all downstream experiments.The ablation compares feature-map distillation locations across Swin transformer stages.
  • Student Initialization: Initializing the student with ImageNet weights is not optimal; the framework instead keeps the ImageNet teacher initialized and frozen while randomly initializing the student.The framework integrates ImageNet representations through distillation during geospatial pretraining.
  • GeoPile Composition: Using labeled GeoPile datasets outperforms using only unlabeled NAIP imagery, even with fewer images.The ablation removes individual datasets and compares curated labeled data with NAIP-only data.
  • Training Objectives: The multi-objective approach performs best overall, showing that both teacher distillation and MIM contribute to balanced downstream performance.The ablation separately removes the teacher and MIM components.
  • Temporal Pairs: Temporal-pair training uses one image for the teacher and the other for the student, while same-image training uses identical inputs.

6. Conclusion

The paper proposes an efficient geospatial foundation-model paradigm that combines diverse remote-sensing data with ImageNet-22k-guided continual pretraining. The resulting approach is intended to support sustainable development of geospatial foundation models.

  • 6. Conclusion: The method aims to produce better geospatial foundation models with substantially lower resource costs.
  • 6. Conclusion: The approach combines a concise, diverse collection of remote-sensing data with multi-objective continual pretraining from ImageNet-22k.ImageNet representations guide learning while self-supervised learning enables acquisition of in-domain features.
  • 6. Conclusion: The proposed GFM approach is presented as an example for investigating efficient and sustainable geospatial foundation-model development.
  • 6. Conclusion: The paper identifies a potential constraint: performance may remain limited by the ImageNet-22k model used for initialization.Training from scratch on an extremely large remote-sensing corpus might improve performance, but would require substantial training time and CO2 impact.

Supplementary Material

The supplementary material provides training details, carbon-impact calculation details, and additional analysis of the SpaceNet2 super-resolution task.

  • Supplementary Material: Section A gives training details for the pretraining stage and all downstream tasks.
  • Supplementary Material: Section B explains how CO2 impact is calculated.
  • Supplementary Material: Section C presents further analysis of the SpaceNet2 super-resolution task.

A. Training Details

The supplementary training details specify implementation settings for change detection, classification, segmentation, and super-resolution experiments, with additional results and ablations reported in Table 12.

  • A. Training Details: Code, model weights, and the GeoPile dataset are publicly available.
  • A. Training Details: The authors use MMsegmentation for change detection and segmentation experiments.Change-detection training uses dataset-specific tiling, image sizes, and iteration counts; segmentation uses 40k iterations with compatible decoders.
  • A. Training Details: Classification experiments define dataset-specific image sizes, batch sizes, learning rates, training duration, and augmentation settings.UC Merced uses Mixup and standard image augmentations, while BigEarthNet images are upscaled to 128×128 for Swin-transformer compatibility.
  • A. Training Details: SpaceNet2 super-resolution training uses 160×160 inputs, 640×640 targets, 100 epochs, and a base learning rate of 1.25e-5.The setup omits random augmentations and uses a decoder followed by convolution-based upsampling.
  • A. Training Details: Detailed downstream results and ablations from the main manuscript are provided in Table 12.

B. Training Time and Carbon Calculations

The authors estimate training carbon impact using the ML CO2 Impact estimator, accounting for hardware, provider, region, and training time. Their GFM pretraining requires 93.3 V100 GPU hours and produces 13.3 kg eq. CO2.

  • B. Training Time and Carbon Calculations: 93.3 V100 GPU hours are needed to train the GFM, resulting in a total carbon impact of 13.3 kg eq. CO2.The experiments were conducted in AWS US East (Ohio), with a carbon efficiency of 0.57 kg eq. CO2 per kWh.
  • B. Training Time and Carbon Calculations: The estimated impact depends on hardware type, GPU provider, region, and total training time.

C. Super-resolution with Residual Connection

Adding a residual connection improves super-resolution performance for Swin but not ViT. Across baseline comparisons, GFM improves over its ImageNet-22k baseline, whereas SatMAE does not.

  • A residual connection benefits Swin in super-resolution but does not benefit ViT.The modification was evaluated for both transformer architectures.
  • SatMAE does not improve over its ImageNet-22k baseline in super-resolution.
  • GFM improves over its ImageNet-22k baseline in super-resolution.
Loading 2302.04476v3…