PaperlayerView exact revision on arXiv

Source-linked AI summary

A Billion-scale Foundation Model for Remote Sensing Images

Keumgang Cha, Junghoon Seo, Taekyung Lee

arXiv:2304.05215v4cs.CVcs.AIcs.LG

TL;DR

Remote-sensing foundation-model research has focused mainly on pretraining methods and dataset size, with less emphasis on model parameters. The paper scales and fine-tunes vision transformers pretrained with MAE on MillionAID, and reports improved performance and sample efficiency as model size increases across downstream tasks.

  • Problem

    Remote-sensing pretraining research has emphasized dataset size and learning methods, with limited evidence on how foundation-model parameter count affects downstream performance.

    Source: §I · p. 2

  • Method

    The study pretrains vision transformers of different sizes with MAE on MillionAID and adapts them for remote-sensing detection and segmentation tasks.

    Source: §I · p. 2, 2) MAE · p. 4, B. Fine Tuning Vision Transformer for… · p. 6

  • Results

    Larger models, especially those pretrained on comprehensive remote-sensing datasets, show better downstream performance and sample efficiency across varied tasks.

    Source: §I · p. 2, 3) Experiment Results · p. 9

  • Takeaways & Limitations

    The findings support building remote-sensing-oriented foundation models by pretraining large vision transformers on extensive remote-sensing imagery and applying parallelism.

    Source: §V · p. 13

Abstract

from arXiv · show

As the potential of foundation models in visual tasks has garnered significant attention, pretraining these models before downstream tasks has become a crucial step. The three key factors in pretraining foundation models are the pretraining method, the size of the pretraining dataset, and the number of model parameters. Recently, research in the remote sensing field has focused primarily on the pretraining method and the size of the dataset, with limited emphasis on the number of model parameters. This paper addresses this gap by examining the effect of increasing the number of model parameters on the performance of foundation models in downstream tasks such as rotated object detection and semantic segmentation. We pretrained foundation models with varying numbers of parameters, including 86M, 605.26M, 1.3B, and 2.4B, to determine whether performance in downstream tasks improved with an increase in parameters. To the best of our knowledge, this is the first billion-scale foundation model in the remote sensing field. Furthermore, we propose an effective method for scaling up and fine-tuning a vision transformer in the remote sensing field. To evaluate general performance in downstream tasks, we employed the DOTA v2.0 and DIOR-R benchmark datasets for rotated object detection, and the Potsdam and LoveDA datasets for semantic segmentation. Experimental results demonstrated that, across all benchmark datasets and downstream tasks, the performance of the foundation models and data efficiency improved as the number of parameters increased. Moreover, our models achieve the state-of-the-art performance on several datasets including DIOR-R, Postdam, and LoveDA.

I. INTRODUCTION

Remote-sensing foundation-model research had emphasized pretraining methods and dataset size, while model scale received limited attention. This paper scales vision transformers from 86M to 2.4B parameters and evaluates them on object detection and segmentation.

I. INTRODUCTION

  • Natural-image pretraining has a domain gap with remote-sensing imagery, so sufficient performance still requires substantial high-quality labeled data.Collecting and labeling remote-sensing imagery is difficult because some targets are rare and expert labeling takes time.Source: §I · p. 1
  • Remote-sensing pretraining research had focused mainly on dataset size and learning method, with limited study of foundation-model size.Prior remote-sensing foundation models typically used ResNet50, ResNet101, or ViT-Base (86M).Source: §I · p. 2
  • The study varies model size from 86M to 2.4B while holding MillionAID, MAE pretraining, and the vision-transformer structure consistent with prior work.DOTA v2.0, DIOR-R, Potsdam, and LoveDA are used for downstream evaluation.Source: §I · p. 2
  • The paper introduces a billion-scale remote-sensing foundation model and a parallelism-based method for scaling vision transformers.The method is intended for remote-sensing tasks requiring high-resolution insights and computational efficiency.Source: §I · p. 2
  • Larger models, especially those pretrained on comprehensive remote-sensing datasets, show better downstream performance and sample efficiency.The evaluation covers rotated object detection and semantic segmentation.Source: §I · p. 2

II. RELATED WORKS

Computer-vision foundation models combine large datasets, model architectures, and pretraining methods. Self-supervised learning uses unlabeled images through approaches including contrastive learning, self-distillation, and masked image modeling.

A. Self-supervised Learning in Computer Vision

  • Foundation models are pretrained on large unlabeled datasets and can be fine-tuned across tasks, reducing task-specific development resources.Their adaptability is attributed to extracting meaningful representations from data.Source: A. Self-supervised Learning in… · p. 3
  • Foundation-model development depends on large-scale datasets, robust architectures, and pretraining methodology.Computer-vision models have scaled to 1.3B, 2B, and 22B parameters.Source: A. Self-supervised Learning in… · p. 3
  • Self-supervised learning trains representations from unlabeled images using contrastive learning, self-distillation, or masked image modeling.The approaches respectively compare image-pair similarities, predict relationships among image views, or reconstruct obscured image regions.Source: A. Self-supervised Learning in… · p. 3

B. Foundation Models in Remote Sensing

Remote-sensing foundation models apply self-supervised learning to data with spatial, temporal, and modality variation. Prior work includes single- and multi-modal contrastive learning and generative reconstruction methods.

B. Foundation Models in Remote Sensing

  • Remote-sensing foundation models use large-scale pretraining to support classification, detection, and segmentation, with quicker convergence and better downstream performance than training from scratch.The passage also describes this approach as mitigating task-specific overfitting through rich feature learning.Source: B. Foundation Models in Remote Sensing · p. 3
  • Single-modal contrastive learning uses augmented views as positive pairs, while time-series methods target temporal robustness and multimodal methods combine diverse data sources.Examples of modalities include optical imagery, SAR, near-infrared, text, and audio.Source: B. Foundation Models in Remote Sensing · pp. 3–4
  • Generative remote-sensing methods reconstruct obscured or removed image regions, and some approaches combine masked-image modeling with contrastive or multimodal learning.The cited approaches include geolocation prediction and multiple-view self-supervision.Source: B. Foundation Models in Remote Sensing · p. 4

III. METHODS

The method pretrains vision transformers with MAE on MillionAID, then adapts the pretrained model for remote-sensing tasks requiring localization and high-resolution imagery.

III. METHODS

  • The method uses vision transformers, MillionAID, and MAE, and adapts the plain transformer for rotated detection and semantic segmentation.These tasks require high-resolution input imagery and object-localization ability.Source: §III · p. 4

1) MillionAID:

  • MillionAID contains about 1,000,848 remote-sensing scenes across 51 classes, and only its images are used for pretraining.The study uses MillionAID alone to focus on model-parameter effects and enable fair comparison with prior work.Source: 1) MillionAID · p. 4

2) MAE:

  • MAE pretraining masks image patches and trains the model to reconstruct them from visible patches.Typically, 75% of patches are masked; the encoder processes the visible patches and the decoder reconstructs masked pixels using a reconstruction-error objective.Source: 2) MAE · p. 4
  • The MAE loss predicts masked-region pixel values from visible regions, and the passage reports that this pretraining enhances downstream performance.The authors suggest the encoder captures useful high-level representations.Source: 2) MAE · p. 4

3) Scaling Up Vision Transformer:

The section explores parallel MHSA and FFN configurations as a way to scale vision transformers for remote-sensing detection and segmentation. Matching layer-count × parallelism can preserve backbone parameters and FLOPs while configurations differ in performance.

3) Scaling Up Vision Transformer:

  • 12 layers with 1 parallelism and 6 layers with 2 parallelism have substantially the same backbone parameters and FLOPs when layer count × parallelism is held constant.The figure notes that performance can differ between serial and parallel configurations despite this matching.Source: Figure 3 · p. 5
  • Parallel MHSA and FFN configurations are proposed for scaling vision transformers to remote-sensing object detection and semantic segmentation.The paper notes that NLP scaling rules may not transfer effectively to vision transformers, where object-detection results are also less commonly reported.Source: 3) Scaling Up Vision Transformer · pp. 4–5
  • The parallel structure hypothesizes that each layer’s incremental effect diminishes as depth increases, approximating r(x + r′(x)) ≈ r(x).The approximation is used to motivate the parallel configuration, not reported as an empirical finding.Source: 3) Scaling Up Vision Transformer · p. 5
  • The parallel configuration aims to maintain computational efficiency while potentially simplifying optimization and improving performance on complex visual tasks.The section frames these as aims of the approach.Source: 3) Scaling Up Vision Transformer · p. 5

4) Implementation Detail for Pretraining:

The paper scales the vision transformer by changing width-related settings and parallelism while keeping depth fixed, then adapts the pretrained backbone for high-resolution downstream tasks. Detection and segmentation use task-specific feature resampling before their respective heads.

4) Implementation Detail for Pretraining:

  • Pretraining varies hidden size, MLP size, heads, and parallelism while fixing depth at 12 layers; models use 224 × 224 crops, 16 × 16 patches, 75% masking, and 400 MAE epochs.Activation checkpointing and fp16 address GPU-memory constraints; ViT-G12×4 uses a lower base learning rate because 1.5e−4 did not converge while 1e−4 did.Source: B. Fine Tuning Vision Transformer for… · p. 5
  • Because downstream tasks require localization and high-resolution inputs such as 1024 × 1024, the plain pretrained backbone is converted to ViTDET with local and global attention.Local attention uses a window size of 14 in blocks 1, 2, 4, 5, 7, 8, 10, and 11; global attention is used in blocks 3, 6, 9, and 12.Source: B. Fine Tuning Vision Transformer for… · pp. 5–6, Figure 4 · p. 6
  • Detection resamples the last-layer feature at ratios 4, 2, 1, and 0.5, while segmentation resamples features from layers 3, 6, 9, and 12 at those ratios before task-specific heads.Scale blocks provide the upsampling and downsampling; their operations include transposed convolution, normalization, GELU, identity, and max pooling.Source: B. Fine Tuning Vision Transformer for… · p. 6, Figure 4 · p. 6

IV. EXPERIMENTAL RESULTS

The experimental section evaluates pretrained models on rotated object detection and semantic segmentation, including a training-data-efficiency analysis. Its detection benchmarks are DOTA v2.0 and DIOR-R, with different dataset access and evaluation procedures.

IV. EXPERIMENTAL RESULTS

  • The evaluation examines whether downstream performance scales with pretrained-model parameter count and compares remote-sensing-pretrained models under the same MillionAID data and MAE method.The authors use this matched pretraining setup for a fair comparison with prior work.Source: §IV · p. 6
  • Sample efficiency is tested on DIOR-R detection and Potsdam segmentation using 1%, 5%, 10%, 50%, or 100% of training data and the full test set.Source: §IV · p. 6

1) Dataset:

  • DOTA v2.0 is used for rotated detection evaluation; its test-dev images are available without labels, so evaluation uses server submissions.The dataset contains 18 categories and 11,268 images.Source: 1) Dataset · pp. 6–7
  • DIOR-R is evaluated locally using its published training, validation, and test images and instances.It contains 20 categories; images are 800 × 800 pixels with resolutions from 0.5 m to 30 m.Source: 1) Dataset · pp. 6–7

2) Implementation Details and Experiment Settings:

Across rotated object detection benchmarks, larger backbones improve performance, with diminishing gains; on DIOR-R, larger backbones can also match full-data performance using half the training images. The experiments vary the pretrained backbone while holding detection settings largely fixed.

2) Implementation Details and Experiment Settings:

  • Detection uses a fixed RoI Transformer head, and models are trained with common hyperparameters across datasets while changing the pretrained backbone.The setup uses mmrotated; the reported schedule is 12 epochs with learning-rate reductions at epochs 8 and 11.Source: 2) Implementation Details and… · p. 7
  • For DOTA v2.0, images are cropped to 1024 × 1024 with stride 824; DIOR-R images remain uncropped at 800×800.Both datasets use random horizontal and vertical flips, with no test-time augmentation.Source: 2) Implementation Details and… · p. 7

3) Experiment Results:

  • Table IV reports class-wise AP and mAP on DOTA v2.0, while Table V reports the same measures on DIOR-R.The tables include re-implemented comparisons using the RoI Transformer framework.Source: 3) Experiment Results · pp. 7, 9, Table IV · p. 9, Table V · p. 9
  • On DIOR-R, ViT-H12×4 and ViT-G12×4 at 50% data achieve performance comparable to ViT-B12×1 at 100%.Training subsets are randomly sampled by image ratio.Source: 3) Experiment Results · p. 9
  • Larger backbones improve rotated-detection performance on DOTA v2.0 and DIOR-R, although mAP gains become less significant as parameter counts grow.The reported comparisons vary backbone size while holding other factors constant.Source: 3) Experiment Results · pp. 7, 9

B. Semantic Segmentation

The paper evaluates semantic segmentation on Potsdam and LoveDA, which differ in image composition, classes, and evaluation protocol.

B. Semantic Segmentation

  • Potsdam contains 38 images at 0.5m resolution, split into 24 training and 14 test images, with six pixel-level classes.Evaluation uses F1 score and overall accuracy, excluding clutter.Source: 1) Dataset · pp. 9–10
  • LoveDA contains 2,522 training, 1,669 validation, and 1,796 test images across rural and urban areas, with seven classes.The background class is excluded from learning and evaluation.Source: 1) Dataset · p. 10
  • LoveDA evaluation uses mean IoU across categories and server-submitted inference results, whereas Potsdam is evaluated locally on its published test images.The LoveDA training and test sets follow official criteria without separating rural and urban images.Source: 1) Dataset · p. 10

2) Implementation Details and Experiment Settings:

Semantic-segmentation experiments use a shared ViTDET-oriented pipeline with UperNet, dataset-specific training schedules, and patch-based processing.

2) Implementation Details and Experiment Settings:

  • UperNet is the segmentation head, and intermediate features from layers 3, 6, 9, and 12 are resampled at ratios 4, 2, 1, and 0.5.The pipeline uses mmsegmentation and applies layer-wise learning-rate decay and drop path to ViTDET.Source: 2) Implementation Details and… · p. 10
  • Potsdam training images are cropped into 512 × 512 patches at stride 384, while LoveDA images are randomly cropped to 512 × 512.Both datasets use resize-and-crop augmentation, flips, and photometric distortion; full-scene inference uses overlapping 512 × 512 patches.Source: 2) Implementation Details and… · pp. 10–11
  • Table VIII reports class-wise F1 and overall accuracy on Potsdam, while Table IX reports class-wise IoU and mIoU on LoveDA.The Potsdam evaluation excludes the clutter class.Source: Table VIII · p. 11, Table IX · p. 11

3) Experiment Results:

Across Potsdam and LoveDA, larger proposed backbones tend to improve semantic-segmentation performance. On Potsdam, the same pattern also appears in reduced-data experiments.

3) Experiment Results:

  • For Potsdam, the proposed models show a positive relationship between parameter count and both mF1 score and overall accuracy.The proposed models also outperform ImageNet-pretrained models on those two metrics.Source: 3) Experiment Results · p. 11
  • LoveDA shows a similar performance pattern as model size increases, despite using IoU rather than Potsdam’s metrics.Table IX presents the LoveDA IoU results.Source: 3) Experiment Results · p. 11
  • Figure 6 compares Potsdam image, label, and model predictions, and LoveDA image and model predictions across backbone sizes.Red marks false predictions in Potsdam; LoveDA test labels are unavailable.Source: Figure 6 · p. 12
  • On Potsdam, performance improves with larger backbones at the same sample ratio, and 5% or 10% subsets sometimes outperform the 50% subset.The subsets are sampled by cropped images, so class-pixel proportions are not exact.Source: 3) Experiment Results · pp. 11, 13

V. CONCLUSION

The authors report that large pretrained models demonstrate foundation-model capabilities in remote-sensing detection and segmentation. They identify fine-tuning, sample efficiency, and parallel vision-transformer layer stacking as effective for these tasks, while proposing broader data and tuning strategies for future study.

V. CONCLUSION

  • Large pretrained models demonstrate fine-tuning and sample-efficiency capabilities in remote-sensing rotated object detection and semantic segmentation.The authors also report that parallel stacking of vision-transformer layers effectively addresses these tasks.Source: §V · p. 13
  • The authors conclude that remote-sensing foundation models can be built by pretraining large vision transformers on extensive remote-sensing imagery and applying parallelism.Source: §V · p. 13
  • Future work will use a more comprehensive image collection, including MillionAID, and investigate prompt tuning, few-shot learning, and fine-tuning.Source: §V · p. 13
Loading 2304.05215v4…