PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Study of Remote Sensing Pretraining

Di Wang, Jing Zhang, Bo Du, Gui-Song Xia, Dacheng Tao

arXiv:2204.02825v4cs.CV

TL;DR

Existing aerial-image models commonly rely on ImageNet pretraining despite the domain gap between natural and aerial imagery. The paper trains CNN and vision-transformer backbones on MillionAID, then evaluates remote sensing pretraining across four downstream tasks. RSP is comparable with ImageNet pretraining overall, performs especially well for some remote-sensing semantics, and may still face task discrepancies.

  • Problem

    Most aerial-image models use ImageNet-pretrained weights even though natural and aerial images have a large domain gap, motivating empirical study of remote sensing pretraining.

  • Method

    The study trains CNN and vision-transformer backbones on MillionAID and compares remote sensing versus ImageNet pretraining across scene recognition, segmentation, detection, and change detection.

  • Results

    RSP is comparable with ImageNet pretraining overall and performs extremely well for specific categories such as “Bridge” and “Airplane”; ViTAEv2-S achieves the best performance on almost all settings.

  • Takeaways & Limitations

    RSP reduces data-level discrepancy for remote-sensing imagery, while pretraining benefits depend on how closely downstream task representations match scene recognition.

  • Takeaways & Limitations

    RSP may still suffer from task discrepancies because downstream tasks can require representations different from those learned for scene recognition.

Abstract

from arXiv · show

Deep learning has largely reshaped remote sensing (RS) research for aerial image understanding and made a great success. Nevertheless, most of the existing deep models are initialized with the ImageNet pretrained weights. Since natural images inevitably present a large domain gap relative to aerial images, probably limiting the finetuning performance on downstream aerial scene tasks. This issue motivates us to conduct an empirical study of remote sensing pretraining (RSP) on aerial images. To this end, we train different networks from scratch with the help of the largest RS scene recognition dataset up to now -- MillionAID, to obtain a series of RS pretrained backbones, including both convolutional neural networks (CNN) and vision transformers such as Swin and ViTAE, which have shown promising performance on computer vision tasks. Then, we investigate the impact of RSP on representative downstream tasks including scene recognition, semantic segmentation, object detection, and change detection using these CNN and vision transformer backbones. Empirical study shows that RSP can help deliver distinctive performances in scene recognition tasks and in perceiving RS related semantics such as "Bridge" and "Airplane". We also find that, although RSP mitigates the data discrepancies of traditional ImageNet pretraining on RS images, it may still suffer from task discrepancies, where downstream tasks require different representations from scene recognition tasks. These findings call for further research efforts on both large-scale pretraining datasets and effective pretraining methods. The codes and pretrained models will be released at https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing.

I. INTRODUCTION

Aerial scene recognition is difficult because bird’s-eye imagery contains inter-class similarity, large intra-class variation, and a domain gap from natural images. This motivates evaluating remote sensing pretraining with CNNs and vision transformers across four downstream tasks.

  • Challenges: Bird’s-eye imagery is difficult to interpret because scene-irrelevant regions and complicated object layouts create inter-class similarity and large intra-class variation.Scenes from different categories may share characteristics, while scenes within one category can differ substantially.
  • Motivation: Natural and aerial images have a large domain gap in view, color, texture, layout, and object characteristics.ImageNet pretraining may introduce systematic bias that affects remote-sensing performance.
  • Motivation: MillionAID makes remote sensing pretraining feasible because it is a million-level aerial image dataset comparable in scale to ImageNet-1K.The dataset contains abundant aerial imagery captured by diverse sensors.
  • Study Design: The study trains CNN and vision-transformer backbones with remote sensing pretraining and compares them with ImageNet-pretrained models.The investigated transformers include Swin and ViTAE, which model global context and, for ViTAE, combine convolutional and attention-based representations.
  • Study Design: The pretrained backbones are evaluated on scene recognition, semantic segmentation, object detection, and change detection across multiple datasets.The comparison includes remote sensing or ImageNet initialization and other methods.
  • Findings: Vision transformers can match or outperform CNNs, with ViTAE reported as best on almost all evaluated settings.The paper also presents findings comparing remote sensing and ImageNet pretraining across downstream tasks.

II. RELATED WORK

Related work covers CNN-based aerial scene recognition, feature-enhancement mechanisms, downstream remote-sensing tasks, and vision transformers. These approaches address local, global, multiscale, and long-range visual representation needs.

  • A. Aerial Scene Recognition: CNN-based aerial scene recognition commonly uses ImageNet-pretrained VGG, ResNet, and DenseNet models fine-tuned on aerial images.Because last-layer global features may be insufficient for scenes with inter-class similarity and intra-class diversity, methods also use multilevel features.
  • A. Aerial Scene Recognition: Attention mechanisms enhance local features by assigning different weights to regions, with channel and spatial attention used in CNN-based methods.These modules are used alongside feature-fusion strategies for aerial scene recognition.
  • Downstream Tasks: Scene-recognition backbones can serve as feature encoders for semantic segmentation, object detection, and change detection.Change detection is identified as a remote-sensing-specific task, whereas segmentation and detection are common in computer vision.
  • B. Vision Transformer: Vision transformers use self-attention to capture long-range context and relationships between different image positions.Multi-head self-attention applies attention in separate projected subspaces to strengthen representation capacity.
  • B. Vision Transformer: Hierarchical and windowed transformer designs improve transferability or reduce computational complexity compared with fixed-patch or global-attention designs.PVT introduces pyramid multiscale features, while Swin replaces global attention with shifted-window attention.

C. Remote Sensing Pretraining

The paper motivates remote sensing pretraining because large-scale RS datasets were scarce, then adopts MillionAID and ViTAE-family architectures to study RS-specific pretrained representations.

  • RS pretraining is proposed as an intuitive direction, but prior exploration was limited by the scarcity of large-capacity RS datasets.
  • MillionAID contains 1,000,848 non-overlapping RGB scenes across 51 leaf categories, making it the largest RS dataset described in the paper.
  • ViTAEv2 adopts a stage-wise backbone design to provide intermediate multiscale features needed for segmentation, detection, and pose estimation.
  • The adopted ViTAEv2 replaces MHSA with WMHSA in its first two stages to reduce computational cost while retaining multiscale communication through the PRM.
  • The experiments use ViTAE-S and ViTAEv2-S, with reduction cells, normal cells, attention types, and stage dimensions specified for the small architectures.

C. Implementation

The implementation compares CNN and vision-transformer candidates trained on MillionAID, using a mini-training and mini-evaluation setup to select models for later experiments.

  • The candidate study trains ResNet-50, DeiT-S, PVT-S, and Swin-T on balanced MillionAID mini-training and mini-evaluation sets.
  • All models use 300 epochs and batch size 16 with AdamW, momentum 0.9, weight decay 0.05, and initial learning rate 1e-3.
  • Training applies 224 × 224 random crops, cosine learning-rate scheduling, warmup, and augmentations including AutoAugment, Random Erasing, Mixup, CutMix, and color jitter.
  • ViTAEv2-S achieves the best candidate performance and exceeds the second place by 2.3% top-1 accuracy, while ViT-B does not outperform ResNet-50.
  • The selected downstream candidates are ViTAEv2-S for ViTAE, ResNet-50 for CNN, and corresponding RS-pretrained baselines.
  • RSP partitions MillionAID into 51,000 validation images and 949,848 training images while maintaining category balance.

2) Obtain the Suitable Weights:

The authors select pretrained weights by comparing training durations and validation performance, balancing saturation behavior with possible generalization from longer training.

  • The weight-selection experiments are reported on the MillionAID validation set through candidate-model and epoch-comparison tables.
  • ViTAEv2-S begins to saturate after about 40 epochs, with only 0.64% improvement from 20 to 40 epochs and 0.23% from the next 20 epochs.
  • The 40-epoch ViTAEv2-S weights are selected as the initial RSP parameters for downstream experiments.
  • Swin-T-E120 is selected because its validation top-1 accuracy is roughly equivalent to ViTAEv2-S-E40, while 40-epoch ResNet-50 and Swin-T weights support fair comparisons.
  • Validation accuracy generally increases with training epochs, but Swin-T-E300 performs worse than Swin-T-E120 and is retained as a possible stronger-generalization model.

IV. FINETUNING ON DOWNSTREAM TASKS

The downstream study fine-tunes RS-pretrained backbones across aerial scene recognition, semantic segmentation, object detection, and change detection, using established scene-recognition benchmarks.

  • The downstream evaluation covers scene recognition, semantic segmentation, object detection, and change detection in aerial imagery.
  • Scene-recognition models are evaluated on common aerial datasets rather than MillionAID, which is used for remote sensing pretraining.
  • The scene-recognition benchmarks are UCM, AID, and NWPU-RESISC.
  • UCM contains 2,100 images across 21 balanced categories, with 256 × 256 images and 0.3m pixel resolution.
  • AID contains 10,000 600 × 600 images across 30 categories collected under varied countries, seasons, times, sensors, and imaging conditions.
  • NWPU-RESISC contains 31,500 images across 45 categories, with spatial resolutions ranging from 0.2m to 30m.

2) Implementation Detail and Experimental Setting:

The study compares ImageNet, remote-sensing, and SeCo pretraining across CNN and vision-transformer backbones for aerial scene recognition. Remote-sensing pretraining generally improves recognition, with the strongest results from RSP-ViTAEv2-S across most settings.

  • Experimental setting: Table V compares IMP, RSP, and SeCo initialization across ResNet-50, Swin-T, and ViTAEv2-S on three scene-recognition datasets.The evaluation includes multiple training/testing splits and pretrained-weight durations.
  • Scene recognition results: RSP-ResNet-50 improves accuracy over ImageNet-pretrained ResNet-50 on all evaluated settings.The authors attribute this to aerial images in the pretraining data providing a better optimization starting point than natural images.
  • Scene recognition results: RSP-Swin-T outperforms IMP-Swin-T on three settings and achieves comparable results on the other two.RSP also makes ResNet-50 competitive with, and sometimes better than, ImageNet-pretrained Swin-T.
  • Scene recognition results: RSP-ViTAEv2-S achieves the best performance on almost all settings, except AID (5:5), where it remains comparable to RSP-Swin-T-E300.ViTAEv2-S combines locality modeling with long-range dependency modeling.
  • Analysis: RSP helps most on smaller datasets, while abundant training data can expose the representation capacity of deeper models.The study also notes that multilayer features and longer pretraining may further improve the evaluated RSP backbones.
  • Qualitative analysis: Response maps show RSP shifting attention toward semantically important targets, foreground objects, and related background regions.RSP-ViTAEv2-S particularly highlights airplanes and recognizes natural scenes such as terraces, mountains, and rivers.

B. Aerial Semantic Segmentation

The semantic-segmentation evaluation covers scene parsing on Potsdam and object segmentation on iSAID. These datasets test pixel-level labeling across whole scenes and aerial foreground objects.

  • Task and datasets: Semantic segmentation is evaluated as pixel-level classification through scene parsing and object segmentation subtasks.Scene parsing labels every pixel in a scene, whereas object segmentation emphasizes foreground objects.
  • Task and datasets: Potsdam supplies the scene-parsing testbed with six categories across 38 high-resolution images.The dataset covers impervious surface, building, low vegetation, tree, car, and clutter.
  • Task and datasets: iSAID supplies the object-segmentation testbed with 15 foreground categories and one background category across 2,806 high-resolution images.Only its validation set is used because the testing set is unavailable.
  • Task and datasets: Visual samples compare the MillionAID, Potsdam, and iSAID datasets.The figure provides a qualitative view of the image domains used in pretraining and segmentation evaluation.
  • Implementation detail: The segmentation experiments use standard backbone-specific optimization settings, with SGDM for ResNet models and AdamW for vision transformers.The learning-rate schedules and optimizer hyperparameters differ between the CNN and transformer families.

2) Implementation Detail and Experimental Setting:

Segmentation performance improves with stronger backbones, but the benefit of remote-sensing pretraining is task- and dataset-dependent. ImageNet initialization remains stronger for the evaluated segmentation setting, while RSP captures selected remote-sensing semantics.

  • Quantitative results: Changing the backbone from ResNet-50 to Swin-T and then ViTAEv2-S increases segmentation performance.This follows the scene-recognition pattern and is attributed to stronger vision-transformer representations.
  • Potsdam: ViTAEv2-S obtains the highest OA on Potsdam, although its mF1 remains below LANet.The selected models also underperform other methods on the Car category.
  • iSAID: IMP-ViTAEv2-S achieves the best performance on iSAID and outperforms several specialized aerial-segmentation methods.IMP-Swin-T is also competitive on this dataset.
  • Cross-task analysis: RSP models show an advantage in the Bridge category, consistent with their scene-recognition behavior.This indicates that some remote-sensing semantics transfer across recognition and segmentation evaluations.
  • Limitations of RSP: RSP performs worse than classical IMP on segmentation, partly because of dataset volume, spectral differences, and task differences.Scene recognition favors global understanding, whereas segmentation requires detailed features with high-level semantics.
  • Qualitative results: Only ViTAEv2-S successfully connects long-strip low vegetation in the Potsdam visualizations.IMP-ViTAEv2-S performs slightly better than RSP-ViTAEv2-S, matching the quantitative results.

C. Aerial Object Detection

The object-detection experiments address oriented bounding-box detection in aerial imagery. They evaluate multi-category detection on DOTA and single-category ship detection on HRSC2016 using ORCN with different backbones and pretrained weights.

  • Task and datasets: Aerial detection uses oriented bounding boxes because top-down objects may appear in arbitrary directions.This distinguishes the task from horizontal bounding-box detection commonly used for natural images.
  • Task and datasets: DOTA evaluates multi-category oriented detection, while HRSC2016 evaluates single-category ship detection.The two datasets are used for complementary detection objectives.
  • DOTA: DOTA contains 2,806 images and 188,282 instances across 15 categories.Its training, validation, and testing splits contain 1,411, 458, and 937 tiles, respectively.
  • HRSC2016: HRSC2016 contains 1,061 images with arbitrarily oriented bounding boxes for one ship category.The official split includes 436 training, 181 validation, and 444 testing images.
  • Implementation detail: ResNet detection models use SGDM, whereas vision transformers use AdamW, with training lasting 12 epochs on DOTA and 36 on HRSC2016.Learning rates and weight decay settings differ between the backbone families.
  • Evaluation: The detection study reports mean average precision across categories and average precision for each class on the corresponding test sets.All models are trained on a single V100 GPU.

3) Experimental Results:

On DOTA object detection, ViTAEv2-S performs strongly, and remote sensing pretraining generally surpasses ImageNet pretraining, especially for bridges and aerial vehicles.

  • Nearly 2% mAP: ViTAEv2-S improves the ORCN baseline on DOTA object detection.
  • RSP weights outperform their ImageNet pretrained counterparts across ResNet-50, Swin-T, and ViTAEv2-S detection backbones.
  • RSP-ViTAEv2-S achieves higher mAP than IMP-ViTAEv2-S through advantages on Bridge, Helicopter, and Plane categories.IMP remains more universal, with higher APs on most individual categories.
  • Visualizations show RSP-ViTAEv2-S handles densely distributed objects and detects long bridges that IMP-ViTAEv2-S misses or misclassifies.
  • Change detection compares paired aerial images as a pixel-level binary classification task, using CDD and LEVIR datasets.

1) Dataset:

The change-detection evaluation uses CDD and LEVIR with the BIT framework, comparing different pretrained CNN and vision-transformer backbones.

  • CDD contains seasonal, multisource imagery clipped into 256 × 256 patches, with 10,000/3,000/3,000 training, validation, and testing patches.Its source images range from 0.03m to 1m resolution.
  • LEVIR contains 637 high-resolution image pairs focused mainly on man-made building changes, split into 445/64/128 training, validation, and testing pairs.
  • ViTAEv2-S pretrained with either IMP or RSP outperforms SeCo-ResNet-50 on change detection, while ViTAEv2-S achieves the best performance overall.
  • The required change-detection representation lies between segmentation and detection because the task is binary segmentation without specific object-category recognition.
  • ViTAEv2-S captures road details and reduces object adhesion in artificial-change scenes more effectively than the other evaluated models.

E. Overall Comparison of Different Backbones on All Tasks

Across four downstream tasks, longer pretraining generally strengthens representations, while ViTAEv2-S consistently outperforms ResNet-50 and Swin-T; task discrepancy still affects outcomes.

  • E. Overall Comparison of Different Backbones on All Tasks: ViTAEv2-S outperforms ResNet-50 and Swin-T on all downstream tasks when trained with more epochs.
  • E. Overall Comparison of Different Backbones on All Tasks: More pretraining epochs generally improve downstream performance, although Swin-T on object detection is an exception.The comparison averages scores across datasets for each task.
  • E. Overall Comparison of Different Backbones on All Tasks: The study evaluates CNN and vision-transformer RSP backbones across scene recognition, semantic segmentation, object detection, and change detection.
  • E. Overall Comparison of Different Backbones on All Tasks: Vision transformers perform competitively or better than CNNs, with ViTAEv2-S achieving the best performance on almost all settings.
  • E. Overall Comparison of Different Backbones on All Tasks: IMP provides universal representations, whereas RSP is comparable overall and especially strong for categories such as Bridge and Airplane.
  • E. Overall Comparison of Different Backbones on All Tasks: Task-level discrepancy can reduce RSP benefits when downstream tasks require representation granularity unlike scene recognition.
  • E. Overall Comparison of Different Backbones on All Tasks: Future work will examine RSP on larger downstream datasets and unsupervised pretraining using abundant unlabeled remote-sensing data.
Loading 2204.02825v4…