PaperlayerView exact revision on arXiv

Source-linked AI summary

Advancing Plain Vision Transformer Towards Remote Sensing Foundation Model

Di Wang, Qiming Zhang, Yufei Xu, Jing Zhang, Bo Du, Dacheng Tao, Liangpei Zhang

arXiv:2208.03987v4cs.CV

TL;DR

Large-scale models remain underexplored for remote sensing, where high-resolution images contain arbitrarily oriented objects and full attention is costly. The paper scales plain ViTs, pretrains them with MAE, and introduces RVSA to use efficient, oriented varied-size windows. The models achieve state-of-the-art detection and competitive classification and segmentation results, with reported advantages in computational complexity and transfer data efficiency.

  • Problem

    Large-scale vision models have been insufficiently explored for remote sensing, while full attention limits plain ViTs on high-resolution images and fixed windows restrict contextual representation.

  • Method

    The paper pretrains approximately 100-million-parameter plain ViTs with MAE and replaces full attention with rotated varied-size windows for downstream RS tasks.

  • Results

    The models set state-of-the-art results on detection, achieve competitive classification and segmentation performance, and show advantages in computational complexity and transfer data efficiency.

  • Takeaways & Limitations

    Plain ViTs with MAE pretraining and RVSA provide a competitive route toward remote-sensing foundation models across detection, classification, and segmentation.

Abstract

from arXiv · show

Large-scale vision foundation models have made significant progress in visual tasks on natural images, with vision transformers being the primary choice due to their good scalability and representation ability. However, large-scale models in remote sensing (RS) have not yet been sufficiently explored. In this paper, we resort to plain vision transformers with about 100 million parameters and make the first attempt to propose large vision models tailored to RS tasks and investigate how such large models perform. To handle the large sizes and objects of arbitrary orientations in RS images, we propose a new rotated varied-size window attention to replace the original full attention in transformers, which can significantly reduce the computational cost and memory footprint while learning better object representation by extracting rich context from the generated diverse windows. Experiments on detection tasks show the superiority of our model over all state-of-the-art models, achieving 81.24% mAP on the DOTA-V1.0 dataset. The results of our models on downstream classification and segmentation tasks also show competitive performance compared to existing advanced methods. Further experiments show the advantages of our models in terms of computational complexity and data efficiency in transferring.

I. INTRODUCTION

The paper investigates whether non-hierarchical plain ViTs can serve as large RS models, addressing the challenge of representing arbitrary-oriented objects efficiently. It combines large-scale MAE pretraining with rotated varied-size attention and reports strong results across RS tasks.

  • Motivation: RSIs support earth observation and applications including scene recognition, land-use classification, precision agriculture, and maritime object detection.Their accessibility, real-time availability, and abundant geospatial and spectral information motivate effective representation learning.
  • Motivation: Existing CNNs struggle to capture long-range pixels and global context because convolution has a limited receptive field.Self-attention addresses this limitation by enabling interactions between arbitrary image pixels.
  • Research question: The paper tests whether hierarchical structure is necessary by applying non-hierarchical plain ViTs to RS tasks with masked-image-modeling pretraining.The approach is motivated by evidence that multiscale priors can be learned during pretraining and by the abundance of unlabeled RSIs.
  • Method: RVSA extends varied-size attention with learnable rotation, generating windows with different angles, sizes, shapes, and locations for arbitrary-oriented RSI objects.The generated windows extract richer context while replacing computationally expensive full attention during downstream adaptation.
  • Method: The study pretrains approximately 100-million-parameter plain ViTs, adapts them to downstream RS tasks, and positions them as a step toward RS foundation models.The models are evaluated through downstream finetuning and are described as delivering competitive performance.
  • Results: The proposed models achieve state-of-the-art detection performance and competitive classification and segmentation performance, while showing advantages in computational complexity and transfer data efficiency.These claims summarize the reported results across the three evaluated task types.

II. RELATED WORK

Related work establishes the scalability of transformer-based vision models, the efficiency of window attention, and the importance of RS-specific pretraining. The paper builds on these directions by combining MAE pretraining with RVSA for plain ViTs.

  • Vision foundation models: Plain ViTs offer simple, flexible architectures that can scale through repeated transformer layers and widened dimensions.Transformer foundation models have expanded in vision because of their scalability and structural flexibility.
  • Window-based vision transformers: Full attention has quadratic complexity over image size, whereas window-based attention provides linear complexity for high-resolution images.Window partitioning reduces memory demands but can constrain context when window sizes and locations are fixed.
  • Remote-sensing pretraining: RS pretraining faces a domain gap between natural images and RSIs, motivating self-supervised methods that incorporate RS characteristics.Earlier approaches include seasonal, spatial, and temporal information, while RingMo targets large-scale vision transformers with SimMiM.
  • Paper approach: The paper uses MAE pretraining on MillionAID and transfers the pretrained models to scene classification, object detection, and semantic segmentation.The method section frames MAE, RVSA, and downstream transfer as the core components of the experimental pipeline.

1) MillionAID:

MillionAID supplies large-scale, variable-resolution remote-sensing imagery for unsupervised MAE pretraining. The paper uses plain ViT and ViTAE backbones, adapting their components to token-sequence pretraining.

  • MillionAID: MillionAID contains 1,000,848 non-overlapping RGB remote-sensing scenes across 51 classes, with image sizes ranging from 110 × 110 to 31,672 × 31,672 pixels.Although labels exist, the pretraining routine discards them and uses only images.
  • MAE: MAE masks patch-derived visual tokens, encodes the visible tokens, and decodes them to reconstruct masked regions using normalized-pixel loss.The encoder receives visible tokens after masking, while learnable mask tokens are introduced in the decoder.
  • Pretrained Backbone Networks: During ViTAE pretraining, the convolutional branch uses 1×1 convolutions to avoid misleading inductive bias caused by MAE's random masking.The shallow PCM contains convolution, batch normalization, SiLU, and convolution layers to reduce memory footprint.
  • Pretrained Backbone Networks: The network adds positional encoding after patch embedding, and classification averages final output tokens with global average pooling.The figure identifies “P” as positional encoding and GAP as the averaging operation for classification.
  • Pretrained Backbone Networks: The pretrained backbones are ViT-B and ViTAE-B, each with about 100M parameters; ViT uses plain transformer encoders, whereas ViTAE adds convolutional branches.ViT directly treats images as 1-D token sequences, while ViTAE combines convolutional locality with full self-attention.
  • Implementation Details: Pretraining uses two MillionAID subsets containing 949,848 and 51,000 images, with the second subset class-balanced by sampling 1,000 images per category.Training uses batch size 2048 across 8 A100 GPUs with AdamW, typically for 1,600 epochs.

4) Implementation Details:

The implementation evaluates MAE pretraining and window-based attention for remote-sensing transfer. It replaces computationally expensive full attention with varied-size windows whose scales, offsets, and sampled features are learned from the input.

  • Implementation Details: The mask-ratio study evaluates ViT-B pretraining on UCM, AID, and NWPU using both linear probing and finetuning settings.Linear probing updates only a newly added classifier, whereas finetuning jointly updates the backbone and classifier.
  • Window-based attention: Full self-attention has quadratic complexity with image size, motivating window-based attention during downstream finetuning for high-resolution remote-sensing images.Window attention partitions features into local windows before calculating attention.
  • Window-based attention: Attention is computed separately within each window and head, then outputs are concatenated across heads and windows to recover the input feature shape.The sampled key and value token count matches the query token count, preserving computational complexity relative to window attention.
  • Varied-size window attention: Fixed-size windows restrict contextual coverage, so varied-size attention learns target-window scale and offset factors from globally pooled window features.The initial window is transformed using predicted scale and offset factors.
  • Varied-size window attention: The proposed RVSA method is illustrated as a pipeline operating in the i-th attention head.Its design combines learned window variation with attention over transformed windows.

2) Rotated Varied-Size Attention:

Rotated varied-size attention extends varied-size attention with a learnable window orientation to better match arbitrarily oriented remote-sensing objects. A flexible variant separately predicts window factors for key and value tokens.

  • Rotated Varied-Size Attention: RVSA adds a predicted rotation angle to varied-size attention, alongside scale and offset factors, for remote-sensing images.The angle is predicted from the input feature together with window scale and offset factors.
  • Rotated Varied-Size Attention: The rotation mechanism addresses the mismatch between arbitrarily oriented remote-sensing objects and horizontal or vertical attention windows.Both default windows and VSA-generated windows remain horizontal or vertical without this additional orientation dimension.
  • Rotated Varied-Size Attention: Figure 4 provides a diagram of the proposed RVSA method.The passage identifies the figure as a visual depiction of RVSA.
  • Rotated Varied-Size Attention: RVSA♦ separately predicts scale, shift, and rotation factors for key and value tokens, allowing them to sample from different windows.This design is presented as a more flexible variant of RVSA.

3) Computational Complexity Analysis:

RVSA partitions features into windows and predicts window transformations, adding only marginal computational cost compared with original window attention. The modified blocks replace MHSA with RVSA while retaining the surrounding block structure.

  • Computational Complexity Analysis: RVSA partitions the input into non-overlapping s × s windows and predicts each window’s scale, shift, and rotation factors from its features.Prediction uses global average pooling, an activation layer, and a linear layer.
  • Computational Complexity Analysis: In the modified ViT-B and ViTAE-B blocks, RVSA simply replaces MHSA.Figure 5 shows the corresponding structures for ViT-B + RVSA and ViTAE-B + RVSA.
  • Computational Complexity Analysis: RVSA adds about 11% computational cost over original window attention, whose complexity is O(2s2HWC).The comparison assumes s = 7 and h = 12, while RVSA’s extra complexity is given as O(5HWC(1 + 5h/s2)).

4) Implementation Details:

The implementation pretrains plain ViTs with MAE on MillionAID, replaces selected MHSA blocks with RVSA for transfer, and evaluates the resulting models across multiple remote-sensing tasks.

  • Implementation Details: For 12-layer ViT-B and ViTAE-B models, full attention is used at layers 3, 6, 9, and 12, while RVSA is used elsewhere.The modified networks are named ViT-B + RVSA and ViTAE-B + RVSA.
  • Implementation Details: Plain ViTs are unsupervised pretrained with MAE on MillionAID by reconstructing masked image patches.The pretraining dataset is a large-scale remote-sensing dataset.
  • Implementation Details: After pretraining, selected MHSA blocks are replaced with window-attention modules, using RVSA as an example, before downstream finetuning.The resulting networks are transferred and finetuned on different remote-sensing tasks.
  • Implementation Details: The experiments evaluate scene classification, object detection, and semantic segmentation, including ablations and comparisons with state-of-the-art methods.The evaluation uses existing popular frameworks and public benchmarks.

A. Remote Sensing Object Detection

Remote-sensing object detection is evaluated on DOTA-V1.0 and DIOR-R using oriented bounding boxes, standardized training settings, and Oriented R-CNN with modified backbones. Results report class AP and overall mAP, with window-attention variants included in the comparisons.

  • Dataset: DOTA-V1.0 contains 2,806 images, 188,282 annotated instances, and 15 categories for oriented bounding-box detection.Its images range from 800 × 800 to 4,000 × 4,000 pixels, with separate training, validation, and testing splits.
  • Dataset: DIOR-R contains 23,463 images, 192,518 instances, and 20 common object categories for oriented bounding-box detection.All images are cropped to 800 × 800, with pixel resolutions ranging from 0.5 to 30m.
  • Implementation: Models use AdamW with learning rate 10^-4, weight decay 0.05, a 12-epoch schedule, batch size 2, and layer-wise decay 0.75.The learning rate is reduced by 10× at epochs 8 and 11.
  • Implementation: The comparison uses the same Oriented R-CNN framework while changing only the backbone, with feature pyramids built from the final-layer output.The feature is separately upsampled and downsampled through deconvolution and pooling layers.
  • Evaluation: DOTA images are cropped to 1,024 × 1,024 patches for single-scale and multiscale training and testing, with scales of 0.5, 1.0, and 1.5 in the multiscale setting.The reported evaluation uses average precision for each class and mean average precision overall.
  • Ablation Studies: The ablation tables compare mAP across window sizes and ViT-B variants using window, scale-factor, offset-factor, and rotation-factor configurations.The table captions define WA, SF, OF, and RF as the corresponding attention and factor abbreviations.

3) Determining the Suitable Window Size:

The experiments identify s = 7 as a balance between receptive-field context and window diversity. RVSA improves representation of arbitrarily oriented remote-sensing objects while reducing the costs associated with full attention.

  • Determining the Suitable Window Size: s = 7 provides a good balance between attended-token quantity and window diversity on DOTA-V1.0 and DIOR-R.Larger windows increase context but reduce window diversity; smaller windows attend to fewer tokens.
  • Comparison of Attention Methods: RVSA adds learnable rotation to VSA, generating windows in varied directions that better fit overhead objects and extract suitable context.RVSA outperforms VSA on both DOTA-V1.0 and DIOR-R.
  • Comparison Against State-of-the-art Methods: 81.24% mAP establishes a new state of the art on DOTA-V1.0, outperforming previous methods.The gains include improved results in challenging roundabout and harbor categories.
  • Comparison Against State-of-the-art Methods: RVSA models outperform corresponding ViT-B and ViTAE-B baselines and achieve higher accuracy than VSA models on DOTA-V1.0.The additional rotation mechanism is identified as the reason for the improvement over VSA.
  • Qualitative Detection Results: RVSA-equipped plain ViTs accurately recognize and localize diverse objects across city and rural scenes with dense or sparse distributions.The qualitative examples use ViTAE-B + RVSA on DOTA-V1.0 and ViTAE-B + RVSA♦ on DIOR-R.

B. Remote Sensing Scene Classification

The models are evaluated across five remote-sensing scene-classification settings using pretrained backbones and top-1 accuracy. RVSA is strongest in several settings, while limited training data can constrain its learned window configurations.

  • Experimental Setup: Five settings—UCM-55, AID-28, AID-55, NWPU-19, and NWPU-28—are used to evaluate scene classification with top-1 accuracy.The suffix −mn denotes that 10 × m% of samples are used for training.
  • Results: RVSA models outperform previous methods in UCM-55, AID-28, and NWPU-28.The comparison includes conventional CNN backbones and advanced pretraining-based vision transformers.
  • Results: The models remain competitive with RSP-ViTAEv2-S in settings where they do not achieve the best performance.RSP-ViTAEv2-S is pretrained on MillionAID.
  • Data Dependence: NWPU-19 is the main weaker setting because RVSA needs sufficient training data to learn optimal window configurations.With the larger NWPU-28 training set, the models can surpass RSP-ViTAEv2-S and achieve comparable performance.

C. Remote Sensing Semantic Segmentation

The plain-ViT models with RVSA achieve performance comparable to state-of-the-art methods for semantic segmentation. Their results are less impressive than on detection and classification, partly because the segmentation framework and plain architecture limit high-resolution detail handling.

  • Experimental Setup: The study transfers the pretrained models to Potsdam2, iSAID, and LoveDA semantic-segmentation datasets.LoveDA is used as a domain-adaptation segmentation dataset without distinguishing urban and rural areas.
  • Experimental Setup: UperNet forms the segmentation framework, using outputs from blocks 4, 6, 8, and 12 to build a feature pyramid.Training iterations are increased to 160k for the larger models.
  • Results: The proposed models obtain performance comparable to state-of-the-art semantic-segmentation methods.RVSA advances plain ViTs to performance similar to RSP-ViTAEv2-S.
  • Limitations: Segmentation performance is less impressive than detection and scene classification performance.The paper attributes this partly to UperNet’s limited propagation of high-level semantics to high-resolution features.
  • Limitations: Plain ViTs may lose spatial details because their feature-map resolution remains 1/16 of the input size.This contrasts with hierarchical backbones that generate higher-resolution feature maps at earlier stages.

E. Finetuning on Fewer Samples

The transfer experiments test performance with 20%–80% of DIOR-R training images and visualize learned attention windows. RVSA improves data efficiency, while RVSA♦ benefits more from larger training sets and provides greater window flexibility.

  • Finetuning on Fewer Samples: The models outperform corresponding ViT-B and ViTAE-B baselines regardless of whether 20%, 40%, 60%, or 80% of DIOR-R training images are used.The models are finetuned on randomly selected subsets and evaluated on the original test set.
  • Finetuning on Fewer Samples: RVSA outperforms VSA in most transfer settings because its learnable rotation mechanism addresses oriented objects in remote-sensing images.RVSA♦ requires more training samples than RVSA to perform better.
  • Visualization of RVSA: RVSA generates windows that adapt to arbitrary-oriented objects, with different attention heads producing distinct window sets.The visualization compares window attention, VSA, RVSA, and RVSA♦.
  • Visualization of RVSA: RVSA♦ further increases flexibility by sampling key and value tokens from slightly different window shapes.This flexibility may help with challenging samples when more training data is available.
Loading 2208.03987v4…