Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images

Haifeng Li; Yi Li; Guo Zhang; Ruoyun Liu; Haozhe Huang; Qing Zhu; Chao Tao

Source-linked AI summary

Global and Local Contrastive Self-Supervised Learning for Semantic Segmentation of HR Remote Sensing Images

Haifeng Li, Yi Li, Guo Zhang, Ruoyun Liu, Haozhe Huang, Qing Zhu, Chao Tao

arXiv:2106.10605v2Jun 20, 2021cs.CV

TL;DR

Remote-sensing semantic segmentation is limited by scarce, costly pixel-level labels, while standard contrastive learning may provide inadequate image-level features for pixel-level tasks. GLCNet combines global style and local matching contrastive learning, and the paper reports better performance than competing self-supervised and ImageNet-pretrained approaches, including under dataset differences.

Problem
Remote-sensing semantic segmentation requires many costly labels, and existing contrastive learning methods mainly learn image-level representations that may be suboptimal for pixel-level discrimination.
Method
GLCNet combines global style contrastive learning for overall image representation with local matching contrastive learning for local-region representation.
Results
The method mostly outperforms traditional ImageNet pre-training and other self-supervised methods, and outperforms supervised learning when upstream and downstream datasets differ.
Takeaways & Limitations
Learning from abundant unlabeled remote-sensing imagery can support semantic segmentation with limited annotations and may help develop general models for local-area transfer.
Takeaways & Limitations
Temporal invariance is only simulated with random color and texture enhancements because real multitemporal imagery is unavailable, so complex seasonal and imaging-condition changes may not be sufficiently learned.

Abstract

from arXiv · show

Supervised learning for semantic segmentation requires a large number of labeled samples, which is difficult to obtain in the field of remote sensing. Self-supervised learning (SSL), can be used to solve such problems by pre-training a general model with a large number of unlabeled images and then fine-tuning it on a downstream task with very few labeled samples. Contrastive learning is a typical method of SSL that can learn general invariant features. However, most existing contrastive learning methods are designed for classification tasks to obtain an image-level representation, which may be suboptimal for semantic segmentation tasks requiring pixel-level discrimination. Therefore, we propose a global style and local matching contrastive learning network (GLCNet) for remote sensing image semantic segmentation. Specifically, 1) the global style contrastive learning module is used to better learn an image-level representation, as we consider that style features can better represent the overall image features. 2) The local features matching contrastive learning module is designed to learn representations of local regions, which is beneficial for semantic segmentation. The experimental results show that our method mostly outperforms SOTA self-supervised methods and the ImageNet pre-training method. Specifically, with 1\% annotation from the original dataset, our approach improves Kappa by 6\% on the ISPRS Potsdam dataset relative to the existing baseline. Moreover, our method outperforms supervised learning methods when there are some differences between the datasets of upstream tasks and downstream tasks. Since SSL could directly learn the essential characteristics of data from unlabeled data, which is easy to obtain in the remote sensing field, this may be of great significance for tasks such as global mapping. The source code is available at https://github.com/GeoX-Lab/G-RSIM.

I. INTRODUCTION

Remote-sensing semantic segmentation needs costly pixel-level labels, while unlabeled imagery is abundant. The paper introduces GLCNet to learn global and local representations through self-supervised contrastive learning, improving limited-label segmentation.

I. INTRODUCTION: Remote-sensing semantic segmentation supports applications including urban planning, disaster monitoring, environmental protection, and agricultural management.The paper presents semantic segmentation as an important and challenging pixel-level interpretation task.
I. INTRODUCTION: Existing instance-wise contrastive learning is designed mainly for image-level representations, making it potentially suboptimal for semantic segmentation's pixel-level discrimination.The paper identifies this representation mismatch as a central motivation for GLCNet.
I. INTRODUCTION: Self-supervised learning transfers knowledge learned from diverse unlabeled remote-sensing images to downstream segmentation with limited annotations.This addresses the cost and scarcity of globally diverse pixel-level labels.
I. INTRODUCTION: GLCNet balances global and local feature learning for remote-sensing semantic segmentation by combining global style and local matching contrastive modules.The framework targets the mismatch between image-level contrastive representations and pixel-level segmentation needs.
I. INTRODUCTION: The method is evaluated on two public and two realistic datasets, where it outperforms other self-supervised methods and supervised learning when upstream and downstream data differ.The reported comparison covers both self-supervised baselines and supervised learning under dataset dissimilarity.

II. METHOD

The method adapts contrastive learning to remote-sensing segmentation by combining global style representations with local matching and transformation-based invariance learning. Its design addresses the loss of local information in image-level contrastive approaches.

A. Overview: GLCNet combines global style contrastive learning with local matching contrastive learning to balance whole-image and local-region representations.The architecture is designed specifically for semantic segmentation rather than only image-level representation learning.
A. Overview: Global style learning uses style features because global average-pooled features may not adequately represent complex remote-sensing images.Style features are intended to better capture overall image characteristics.
A. Overview: Local matching learning preserves information from diverse land-cover regions that whole-image features can lose in pixel-level segmentation.The module responds to the richness of land-cover categories within individual images.
B. Contrastive learning: Contrastive learning trains representations by making positive pairs similar and negative pairs dissimilar, with augmented views forming positives and other samples serving as negatives.This instance-wise formulation encourages transformation invariance and discrimination between samples.
B. Contrastive learning: The pretext task uses spatial transformations such as cropping, resizing, flipping, and rotation, plus color distortion, blur, and noise to simulate temporal variation.The augmentations target spatial and temporal invariance in remote-sensing imagery.

1) Data augmentation:

GLCNet generates two augmented views to learn spatial and temporal invariance, then extracts encoder features, projects them through an MLP, and applies contrastive learning to positive and negative pairs.

1) Data augmentation:: Two augmented views are generated using spatial transformations and simulated temporal transformations to encourage general spatiotemporal invariance features.Spatial operations include cropping, resizing, flipping, and rotation; temporal transformations include color distortion, blur, and noise.
1) Data augmentation:: Global features are extracted from augmented samples with the encoder, using channel-wise mean features as global average pooling.The encoder is the DeepLabV3+ semantic-segmentation encoder.
1) Data augmentation:: The projection head is a one-hidden-layer ReLU MLP applied after feature extraction.The projection head is intended to preserve useful information for downstream tasks.
1) Data augmentation:: Contrastive loss makes augmented views from the same sample similar while making views from different samples dissimilar.Each minibatch produces positive pairs from the same sample and negative samples from other samples.
1) Data augmentation:: Existing instance-wise contrastive learning may learn image-level representations that are suboptimal for semantic segmentation requiring pixel-level discrimination.The section identifies this representation mismatch as a central motivation for the proposed design.

C. Global style and Local matching Contrastive learning Network (GLCNet)

GLCNet combines global style contrastive learning with local matching contrastive learning, replacing simple average-pooled features with style features based on channel-wise means and variances.

C. Global style and Local matching Contrastive learning Network (GLCNet): GLCNet contains global style and local matching contrastive learning modules to address representation needs in complex remote sensing segmentation.The global module targets inadequate global average-pooling representations, while the local module targets classification-oriented image-level representations.
C. Global style and Local matching Contrastive learning Network (GLCNet): Global style contrastive learning treats augmented views of one sample as similar and views from other samples as dissimilar.This follows the instance-wise contrastive learning objective while changing the feature representation.
C. Global style and Local matching Contrastive learning Network (GLCNet): The global style feature vector concatenates channel-wise means and variances from encoder feature maps.The paper uses these statistics as a representation of image style and overall image features.
C. Global style and Local matching Contrastive learning Network (GLCNet): The style representation is motivated as more representative of overall image features than simple average-pooling features.This design specifically addresses the complexity of remote sensing images.

2) Local matching contrastive learning:

The local matching module selects corresponding regions across augmented views and contrasts their local representations, preserving information needed for pixel-level semantic discrimination.

2) Local matching contrastive learning:: Local matching contrastive learning addresses rich within-image land-cover categories and the limitations of image-level representations for pixel-level segmentation.Using only global features can lose useful local information, while instance-wise learning may be suboptimal for semantic segmentation.
2) Local matching contrastive learning:: Figure 5 schematically depicts the local-area selection and matching process.The figure corresponds to selecting and matching regions across the two transformed views.
2) Local matching contrastive learning:: Matching local regions are selected from two transformed views by tracking corresponding original-image positions despite cropping, flipping, and rotation.The procedure is repeated to obtain multiple matched local regions.
2) Local matching contrastive learning:: The method extracts local feature maps from the DeepLabV3+ encoder-decoder outputs for each pair of matched regions.The local regions are obtained from feature maps generated by the encoder-decoder network.
2) Local matching contrastive learning:: Local matching loss makes representations of matched regions similar and representations of different local regions dissimilar.The loss uses all selected local regions in the minibatch, excluding the matched positive pair from the negatives.

3) Total loss:

GLCNet combines global and local contrastive objectives in a total loss, using global learning for image-level information and local learning for region-level representations.

3) Total loss:: Global and local contrastive learning are combined because each captures information needed for semantic segmentation.Global learning captures image-level information, while local learning focuses on local-region representations.
3) Total loss:: Using only global learning produces representations that are not optimal for segmentation, whereas using only local learning can over-distinguish regions from the same category.The combined objective balances these two failure modes.
3) Total loss:: The global loss updates only the encoder, while the local loss updates both the encoder and decoder.The loss weight λ is fixed at 0.5 in this implementation.
3) Total loss:: The pre-training algorithm requires images, network structures, augmentations, batch size, temperature, loss weight, and local-region parameters.It then extracts global style features, obtains local matching feature maps, computes both losses, and updates the networks.
3) Total loss:: The total loss is computed from global style and local matching losses and then minimized by updating the network components.The algorithm computes both losses before updating the encoder, decoder, and projection heads.

III. EXPERIMENTS AND RESULTS

The experiments evaluate GLCNet and competing self-supervised methods across four remote sensing semantic segmentation datasets with limited annotations.

III. EXPERIMENTS AND RESULTS: Four datasets are used: ISPRS Potsdam and DeepGlobe are public, while Hubei and Xiangtan enable analysis of domain differences.The Hubei and Xiangtan datasets share spatial resolution and similar classification systems.
III. EXPERIMENTS AND RESULTS: ISPRS Potsdam contains 38 high-resolution aerial images with 5 cm resolution, four spectral bands, and six annotated classes.The downstream training set uses 1% of the self-supervised training data by default.
III. EXPERIMENTS AND RESULTS: DeepGlobe provides sub-meter satellite images with seven imperfectly labeled land-cover classes and uses separate training and downstream testing image sets.Images are cropped to 512×512 pixels.
III. EXPERIMENTS AND RESULTS: Hubei imagery comes from Gaofen-2 and has 2 m resolution, three RGB bands, and ten classes with poor-quality labels.

4) Xiangtan Dataset:

The Xiangtan dataset is used alongside several baselines and evaluation metrics to assess semantic segmentation with limited labels.

4) Xiangtan Dataset:: Xiangtan imagery comes from Gaofen-2, covers Xiangtan, and provides nine classes with higher-quality labels than Hubei.The city is divided into 4096×4096 patches, with 85 patches for training and 21 for testing.
1) Baseline:: The evaluation compares random initialization, ImageNet pre-training, Jigsaw, Inpainting, SimCLR, and MoCo v2.
1) Baseline:: Jigsaw shuffles image patches for contextual learning, whereas Inpainting reconstructs discarded image regions.
2) Evaluation Metrics:: Overall accuracy is measured with OA and Kappa, while F1-score measures accuracy for individual categories.
3) Implementation Details:: Self-supervised pre-training runs for 400 epochs with Adam; GLCNet samples four local 48×48 regions per image.

3) Implementation Details:

GLCNet is evaluated against multiple pre-training strategies on four segmentation tasks, with limited labeled data used for fine-tuning.

1) Comparison with other methods:: GLCNet achieves state-of-the-art results across the four datasets and improves over no pre-training, while some inappropriate self-supervised methods hurt performance.Visualization results also show relatively better overall performance for GLCNet.
1) Comparison with other methods:: GLCNet outperforms other self-supervised methods in most classes across multiple datasets.
1) Comparison with other methods:: GLCNet outperforms ImageNet pre-training on most datasets despite ImageNet using millions of supervised images.
1) Comparison with other methods:: The advantage over ImageNet is especially evident on Potsdam, possibly because its four bands differ most from RGB natural imagery.
2) Effect of the amount of self-supervised data:: With 20%, 50%, and 100% of self-supervised data, both Potsdam and Xiangtan show an overall performance increase as data grows.The improvement is relatively more obvious for GLCNet than SimCLR.

2) Effect of the amount of self-supervised data:

Domain similarity between self-supervised pre-training data and downstream data strongly affects transfer performance.

3) Effect of the domain difference:: More similar self-supervised data produces better downstream performance, and GLCNet mostly outperforms supervised learning when domains differ.The exception occurs when domain differences are extremely small, such as Hubei→Xiangtan and Xiangtan→Hubei.
3) Effect of the domain difference:: Mixing many dissimilar pre-training images may fail to improve performance or even damage it.
4) Ablation Study:: The ablation experiments examine GLCNet modules, decoder parameters, and the loss-weight λ.

1) The effectiveness of the modules of the proposed GLCNet:

Ablations show that the complete GLCNet performs best overall, with local matching generally providing the largest benefit and global style remaining important on some datasets.

The complete GLCNet achieves optimal performance, while each module benefits most experiments.
Local matching improves performance over its removal on most datasets, indicating the value of local differentiation for remote sensing segmentation.
On DGLC, removing the global module performs worst, likely because local-only learning over-distinguishes regions in images often containing one category.
Decoder parameters trained by GLCNet do not significantly improve fine-tuning, likely because local average pooling loses edge-localization details needed for decoder recovery.
The optimal λ varies across datasets, while using only local or only global contrastive learning causes sharp performance drops in most datasets.

IV. DISCUSSIONS

The discussion emphasizes that GLCNet benefits from task design and abundant unlabeled data, while domain mixing and local-region sampling remain important constraints.

GLCNet achieves optimal results, and increasing self-supervised data further improves fine-tuning accuracy.
Self-supervised learning can outperform supervised learning across differing datasets, but mixing strongly different domains can impair migration to a local dataset.
More homogeneous local-area selection may improve results because random sampling can favor dominant feature classes in heterogeneous images.
The method uses intrinsic signals from unlabeled images to learn spatio-temporal invariant features and reduce dependence on labeled samples.
The method does not yet fully model temporal variation because color and texture augmentation only approximates seasonal and imaging-condition changes.