PaperlayerView exact revision on arXiv

Source-linked AI summary

Looking Outside the Window: Wide-Context Transformer for the Semantic Segmentation of High-Resolution Remote Sensing Images

Lei Ding, Dong Lin, Shaofu Lin, Jing Zhang, Xiaojie Cui, Yuebin Wang, Hao Tang, Lorenzo Bruzzone

arXiv:2106.15754v6cs.CV

TL;DR

Cropped training inputs and CNN locality limit long-range context for semantic segmentation of high-resolution remote sensing images. WiCoNet addresses this with local and context branches connected by a Context Transformer, and experiments report accuracy improvements over FCN across three datasets. The paper also introduces the BLU benchmark dataset.

  • Problem

    Cropping and CNN locality limit long-range contextual information needed to segment LCLU classes in high-resolution remote sensing images.

  • Method

    WiCoNet uses local and context CNN branches, while a Context Transformer embeds wider contextual information and projects it onto local features.

  • Results

    WiCoNet improves over baseline FCN by 0.84%, 1.01%, and 1.41% in OA and 1.41%, 4.05, and 1.69% in mIoU on BLU, GID, and Potsdam, respectively.

  • Takeaways & Limitations

    Wide-context modeling improves the reported LCLU segmentation accuracy across the BLU, GID, and Potsdam datasets, while the BLU dataset supports future research.

Abstract

from arXiv · show

Long-range contextual information is crucial for the semantic segmentation of High-Resolution (HR) Remote Sensing Images (RSIs). However, image cropping operations, commonly used for training neural networks, limit the perception of long-range contexts in large RSIs. To overcome this limitation, we propose a Wide-Context Network (WiCoNet) for the semantic segmentation of HR RSIs. Apart from extracting local features with a conventional CNN, the WiCoNet has an extra context branch to aggregate information from a larger image area. Moreover, we introduce a Context Transformer to embed contextual information from the context branch and selectively project it onto the local features. The Context Transformer extends the Vision Transformer, an emerging kind of neural network, to model the dual-branch semantic correlations. It overcomes the locality limitation of CNNs and enables the WiCoNet to see the bigger picture before segmenting the land-cover/land-use (LCLU) classes. Ablation studies and comparative experiments conducted on several benchmark datasets demonstrate the effectiveness of the proposed method. In addition, we present a new Beijing Land-Use (BLU) dataset. This is a large-scale HR satellite dataset with high-quality and fine-grained reference labels, which can facilitate future studies in this field.

I. INTRODUCTION

Semantic segmentation of HR remote sensing images requires pixel-wise LCLU labeling, but cropped training inputs and CNN locality restrict access to long-range context. WiCoNet combines local CNN features with wider contextual modeling through dual branches and a Context Transformer.

  • I. INTRODUCTION: CNNs primarily model local patterns, while cropping HR remote sensing images further limits the long-range context available during training.The valid receptive field of CNN units can remain small, and cropped inputs constrain the maximum possible receptive field.
  • I. INTRODUCTION: WiCoNet combines CNNs for local spatial information with a transformer for modeling long-range dependencies in HR remote sensing segmentation.The approach aims to exploit complementary strengths of CNNs and transformers rather than using a plain transformer after a CNN.
  • I. INTRODUCTION: The proposed Context Transformer models dual-branch semantic dependencies and projects wider contextual information onto local features.It embeds local and contextual CNN features into tokens and learns correlations through repeated attention operations.
  • I. INTRODUCTION: The paper also presents the Beijing Land-Use dataset as a large-scale HR satellite benchmark with fine-grained land-use annotations.The dataset is intended to support future semantic-segmentation studies.

II. RELATED WORK

Related work develops CNN architectures, attention mechanisms, and transformers for semantic segmentation and remote sensing analysis. These approaches target local, multi-scale, geometric, non-local, or global semantic information through varied network designs.

  • II. RELATED WORK: CNN-based segmentation commonly uses encoder-decoder architectures, with modules for multi-scale context, feature fusion, and improved local representation.Examples include deformable and dilated convolutions, pyramid pooling, HRNet, RefineNet, and ExFuse.
  • II. RELATED WORK: Attention mechanisms augment CNN features by modeling channel, spatial, local, or non-local focus in remote sensing segmentation.The cited approaches integrate local and long-range attention in different branches or extend channel attention spatially.
  • II. RELATED WORK: Remote sensing segmentation methods emphasize spatial accuracy through U-shaped networks, context aggregation, edge supervision, and geometric modeling.Reported designs address object boundaries, multi-scale features, roads, buildings, and other ground-object structures.
  • II. RELATED WORK: Vision transformers model global relationships among image patches and have been applied to classification, detection, panoptic segmentation, and medical-image segmentation.Related architectures include ViT, DETR, dual-path transformers, and joint CNN-transformer designs.
  • II. RELATED WORK: Transformers have also been applied to remote sensing scene classification and change detection to enrich global semantic correlations.The cited change-detection method tokenizes and concatenates bi-temporal semantic features before transformer processing.

III. PROPOSED WIDE-CONTEXT NETWORK

WiCoNet addresses the limited long-range context available to CNNs on cropped HR RSIs by combining local and wider-context branches. A Context Transformer embeds context into local features so segmentation uses both fine detail and broader image information.

  • A. Motivation of the Wide-Context Modelling: Cropping limits a standard CNN’s maximum receptive field to the crop size, while its valid receptive field is usually smaller, leaving long-range context insufficiently exploited.This limitation is especially problematic for large or elongated LCLU objects whose geometry and semantic correlations exceed local windows.
  • B. Network Architecture: WiCoNet combines a local ResNet branch with a context CNN branch to model local details and wider-range information in HR RSIs.The context branch learns coarse environmental context, while the local branch remains the main segmentation pathway.
  • B. Network Architecture: A Context Transformer embeds information from the context branch into the local branch, and the context-enriched local features produce the final segmentation result.No additional feature fusion is performed because the context information has already been modeled through the transformer.
  • B. Network Architecture: The WiCoNet is trained with a local crop and a down-sampled surrounding image, producing segmentation maps for both local and context branches.The local crop corresponds to the central area of the larger context input.
  • B. Network Architecture: Training uses the total multi-class cross-entropy losses of the local and context branches, weighted by a parameter α.The losses use separate ground-truth maps for the local and context branches.

C. Context Transformer

The Context Transformer extends Vision Transformer attention to communicate between local and context tokens. It uses local features to query context features, repeatedly projecting long-range contextual dependencies onto local representations.

  • C. Context Transformer: The Context Transformer projects long-range contextual information onto local features using a Vision Transformer-based architecture.Its transformer blocks contain multi-head self-attention and MLP units with normalization and residual connections.
  • C. Context Transformer: The context attention matrix updates local tokens by calculating responses from all context-window features for each local position.This design projects a biased contextual focus rather than directly adding context values.
  • C. Context Transformer: Local tokens provide queries, while context tokens provide keys and values for cross-branch attention.The local and context token sequences are embedded from M1 and M2, respectively.
  • C. Context Transformer: The transformer represents attention projections with weights Wq, Wk, and Wv of size R^D×D/n.These weights define the corresponding query, key, and value projection functions.
  • C. Context Transformer: Repeated attention and MLP operations model dependencies between local and context tokens, after which both token sequences are reshaped into two-dimensional features.The contextual dependencies are repeated for L transformer blocks.

D. Implementation Details

The implementation uses ResNet50 for local feature extraction and a separate convolutional Context Encoder for coarse wider-context features. The context window is substantially larger than the local window, while transformer settings control token modeling capacity.

  • 1) The feature extraction networks: ResNet50 extracts local features at a ×1/8 down-sampling stride, while the Context Encoder uses eight convolutional and three max-pooling layers.The local stride is selected to better preserve spatial information.
  • 2) Area of the context modelling: The context window is set to nine times the local-window area, with w = 3w_l and h = 3h_l.The local window is 256 × 256, and empty border regions are padded when necessary.
  • 2) Area of the context modelling: The context branch receives input down-sampled by ×1/4, while its encoder uses a ×1/8 down-sampling stride.These settings define the spatial scale used for context modeling.
  • 3) Context Transformer: The Context Transformer uses patch size p = 1, token dimension D = 512, and configurable block and head counts L and n.Patch size one is used to retain spatial information, while L and n are selected through experiments.

IV. EXPERIMENTAL DATASETS AND SETTINGS

The experiments use a novel Beijing Land-Use dataset together with two open datasets, and report the associated experimental settings and evaluation metrics.

  • IV. EXPERIMENTAL DATASETS AND SETTINGS: The experimental section introduces the Beijing Land-Use dataset and two open datasets before describing settings and evaluation metrics.

A. Beijing Land-Use Dataset

The BLU dataset is a large-scale Beijing HR satellite benchmark with fine-grained labels across six land-use classes and diverse urban and rural scenes.

  • A. Beijing Land-Use Dataset: BLU contains RGB satellite imagery collected in Beijing at 0.8 m ground sampling distance, with fine-grained annotations for six land-use classes.The classes are background/barren, built-up, vegetation, water, agricultural land, and road.
  • A. Beijing Land-Use Dataset: The dataset provides pixel-level, geometrically optimized annotations produced by an experienced remote-sensing annotation team.The design targets local consistency and topological correctness.
  • A. Beijing Land-Use Dataset: BLU covers diverse scenes including farmland, residential areas, highways, airports, and wetlands, with varied examples within individual classes.For example, built-up areas include residential and industrial buildings, while water includes rivers, ponds, and wetlands.
  • A. Beijing Land-Use Dataset: The dataset comprises four large 15680 × 15680-pixel tiles covering around 150 km2, cropped into non-overlapping training, validation, and testing areas.Each tile yields 64 images of 2048 × 2048 pixels: 49 for training, 7 for validation, and 8 for testing.
  • A. Beijing Land-Use Dataset: The study also evaluates WiCoNet on the ISPRS Potsdam and Gaofen Image Dataset benchmarks.Potsdam contains six labeled urban land-cover categories, while GID contains 16 annotated land-cover classes.

C. Experimental Settings

Experiments use fixed optimization settings, standard segmentation metrics, and class-distribution reporting for the BLU benchmark.

  • C. Experimental Settings: Training uses 50 epochs, batch size 32, initial learning rate 0.1, SGD with momentum 0.9, and random flipping and cropping augmentation.The best model according to validation OA is saved.
  • C. Experimental Settings: The BLU class distribution is documented in Table I as part of the experimental dataset description.The supplied table passage identifies the distribution table but does not provide its cell values.
  • C. Experimental Settings: Evaluation reports Overall Accuracy, per-class F1 score, and mean Intersection over Union.F1 is the harmonic mean of precision and recall, while IoU is TP/(TP + FP + FN).

V. EXPERIMENTAL RESULTS

Experiments show that wide-context modeling improves segmentation accuracy across datasets, while context-window size affects performance and qualitative coherence.

  • A. Ablation Study: 0.84%, 1.01%, and 1.41% OA improvements over FCN, with corresponding mIoU gains of 1.41%, 4.05, and 1.69%, are reported on BLU, GID, and Potsdam.Adding a transformer only at the FCN encoder head yields limited improvements, whereas WiCoNet's wide-context modeling produces larger gains.
  • A. Ablation Study: WiCoNet improves discrimination of critical or minority classes by modeling contextual dependencies among similar samples in a wider window.The reported examples include challenging scenes in the BLU, GID, and Potsdam datasets.
  • A. Ablation Study: WiCoNet improves object connectivity and reduces fragmentation errors in qualitative segmentation results.Examples include roads, rivers, lakes, and empty fields where wider context clarifies spatial layout or suppresses false predictions.
  • A. Ablation Study: The best context-window area is dataset-dependent: ×9 performs best on BLU, whereas ×16 performs best on GID and Potsdam.The study tests windows covering ×4, ×9, and ×16 the local-window area.

B. Comparative Study

WiCoNet is compared with established context-aggregation models and shows the strongest reported accuracy across the three benchmark datasets, with moderate computational cost.

  • B. Comparative Study: WiCoNet obtains the best OA, mean F1, and mIoU among the compared methods on BLU, GID, and Potsdam.The comparison includes FCN, DeepLabv3+, PSPNet, DANet, SCAttNet, MSCA, and LANet, using averages from three trials.
  • B. Comparative Study: WiCoNet's improvements are particularly noticeable on GID, where wider contextual information is important for determining land-cover classes.This connects the comparative result to the dataset's context-sensitive classification setting.
  • B. Comparative Study: The ablation and hyper-parameter results are documented in Tables II and III, while qualitative ablation examples cover challenging BLU scenes.The supplied visual passages identify the table and figure scopes but do not provide unseen table values.
  • B. Comparative Study: WiCoNet has higher overall consumption than FCN, SCAttNet, and LANet but lower consumption than PSPNet and DANet.Its parameter size and FLOPS are reported as very close to DeepLabv3+ under the BLU experimental setting.

VI. CONCLUSIONS

WiCoNet addresses the limited long-range context of cropped-image segmentation by combining wider context aggregation with dual-branch contextual communication. Experiments across three datasets report significant accuracy improvements, while remaining global inconsistency and local fragmentation leave room for further improvement.

  • VI. CONCLUSIONS: An extra context branch aggregates information from larger context windows, broadening the model’s possible receptive fields beyond cropped local regions.This design targets the limitation that existing studies mainly model local context within cropped image patches.
  • VI. CONCLUSIONS: The Context Transformer communicates dual-branch information by calculating context and projecting it into local query tokens.This replaces simple feature fusion with contextual communication between the local and context branches.
  • VI. CONCLUSIONS: Context-window size is evaluated as a factor affecting segmentation accuracy.Table IV and Fig. 7 address the effects of context modeling range and different context-window sizes.
  • VI. CONCLUSIONS: WiCoNet brings significant accuracy improvements by modeling both local scene information and global class distribution across experiments on three datasets.The study evaluates long-range context modeling, context-window sizes, and comparisons with existing context-modeling methods.
  • VI. CONCLUSIONS: Global inconsistency and local fragmentation errors remain, showing that long-range context modeling in large RSIs still has room for improvement.The paper identifies adversarial learning strategies as a possible direction for future work.
Loading 2106.15754v6…