PaperlayerView exact revision on arXiv

Source-linked AI summary

Building extraction with vision transformer

Libo Wang, Shenghui Fang, Rui Li, Xiaoliang Meng

arXiv:2111.15637v2cs.CV

TL;DR

Accurate building extraction from fine-resolution remote sensing images remains difficult because CNNs lack global-context modelling, while Vision Transformers are resource-intensive and may lose spatial details. BuildFormer addresses these issues with dual context paths and linear windowed attention, achieving 75.74% IoU on the Massachusetts building dataset.

  • Problem

    Fine-resolution remote-sensing building extraction is challenged by complex backgrounds, diverse building appearances, ViT resource demands, and insufficient spatial-detail preservation.

  • Method

    BuildFormer combines global and spatial-detailed context paths with window-based linear multi-head self-attention and convolutional multilayer perceptrons.

  • Results

    75.74% IoU was achieved on the Massachusetts building dataset, outperforming CBRNet by 1.19%.

  • Takeaways & Limitations

    The dual-path design aggregates global context and spatial details to recognize difficult building pixels and maintain building integrity.

Abstract

from arXiv · show

As an important carrier of human productive activities, the extraction of buildings is not only essential for urban dynamic monitoring but also necessary for suburban construction inspection. Nowadays, accurate building extraction from remote sensing images remains a challenge due to the complex background and diverse appearances of buildings. The convolutional neural network (CNN) based building extraction methods, although increased the accuracy significantly, are criticized for their inability for modelling global dependencies. Thus, this paper applies the Vision Transformer for building extraction. However, the actual utilization of the Vision Transformer often comes with two limitations. First, the Vision Transformer requires more GPU memory and computational costs compared to CNNs. This limitation is further magnified when encountering large-sized inputs like fine-resolution remote sensing images. Second, spatial details are not sufficiently preserved during the feature extraction of the Vision Transformer, resulting in the inability for fine-grained building segmentation. To handle these issues, we propose a novel Vision Transformer (BuildFormer), with a dual-path structure. Specifically, we design a spatial-detailed context path to encode rich spatial details and a global context path to capture global dependencies. Besides, we develop a window-based linear multi-head self-attention to make the complexity of the multi-head self-attention linear with the window size, which strengthens the global context extraction by using large windows and greatly improves the potential of the Vision Transformer in processing large-sized remote sensing images. The proposed method yields state-of-the-art performance (75.74% IoU) on the Massachusetts building dataset. Code will be available.

I. INTRODUCTION

Building extraction supports several remote-sensing applications, but existing CNN and ViT approaches face complementary limitations: weak global-context modelling, high resource demands, and insufficient spatial-detail preservation. BuildFormer addresses these issues with dual context paths and linear windowed attention.

  • Building extraction identifies building and nonbuilding pixels in fine-resolution remote sensing images for urban planning, population statistics, economic assessment, and disaster management.
  • CNN-based methods capture local context effectively but lack global-context modelling, although global information can simplify ambiguous building-pixel identification.
  • Vision Transformers model global dependencies more strongly than CNNs but require substantial memory and computation, especially for large remote sensing inputs.
  • ViTs also insufficiently preserve spatial-detailed context, limiting fine-grained building segmentation in fine-resolution imagery.
  • BuildFormer uses a dual-path structure combining a global context path with a spatial-detailed context path for building extraction.
  • Its BuildFormer Block combines window-based linear multi-head self-attention with a convolutional multilayer perceptron, reducing attention complexity to linear and strengthening global information modelling.

II. RELATED WORK

Remote-sensing building extraction has progressed from hand-crafted features and CNNs toward Vision Transformers that better model global dependencies. BuildFormer is presented within this evolution as a Transformer-based architecture for the task.

  • CNN-based methods offer hierarchical feature extraction and efficiency compared with conventional approaches, supporting their mainstream use in automatic remote-sensing building extraction.
  • Vision Transformers treat images as ordered sequences and apply self-attention to model global dependencies, motivating their use in remote-sensing segmentation tasks.

III. METHODOLOGY

BuildFormer combines a global context path with a spatial-detailed context path to support large-scale remote-sensing building extraction. Its global path uses linear attention, while convolutional components preserve spatial detail and strengthen local interactions.

  • BuildFormer architecture: BuildFormer uses a Global Context Path and a Spatial-detailed Context Path whose features are aggregated into final semantic features.The global path produces four multiscale feature maps, while the spatial path produces a high-resolution feature map.
  • Spatial-detailed Context Path: The spatial-detailed path uses six CBR blocks and maintains an output feature map at 1/4 of the input resolution.Each block contains a 3×3 convolution, batch normalization, and ReLU6 activation, with progressively expanded channels.
  • Global Context Path: The global context path is a Vision Transformer built from BuildFormer Blocks, Patch Embedding, and Patch Merging modules.Patch Merging supports hierarchical representations by reducing resolution and expanding channel dimensions.
  • Global Context Path: Patch Embedding uses overlapping convolutional patches, reducing resolution to 1/4 while adding depth-wise convolution and residual location information.Two 3×3 convolutions with stride 2 expand patch channels to 96.
  • BuildFormer Block: Each BuildFormer Block combines W-LMHSA, a convolutional multilayer perceptron, batch normalization, and residual connections.The C-MLP strengthens within-window interaction while retaining competitive accuracy and improving efficiency relative to shifted-window processing.
  • BuildFormer Block: W-LMHSA replaces quadratic window attention with O(dN) complexity, reducing computation and memory demands as window size increases.The method uses a first-order Taylor approximation and L2 normalization to rewrite attention without explicitly forming the N×N similarity matrix.

D. Context Aggregation Module

The Context Aggregation Module fuses complementary global semantic and spatial-detailed features into a final semantic feature. It uses multi-level fusion after channel alignment and upsampling.

  • Global Context Path and Spatial-detailed Context Path features are complementary, encoding high-level semantics and rich spatial details, respectively.
  • Four 1×1 convolution layers unify the global feature maps to 384 channels before fusion.
  • Four CBR blocks, upsampling, and addition operations perform multi-level feature fusion.
  • The fused global features are combined with the spatial-detailed SCP feature to generate the final fused feature.

E. Loss Function

BuildFormer uses a joint loss combining region-level classification, overlap, and boundary supervision. The boundary term applies binary cross-entropy to Laplacian-extracted building boundaries.

  • The joint loss combines cross-entropy loss, dice loss, and boundary cross-entropy loss.
  • Laplacian convolution extracts building boundaries from predicted and true labels for boundary supervision.
  • Binary cross-entropy is applied to the extracted predicted and true building boundaries.

A. Datasets

Experiments use the Massachusetts, WHU, and Inria aerial building datasets, covering varied urban scenes, large geographic areas, and multiple cities. Dataset-specific official partitions or prescribed splits are followed.

  • Three public datasets are evaluated: Massachusetts, WHU, and Inria Aerial Image Labeling.
  • Massachusetts: The Massachusetts dataset contains 151 Boston-area aerial images of 1500×1500 pixels at 1 m ground sampling distance.
  • Massachusetts: The Massachusetts scenes include urban and suburban areas with buildings varying in size, shape, texture, and colour.
  • WHU: The WHU aerial subset covers over 450 km², includes 22,000 buildings, and provides 8,189 512×512 tiles at 0.3 m resolution.
  • Inria: The Inria dataset contains 360 fine-resolution images from Austin, Chicago, Kitsap, Tyrol, and Vienna; preprocessing yields 9,737 training and 1,942 validation tiles.

B. Evaluation Metrics

Model performance is evaluated using IoU, F1 score, precision, and recall. These metrics rely on counts of true-positive, false-positive, and false-negative predictions.

  • IoU, F1 score, precision, and recall are used to evaluate building extraction models.
  • TP, FP, and FN denote true positive, false positive, and false negative predictions, respectively.

C. Experimental Setting

Experiments evaluate BuildFormer’s spatial-detailed and global context paths, C-MLP, and W-LMHSA through ablations on the Massachusetts building dataset. The results show gains in IoU and improved efficiency relative to alternative designs.

  • Experimental Setting: The ablation experiments were conducted on the Massachusetts building dataset.
  • Spatial-Detailed Context Path: 1.36% IoU improvement demonstrates the effectiveness of the spatial-detailed context path.Removing this path reduced fine-grained building extraction performance.
  • Global Context Path: BuildFormer improved IoU by 2.04% over Swin-Small and by 5.01% over ResNet101.
  • Convolutional MLP: The C-MLP increased IoU by 5.16% and F1 score by 3.36% compared with the standard MLP.The reported ablation values are 70.58 IoU and 82.75 F1 for MLP versus 75.74 IoU and 86.19 F1 for C-MLP.
  • Window-Based Linear Attention: W-LMHSA improved IoU by 2% while saving about 25% computational complexity compared with W-MHSA.W-LMHSA kept GPU memory and speed stable with large windows, whereas W-MHSA increased memory use and reduced speed.

B. Comparison of State-of-the-art Methods

BuildFormer is compared with convolutional and Vision Transformer methods across the Massachusetts, WHU, and Inria building datasets, achieving leading reported performance.

  • 87.52% Recall on the Massachusetts building dataset is the highest among compared networks, exceeding others by more than 2.33%.The authors associate this result with fewer missed building pixels.
  • 81.44% IoU and 89.77% F1 score are reported for the Inria Aerial Image Labeling dataset.

VI. CONCLUSION

The conclusion presents BuildFormer as a dual-path Vision Transformer for building extraction that jointly captures global information and spatial details while reducing attention complexity.

  • VI. CONCLUSION: Its dual-path structure captures global context and spatial details simultaneously for precise building segmentation.
  • VI. CONCLUSION: Window-based linear multi-head self-attention reduces window-based attention complexity to O(N), enabling large windows for enhanced global-context modelling without high computation.
  • VI. CONCLUSION: Experiments on the Massachusetts, WHU, and Inria building datasets demonstrate superiority over state-of-the-art methods.
Loading 2111.15637v2…