Source-linked AI summary
PolyBuild: An End-to-End Method for Polygonal Building Contour Extraction from High-Resolution Remote Sensing Images
Yaoteng Zhang, Julin Zhang, Guangshuai Wang, Jiwei Deng, Hui Sheng, Yasir Muhammad, Shiqing Wei
TL;DR
Building polygon extraction requires direct, accurate vector contours despite limitations of pixel-based segmentation and difficult imaging conditions. PolyBuild uses ICGM and COM to generate and iteratively refine contours end to end, and the authors report strong performance across three datasets, with remaining errors tied to detection quality.
Problem
Existing mask-based extraction often requires post-processing and produces rough contours, while challenging conditions make accurate building contour extraction difficult.
Method
PolyBuild combines ICGM, which uses coupled sub-region center features, with a CNN-Transformer COM for iterative contour refinement.
Results
Experiments on three building datasets demonstrate state-of-the-art instance-level contour extraction performance, including improved handling of minor and major occlusions.
Takeaways & Limitations
PolyBuild directly produces high-quality building contour polygons without additional post-processing operations.
Takeaways & Limitations
Contour extraction depends on object-detection accuracy, and CenterNet causes missed detections and false positives such as missed small buildings and duplicate detections.
Abstract
from arXiv · showhide
Extracting building polygon contours from high-resolution remote sensing images is a fundamental task for various mapping applications. However, the presence of varying imaging conditions and complex building structures, makes automatic contour extraction extremely challenging. Mainstream approaches for building extraction often rely on pixel-level segmentation followed by multiple post-processing steps to produce building contour, which can be computationally intensive and prone to errors. In this paper, we propose an end-to-end method named PolyBuild, which can directly extract building vector polygons from high-resolution remote sensing images without the need for any post-processing operations. The proposed method leverages two primary modules: an Initial Contour Generation Module (ICGM) and a Contour Optimization Module (COM). The ICGM is designed to generate an initial building contour by utilizing concatenated sub-region center features for each building instance. It performs simultaneous object detection and initial contour extraction by generating bounding boxes and using the center features of four sub-regions to represent each building. The Contour Optimization Module (COM) further refines the generated building contours by iteratively integrating Convolutional Neural Network (CNN) features and contour positional information in a Transformer-based decoder. The hybrid CNN-Transformer architecture effectively captures both local and global spatial relationships within the building contour, ensuring high-quality boundary delineation. Extensive experiments are conducted on three building datasets to evaluate the performance of PolyBuild. The results demonstrate that PolyBuild significantly outperforms state-of-the-art methods, including mask-based and contour-based approaches.
I. INTRODUCTION
PolyBuild addresses limitations of mask- and contour-based building extraction with an end-to-end framework that generates and optimizes vector contours directly. Its ICGM uses coupled sub-region center features, while COM combines CNN and Transformer processing for contour refinement.
- Mask-based methods produce rough contours, lack direct vector contour output, and struggle with obstacles such as trees and shadows.
- Contour-based methods directly regress vectorized building contours but may begin from inaccurate bounding-box or ellipse initializations.
- PolyBuild is an end-to-end framework that jointly performs building detection and contour extraction without requiring post-processing.
- ICGM generates more stable initial contours from coupled center features of four bounding-box sub-regions, reducing subsequent optimization complexity.
- COM integrates CNN and Transformer architectures to capture local and global contour information and automatically associate vertices during iterative optimization.
III. METHODOLOGY
PolyBuild first detects building instances and generates initial contours through ICGM, then uses COM for iterative refinement. ICGM divides each predicted bounding box into four sub-regions and uses their features to predict contour offsets.
- PolyBuild integrates building detection and contour polygon extraction into an end-to-end framework with ICGM followed by three COM refinement iterations.
- ICGM predicts a bounding box for each building, divides it into four sub-regions, and regresses initial contour vertices from sub-region center features.
- The detection heads produce building center and bounding-box information, from which the top K peaks and corresponding widths and heights define building boxes.
- Focal Loss addresses center-point class imbalance by reducing the influence of easy samples and emphasizing difficult minority-class examples.
2) Regression of Initial Contours from Sub-region Center Points:
ICGM regresses initial contours from concatenated features at four sub-region centers rather than a single building center. Predicted offsets are added to those centers to construct the contour sequence.
- Sub-region centers are closer to building boundaries than the overall center and provide richer coupled information for contour regression.
- Bilinearly interpolated sub-region features are concatenated and passed through linear layers and ReLU activations to predict N vertex offsets.
- The predicted offsets are added to the four sub-region centers to form the initial building contour.
- Contour vertices follow a fixed clockwise or counterclockwise sequence aligned with control points and dense contour resampling.
- A scaling factor λ=10 reduces the effect of varying regression distances across buildings and helps maintain model stability.
C. Contour Optimization
Contour optimization refines rough ICGM contours through coarse and precise adjustment stages. The coarse stage combines contour-vertex and sub-region features for preliminary global adjustment.
- ICGM contours are often rough, so contour optimization refines them through coarse and precise adjustment stages.
- The coarse adjustment stage concatenates contour-vertex and sub-region features to perform a preliminary global adjustment of contour vertices.
1) Contour Coarse Adjustment:
The contour coarse adjustment module combines contour-vertex features with four sub-region center features to predict and apply vertex offsets, producing coarse-adjusted contours.
- Each contour vertex feature is extracted from the feature map by bilinear interpolation and concatenated into a contour representation.
- Four sub-region center features are concatenated with the contour representation to form an (N + 4) × C feature vector that improves control over overall offset magnitude.
- A two-hidden-layer MLP predicts offsets for each contour vertex, which are added to the initial contour coordinates.
- The experiments set C to 64 and the scaling factor ω to 4 to control offset ranges and improve regression robustness.
- Smooth L1 loss supervises the contour coarse adjustment offset loss, producing the coarse-adjusted contours.
2) Contour Optimization Module Integrated CNN-Transformer:
The Contour Optimization Module combines local CNN features, contour positions, and Transformer attention to iteratively refine building vertices using both local and global dependencies.
- Because each contour vertex is linked to adjacent vertices, accurate offset prediction requires querying associated vertices rather than relying only on the current vertex.
- The CNN-Transformer structure extracts local and global information, captures dependencies between vertices at different positions, and iteratively predicts more accurate offsets.
- Contour vertex features are combined with relative positions, processed by circular convolution, and passed to a Transformer decoder.
- Predicted vertex offsets are added to the contour from the previous iteration, and contour optimization proceeds through three iterations.
- The decoder uses position encoding, an MLP, multi-head self-attention, and multi-head attention to model contour-vertex dependencies and correlations.
- Dynamic Matching Loss combines distances from predicted vertices to densified boundaries with distances from ground-truth corners to their nearest predicted vertices.
IV. EXPERIMENTAL SETTING
PolyBuild is evaluated on the WHU aerial building dataset and the WHU-Mix (Vector) building dataset using publicly available building annotations.
- The evaluation uses the WHU aerial building dataset, the WHU-Mix (Vector) building dataset, and the Crowd AI dataset.
- The WHU aerial dataset contains 187,000 buildings in 0.2-meter-resolution imagery, cropped into 512 × 512 tiles.
- Its training, validation, and test sets contain 10,961, 2,486, and 6,177 tiles, respectively.
2) WHU-Mix (Vector) building dataset:
The WHU-Mix dataset combines multiple sources with varied building styles and geographic environments, while the training setup and evaluation use standard COCO metrics.
- WHU-Mix combines Crow AI, Open AI, SpaceNet, Inria, WHU, and newly collected data from satellite and other imagery sources.
- A geographically non-overlapping test set is used to further validate method generalization capabilities.
- Crowd AI contains 340,000-plus 300 × 300 RGB satellite-image tiles, but its samples lack diversity and require higher computational costs.
- Training uses Adam with an initial learning rate of 1 × 10^-4, lasting 150 epochs on WHU aerial and 30 epochs on WHU-Mix.
- PolyBuild jointly optimizes heat-map, size, initial-contour, coarse-adjustment, and three contour-optimization losses.
- Performance is assessed with COCO average precision and average recall across IoU thresholds, including AP50 and AP75.
V. RESULTS AND DISCUSSION
PolyBuild is evaluated against mask-based and contour-based methods across three building datasets, with results showing strong accuracy and robust contour extraction. Visual comparisons indicate advantages in boundary delineation, large-building contours, and recognition under complex backgrounds.
- PolyBuild achieves the highest WHU aerial dataset performance, reaching 74.0% AP and 80.7% AR.
- 53.5 mAP and 62.2 mAR on WHU-Mix surpass mask-based SOTA by 0.6 AP and 2.6 AR, and contour-based SOTA by 0.9 mAP.
- PolyBuild maintains high extraction accuracy on the geographically disjoint WHU-Mix test set II, supporting its reported generalization capability.
- Visual results show mask-based contours are blurry and fragmented, while PolyBuild’s additional building detector improves recognition supervision compared with methods exhibiting building omissions.
- For large buildings, four-sub-region center features reduce initial contour error, while CNN-Transformer optimization models dependencies among vertices using global and local information.
B. Ablation study
The ablation study examines initial-contour strategies, module combinations, occlusion handling, multistage refinement, and efficiency. Results favor concatenated sub-region features with CNN-Transformer optimization, while complex occluded topologies remain difficult.
- Initial contour generation: Strategy D achieves the highest initial-contour accuracy by concatenating four sub-region center features and adding predicted offsets to corresponding center points.
- Ablation study: Replacing the baseline initialization with ICGM increases initial-contour AP by 1.7 and AR by 1.1, with AP75 increasing by 1.4.
- Ablation study: Adding ICGM to final contour generation increases baseline AP by 0.8 and AR by 0.2, while adding COM increases AP by 2, AR by 0.2, and AP75 by 1.6.
- Occlusion analysis: PolyBuild outperforms YOLACT under minor and major occlusions, using sub-region centers and contour-vertex relationships to restore more complete structures.
- Occlusion analysis: Complex occluded regions containing multiple contour corners remain difficult to localize accurately, increasing instability in reconstructed contours.
- Efficiency and multistage refinement: PolyBuild generates polygons end-to-end without post-processing, reaches high-quality outputs across three viewing conditions, and runs at 30.46 FPS with 30.3M parameters on an RTX 2080 Ti.
D. Limitations and future work
PolyBuild’s contour accuracy remains dependent on object-detection quality because the method detects instances before extracting contours. The authors identify missed detections and false positives as limitations and plan to use stronger detectors.
- Contour extraction accuracy depends on the precision of PolyBuild’s object detector because detection precedes contour extraction.
- PolyBuild can miss small buildings and produce false positives for trucks, abandoned land, or duplicate detections.
- The authors plan to apply more powerful detectors to further improve contour-extraction performance.