PaperlayerView exact revision on arXiv

Source-linked AI summary

UniBuild: Unified Building Mapping From Multi-Source Optical Remote Sensing Imagery With Detail Decoding and Geometry Regularization

Wei Huang, Chenying Liu, Yilei Shi, Xiao Xiang Zhu

arXiv:2609.37031v1cs.CV

TL;DR

Dataset-specific building extractors often generalize poorly, and limited detail recovery and geometric constraints leave boundaries and adjacent-building gaps difficult to represent. UniBuild unifies multi-source RGB optical training with detail decoding and geometry-aware losses. It reports improvements across diverse datasets, generalization to unseen domains, and building extraction support up to 10 m resolution.

  • Problem

    Building-extraction methods are often dataset-specific, while weak detail recovery and geometric regularization limit generalization, boundary quality, and separation of adjacent buildings.

    Source: §I · p. 1

  • Method

    UniBuild jointly trains across multi-source RGB optical datasets and combines an HR-DPT detail decoder with direction-aware and saddle-aware geometry losses.

    Source: §I · p. 2

  • Results

    Across 12 datasets, joint UniBuild training raises average IoU from 74.03 to 76.27 and B-IoU from 61.06 to 65.60 versus independently trained DINOv3-B-DPT models.

    Source: §I · p. 2, C. Single-dataset vs. Multi-dataset… · p. 7

  • Takeaways & Limitations

    UniBuild generalizes across sensors and resolutions, including RGB optical imagery up to 10 m GSD, and its masks can be polygonized into building footprints.

    Source: §I · p. 2, §V · p. 11

    At 10 m, small buildings and fine footprint details approach or fall below sensor resolution; annotations may also be missing, outdated, or misaligned.

    Source: §V · p. 11

Abstract

from arXiv · show

Building extraction from optical remote sensing (RS) imagery is fundamental to urban mapping, yet existing methods are often dataset-specific and generalize poorly to unseen domains. Their practical use is also limited by insufficient detail recovery and weak geometric regularization, leading to blurred boundaries, irregular shapes, and merged adjacent buildings. To address these issues, we propose UniBuild, a unified building extraction framework for multi-source RGB optical RS imagery. First, a unified multi-dataset training scheme is constructed over heterogeneous RGB optical datasets to learn transferable building representations across sensors and resolutions. Second, a novel detail-preserving HR-DPT decoder is designed to integrate high-level semantic features with high-resolution spatial features, enhancing building detail recovery. Third, geometry-aware regularization is introduced through a structure-tensor-based direction-aware loss for boundary direction consistency and a saddle-aware loss for suppressing false activations in narrow inter-building gaps under low-resolution conditions. We train and evaluate UniBuild on multi-source RGB optical datasets, including 10 public high-resolution datasets and two self-collected low-resolution datasets. Experiments show that UniBuild consistently improves building-region accuracy, boundary sharpness, and adjacent-building separation across diverse datasets. It also generalizes well to unseen domains and supports practical building extraction from RGB optical RS imagery up to 10\,m resolution. The predicted masks can be further converted into GIS-compatible building footprints through simple polygonization. The trained model and inference code are released at https://github.com/zhu-xlab/UniBuild.

I. INTRODUCTION

UniBuild targets dataset-specific generalization, weak detail recovery, and insufficient geometric regularization in building extraction. It combines multi-dataset training, detail decoding, and geometry-aware losses in one framework for multi-source RGB optical imagery.

I. INTRODUCTION

  • Existing methods often rely on specific datasets, while spatially compressed features and weak geometry constraints limit transfer, boundary recovery, and separation of adjacent buildings.Under low-resolution conditions, mixed pixels and over-smoothed features can obscure narrow gaps and cause neighboring buildings to merge.Source: §I · pp. 1–2
  • UniBuild is jointly trained across heterogeneous RGB optical imagery rather than using separate models for each data source.The framework is designed for imagery up to 10 m resolution.Source: §I · p. 2
  • HR-DPT combines high-level semantics with high-resolution shallow features, while direction-aware and saddle-aware losses target boundary consistency and low-resolution inter-building gaps.The direction-aware loss models local boundary orientation; the saddle-aware loss suppresses false activations in narrow gaps.Source: §I · p. 2
  • Across 10 public high-resolution and two self-collected low-resolution datasets, UniBuild shows stable improvements, especially in boundary quality, and stronger cross-sensor and cross-resolution generalization.The evaluated imagery includes 4.8 m Planet and 10 m Sentinel-2 data.Source: §I · p. 2

A. Building Extraction from Optical Remote Sensing Imagery

  • Building extraction methods have progressed from handcrafted cues to deep segmentation, transformers, and visual foundation models, but most remain validated on only a few datasets.The unified setting instead requires one shared model to generalize across heterogeneous sensors and spatial resolutions.Source: A. Building Extraction from Optical… · p. 2

B. High-Resolution Detail Decoding for Dense Prediction

UniBuild motivates high-resolution detail decoding and geometry-aware regularization, then describes its shared input and feature-extraction pipeline. Inputs are normalized and grouped by resolution before VFM features are decoded for building extraction.

B. High-Resolution Detail Decoding for Dense Prediction

C. Geometry-Aware Regularization for Building Extraction

  • Prior boundary methods emphasize contour sharpness or alignment, while structure- and topology-aware approaches encode shape and connectivity; narrow low-resolution gaps and local boundary directions remain under-modeled.Mixed pixels and over-smoothed features can merge neighboring buildings.Source: C. Geometry-Aware Regularization for… · p. 3

III. UNIBUILD

  • UniBuild’s four method components are unified input and VFM feature extraction, HR-DPT decoding, direction-aware boundary regularization, and saddle-aware low-resolution gap suppression.The components are presented as the framework’s overall design.Source: §III · p. 3

A. Unified Input Representation and VFM Feature Extraction

  • UniBuild uses a shared building-extraction model for RGB optical imagery with heterogeneous sensors and spatial resolutions.A unified input representation is introduced to make those inputs compatible.Source: A. Unified Input Representation and… · p. 3

1) Resolution-aware input normalization and batch grouping:

  • Images are grouped as high or low resolution using a 1 m GSD threshold; low-resolution inputs are bilinearly upsampled to 1 m, and all images are cropped into 512 × 512 patches.Saddle-aware regularization applies only to low-resolution batches, while direction-aware regularization applies to both groups.Source: 1) Resolution-aware input… · p. 3

2) Unified augmentation and binary supervision:

  • Training converts annotations from datasets with differing resolution, quality, and taxonomy into a shared binary building-and-background label space.The training pipeline uses weak geometric and strong intensity augmentation.Source: 2) Unified augmentation and binary… · p. 3

3) VFM-based multi-level feature extraction:

  • DINOv3 processes each input image into four intermediate feature maps; its non-hierarchical features share H/16 × W/16 resolution, limiting fine-boundary recovery.The maps encode multi-depth semantic cues, which motivates the HR-DPT decoder.Source: 3) VFM-based multi-level feature… · p. 3

B. HR-DPT Decoder

HR-DPT combines compressed semantic features with a high-resolution shallow branch to recover building details. The two branches are fused progressively to retain spatial structure while injecting semantic cues.

B. HR-DPT Decoder

  • HR-DPT combines a DPT-style low-resolution semantic branch with a high-resolution shallow branch that injects fine image details through gated residual fusion.The semantic branch performs top-down decoding; the shallow branch preserves spatial details.Source: B. HR-DPT Decoder · p. 3

1) LR semantic feature branch:

  • A semantic pyramid is built by projecting backbone features and resizing them to scale-specific levels, but these features remain spatially compressed.The branch uses 1×1 projection and four resizing operations.Source: 1) LR semantic feature branch · pp. 3–4

2) HR shallow feature branch:

  • The HR branch extracts shallow image features, upsamples each semantic level, and fuses them by concatenation and convolution to inject semantic cues while preserving fine spatial structures.The resulting fused feature is used to build an HR feature pyramid through stride-2 downsampling blocks.Source: 2) HR shallow feature branch · p. 4

3) HR-enhanced semantic decoding:

The decoder refines semantic features from coarse to fine scales and injects matched high-resolution features after each refinement. A learned gate controls the residual high-resolution contribution before the finest decoded feature is upsampled for prediction.

3) HR-enhanced semantic decoding:

  • Top-down DPT decoding starts from the coarsest feature and fuses progressively finer semantic features, with matched high-resolution features injected after each refinement.Each refinement block upsamples the decoded feature before fusion.Source: 3) HR-enhanced semantic decoding · p. 4
  • A gate map controls high-resolution detail injection through a residual update that adds the projected shallow feature to the decoded feature.The gate is calculated from the decoded semantic feature and matched high-resolution shallow feature.Source: 3) HR-enhanced semantic decoding · p. 5
  • The finest decoded feature is passed to a prediction head and upsampled to original resolution to produce background and building classes.Source: 3) HR-enhanced semantic decoding · p. 5

C. Direction-aware Consistency

The direction-aware loss regularizes building boundaries by aligning structure-tensor orientations between predictions and labels. Supervision is restricted to boundary-support regions where directional structure is meaningful.

C. Direction-aware Consistency

1) Boundary-focused supervision region:

2) Structure-tensor-based direction representation:

  • Class-wise gradients and structure-tensor components produce a normalized orientation descriptor, with orientation energy also computed for the prediction and smoothed target.A small positive constant prevents division by zero.Source: 2) Structure-tensor-based direction… · p. 5

3) Direction-aware boundary regularization:

The direction-aware loss compares prediction and target orientations within boundary-supported regions. A separate saddle-aware loss is introduced to penalize false positives in building gaps.

3) Direction-aware boundary regularization:

  • Prediction and target orientation descriptors are combined with normalized target orientation energy and the boundary-support mask to define the direction-aware loss.The loss uses the inner product of normalized orientation vectors; better alignment yields a smaller loss.Source: 3) Direction-aware boundary… · p. 5

D. Saddle-aware Suppression

1) Saddle mask construction:

The saddle mask identifies background pixels between nearby buildings, and the saddle-aware Dice loss selectively penalizes false activations in low-resolution gaps.

1) Saddle mask construction:

  • A saddle pixel is a background pixel supported by building pixels from two opposite directions.Eight fixed 5 × 5 directional kernels detect support along horizontal, vertical, and diagonal directions.Source: 1) Saddle mask construction · p. 5
  • The final saddle mask selects supported background pixels between nearby buildings and serves only as a non-gradient spatial weighting mask.Source: 1) Saddle mask construction · p. 6

2) Saddle-weighted Dice loss:

  • Only the false-positive Dice term is reweighted, targeting false building activations in background gaps.For HR samples αHR = 0, recovering standard soft Dice; for LR samples αLR > 0 penalizes false activations in narrow inter-building gaps.Source: 2) Saddle-weighted Dice loss · p. 6

E. Overall Objective

UniBuild combines multi-resolution, multi-source training with a detail-preserving decoder and geometry-aware regularization. It improves region and boundary metrics across datasets and transfers across dataset domains.

E. Overall Objective

  • Direction-aware regularization encourages boundary-orientation consistency across resolutions, while saddle-aware regularization preserves inter-building gaps under low-resolution conditions.The objective uses λdir for direction-aware regularization and αLR for low-resolution saddle-aware suppression, with αHR = 0 by default.Source: E. Overall Objective · p. 6

A. Training Datasets and Experimental Settings

  • Training data span 0.05 m to 10 m GSD, covering aerial, high-resolution satellite, and medium-resolution satellite imagery across public and self-collected datasets.The training labels are unified as building/background, with unlabeled or invalid pixels ignored.Source: A. Training Datasets and Experimental… · p. 6

B. Comparison With SOTA Building Extraction Models

  • UniBuild achieves the best overall performance across four representative datasets, with gains in region-level and boundary-aware metrics over DINOv3-B-DPT using the same backbone.Qualitative examples cover INRIA, WHU-Mix, GF-7, and Planet; the reported masks are more complete, with sharper boundaries and clearer adjacent-building separation.Source: B. Comparison With SOTA Building… · pp. 6–7, Figure 4 · p. 7

C. Single-dataset vs. multi-dataset training

  • Across 12 datasets, UniBuild raises average IoU from 74.03 to 76.27 and average B-IoU from 61.06 to 65.60 versus independently trained DINOv3-B-DPT models.Source: C. Single-dataset vs. Multi-dataset… · p. 7
  • Jointly trained UniBuild achieves positive gains across all datasets, maintaining strong results across multi-source imagery without compromising dataset-specific performance.The paper attributes the gains to both the proposed architecture and diverse multi-dataset training.Source: C. Single-dataset vs. Multi-dataset… · pp. 7–8

D. Ablation Study

Ablations report complementary contributions from HR-DPT, direction-aware regularization, and low-resolution saddle suppression. The decoder adds modest complexity, while direction-aware regularization improves boundary metrics.

1) Overall Module Contribution:

  • Adding saddle-aware suppression raises MIoU from 45.77 to 47.80 on Planet and from 26.97 to 33.60 on ST-2.The ablation reports gains on low-resolution datasets by reducing false-positive connections between adjacent buildings.Source: Table V · p. 8, 1) Overall Module Contribution · p. 8

2) Decoder Effectiveness and Complexity:

3) Direction-Aware Consistency:

  • On INRIA, direction-aware regularization raises B-IoU from 67.81 to 70.87 and B-F1 from 71.15 to 74.09.Its effect is more sensitive on MR/LR data with indistinct boundaries and mixed pixels; λdir = 0.5 is selected to balance boundary quality and region completeness.Source: Table VII · p. 9, 3) Direction-Aware Consistency · p. 9

4) Saddle-Aware Suppression:

Saddle-aware suppression improves low-resolution building separation, while zero-shot evaluations show transfer to unseen domains and flexible inference-time scaling. The conclusion highlights the framework’s broad scope and remaining limits in low-resolution extraction, annotations, and footprint regularization.

4) Saddle-Aware Suppression:

  • MIoU rises from 45.77 to 47.84 on Planet and from 26.97 to 33.60 on ST-2 with αLR = 15, while narrow low-resolution gaps are preserved.The saddle penalty is activated only for low-resolution batches; Table VIII compares penalty strengths, and Fig. 8 visualizes saddle regions.Source: 4) Saddle-Aware Suppression · p. 9, Table VIII · p. 10, Figure 8 · p. 10

E. Out-of-Domain Building Extraction Performance

  • Without fine-tuning, UniBuild performs strongly on Waterloo, WHU-Satellite, and Massachusetts and remains effective on challenging ISPRS-Pforzheim, producing coherent masks and separable instances for polygonization.The qualitative examples cover four unseen datasets; ISPRS-Pforzheim imagery is upsampled from 5.8 m to 1 m.Source: E. Out-of-Domain Building Extraction… · pp. 9–10, Figure 9 · p. 11
  • Small or adjacent buildings that are poorly resolved in low-resolution inputs can still limit accurate instance-level delineation.The conclusion also identifies 10 m Sentinel-2 imagery as challenging when buildings and footprint details approach or fall below sensor resolution.Source: E. Out-of-Domain Building Extraction… · p. 10, §V · p. 11
  • IoU improves from 53.57 to 69.40 on Massachusetts when 1.0 m imagery is upsampled to 0.5 m before inference.This supports flexible inference-time resolution adjustment and a trade-off between extraction accuracy and computational cost without retraining.Source: E. Out-of-Domain Building Extraction… · p. 10

V. CONCLUSION

  • UniBuild combines multi-dataset training, HR-DPT decoding, and geometry-aware regularization, with experiments showing more accurate regions, sharper boundaries, and clearer building separation up to 10 m GSD.The conclusion describes the components as complementary and the unified model as practical for multi-source optical building extraction.Source: §V · p. 11
  • Remaining limitations include low-resolution detail recovery, missing or outdated annotations and image–label misalignment, and footprint regularization applied as postprocessing rather than end-to-end vectorization.Future work targets low-resolution building modeling, noise-robust supervision, and end-to-end footprint vectorization.Source: §V · p. 11

A. Cross-Resolution Building Polygonization

UniBuild converts RGB imagery from different sources and resolutions into building polygons, with higher-resolution imagery yielding more detailed polygons. Its single-date building representations also support zero-shot change detection on bi-temporal imagery.

A. Cross-Resolution Building Polygonization

  • From RGB imagery, UniBuild produces building masks that are polygonized and exported as shapefiles, including examples from unseen Google Maps imagery and a different Planet region.The examples show Google Maps imagery from Fez at 0.7 m and Planet imagery from Carluke upsampled from 4.8 m to 1.0 m; rows show RGB images, predicted polygons, and OSM polygons.Source: A. Cross-Resolution Building… · p. 14, Figure 10 · p. 14
  • Higher-resolution imagery yields more detailed and accurate polygons, while lower-resolution inputs are more suited to large-scale mapping than fine-grained instance delineation.The paper presents this comparison across the 0.7 m Google Maps and upsampled 4.8 m Planet examples.Source: A. Cross-Resolution Building… · p. 14

B. Zero-shot Building Change Detection

  • The change map is formed by independently predicting building masks for both dates and applying XOR, without using change labels during training.The binary masks come from pixel-wise argmax over background and building output channels.Source: B. Zero-shot Building Change Detection · p. 14
  • UniBuild achieves an F1 score of 85.14% and an IoU of 74.12% on LEVIR-CD without training on bi-temporal change labels.The reported evaluation applies a model trained for single-date building extraction.Source: B. Zero-shot Building Change Detection · p. 14, Table XI · p. 14

Related papers

Loading 2609.37031v1…