PaperlayerView exact revision on arXiv

Source-linked AI summary

UrbanSAM: Learning Invariance-Inspired Adapters for Segment Anything Models in Urban Construction

Chenyu Li, Danfeng Hong, Bing Zhang, Yuxuan Li, Gustau Camps-Valls, Xiao Xiang Zhu, Jocelyn Chanussot

arXiv:2502.15199v1cs.CV

TL;DR

UrbanSAM addresses the difficulty of segmenting heterogeneous, scale-varying urban objects when SAM depends on sparse manual prompts and conventional adaptation. It introduces MRA-inspired U-scaling adapters and aligns their multiscale features with the SAM encoder to generate learned prompts. Experiments report superior performance across building, water-body, and road extraction, including IoU gains over Deeplabv3+ for road extraction.

  • Problem

    Urban segmentation is challenged by heterogeneous object characteristics, fine-grained resolution, dense layouts, scale variation, and sparse manual prompts that can yield incomplete or unstable results.

  • Method

    UrbanSAM uses learnable U-scaling adapters inspired by MRA theory, transfers their invariance through encoder alignment, and jointly optimizes segmentation predictions and mask prompts.

  • Results

    UrbanSAM achieves state-of-the-art road-extraction performance, improving IoU over Deeplabv3+ by 3.02%, 4.97%, and 5.27% across its three versions.

  • Takeaways & Limitations

    UrbanSAM provides flexible segmentation of scale-varying urban objects, including buildings, roads, and water bodies, on a global-scale dataset.

Abstract

from arXiv · show

Object extraction and segmentation from remote sensing (RS) images is a critical yet challenging task in urban environment monitoring. Urban morphology is inherently complex, with irregular objects of diverse shapes and varying scales. These challenges are amplified by heterogeneity and scale disparities across RS data sources, including sensors, platforms, and modalities, making accurate object segmentation particularly demanding. While the Segment Anything Model (SAM) has shown significant potential in segmenting complex scenes, its performance in handling form-varying objects remains limited due to manual-interactive prompting. To this end, we propose UrbanSAM, a customized version of SAM specifically designed to analyze complex urban environments while tackling scaling effects from remotely sensed observations. Inspired by multi-resolution analysis (MRA) theory, UrbanSAM incorporates a novel learnable prompter equipped with a Uscaling-Adapter that adheres to the invariance criterion, enabling the model to capture multiscale contextual information of objects and adapt to arbitrary scale variations with theoretical guarantees. Furthermore, features from the Uscaling-Adapter and the trunk encoder are aligned through a masked cross-attention operation, allowing the trunk encoder to inherit the adapter's multiscale aggregation capability. This synergy enhances the segmentation performance, resulting in more powerful and accurate outputs, supported by the learned adapter. Extensive experimental results demonstrate the flexibility and superior segmentation performance of the proposed UrbanSAM on a global-scale dataset, encompassing scale-varying urban objects such as buildings, roads, and water.

I. INTRODUCTION

UrbanSAM addresses heterogeneous urban remote-sensing scenes by learning scale-invariant, adaptive prompts rather than relying on sparse manual interaction. Its customized SAM combines multiscale adapters with encoder alignment to improve segmentation across diverse urban objects.

  • Urban segmentation challenges: Urban scenes contain substantial spectral, textural, morphological, resolution, density, and layout variation that complicates accurate segmentation.These challenges span heterogeneous object characteristics, fine-grained high-resolution imagery, and densely packed global urban structures.
  • Urban segmentation challenges: SAM struggles with scale variations because the same remote-sensing object can exhibit different semantic characteristics across resolutions.Different object categories may also be best represented at particular spatial scales, limiting transfer across tasks.
  • Urban segmentation challenges: Sparse point and box prompts can constrain receptive fields, omit elongated targets, and produce unstable or inconsistent full-area segmentation.The limitation is especially problematic for roads and water bodies, whose spatial extent is difficult to cover with sparse prompts.
  • UrbanSAM contributions: UrbanSAM learns invariant properties across multiple scales through U-scaling adapters, reducing reliance on manual interactive prompts for diverse urban features.The adapters are designed to capture latent commonalities across urban morphologies and provide adaptive prompt guidance.
  • UrbanSAM contributions: Cross-attention aligns adapter features with the backbone encoder, while multi-scale sampling and U-shaped processing preserve contextual information across spatial scales.The design supports domain-specific prompt learning and more complete feature extraction for varying urban objects.
  • UrbanSAM contributions: UrbanSAM achieves superior performance across building, water-body, and road extraction tasks, outperforming existing state-of-the-art methods.The reported contribution emphasizes effectiveness, robustness, and generalizability in complex urban environments.

II. INVARIANCE LEARNING INSPIRED BY MULTI-RESOLUTION ANALYSIS

Multi-resolution analysis provides the theoretical basis for representing complex images through scale-related coefficients and resolution-invariant basis spaces. UrbanSAM uses this invariance perspective to motivate multiscale feature and prompt learning.

  • MRA-inspired UrbanSAM design: UrbanSAM’s workflow embeds MRA-inspired invariance in cascaded U-scaling adapters that capture hidden multiscale cues and transfer them into SAM transformer blocks.The transferred prompt-stream features guide alignment of subsequent representations for urban segmentation.
  • MRA foundations: MRA defines nested resolution spaces in which higher-resolution representations contain information from lower-resolution spaces.The approximation relationship supports deriving target subspaces from representations at other scales.
  • MRA foundations: The theory states that approximation operations are similar across resolutions, enabling comparable characterization of image content at different scales.This cross-resolution similarity is the basis for learning scale-invariant features.
  • MRA foundations: MRA theory models image spaces through coefficients and basis functions whose scale can vary while preserving intrinsic structural properties.This motivates resolution-invariant representations for images observed at different scales.

B. U-Scaling: Constituting a Refined Adapter for Perceptual Invariance

The U-Scaling adapter models scale variation in urban remote-sensing images through multiscale encoder-decoder processing. Its modules aggregate contextual features across resolutions to provide adaptive prompt guidance.

  • Motivation: Urban remote-sensing imagery spans approximately 1000m to 0.5m ground sampling distances, making single-scale analysis insufficient.This range can prevent models from capturing essential details or fully representing semantic context.
  • Adapter design: The U-shaped encoder processes shallow-to-deep multiscale features, while the decoder aggregates deep-to-shallow features to enhance representation capacity.This bidirectional arrangement is described as aligning with the MRA-based invariance approximation.
  • Adapter design: U-Scaling adapters learn inherent scene-scale invariance across multiple resolutions to eliminate dependence on manual prompts.The proposed adapter contains four U-Scaling modules, each approximating optimal basis spaces for different forms and resolutions.
  • Adapter operations: Within each U-Scaling module, convolutions and ReLU transform features, MaxPooling transitions across resolutions, and residual connections aggregate same-scale representations.These operations combine nonlinear transformation, contextual information, and feature completeness.
  • Adapter operations: The adapter’s mapping coefficient Φ is set to 2 in the experiments, while En and De denote encoder and decoder feature calculations.The input feature of the i-th adapter is represented by f(xi,0).

III. URBANSAM: CUSTOMIZED SAM FOR URBAN

UrbanSAM uses an adapter and cross-alignment mechanism to transfer multiscale invariance learning into SAM, while LoRA provides trainable low-rank adaptation around frozen original weights.

  • An adapter of multiple U-Scaling modules serves as a bridge, while cross-alignment transfers its invariance learning capabilities to the image encoder and LoRA parameters.The cross-alignment mechanism operates between U-Scaling outputs and ViT features.
  • The adapter's multiscale features and the trunk encoder's features are integrated through masked cross-attention to transfer multiscale aggregation capability.This alignment connects the adapter branch with the encoder feature stream.
  • Cross-branch masked attention fuses U-Scaling and ViT features into an output guided by foreground probabilities from mask predictions.Softmax-derived weights are multiplied by foreground probabilities after sampling adjustment.
  • LoRA keeps the original projection weight matrix frozen and adds a low-rank shortcut with matrices A and B, where r is much smaller than the input and output dimensions.The learnable parameters are introduced through a low-rank approximation rather than updating the full projection matrix.

B. Adaptive Mask Generation

UrbanSAM generates adaptive foreground prompts from multiscale predictions, then progressively fuses encoder, adapter, and mask features for detail-preserving decoding.

  • Adaptive Mask Generation: Outputs from multiple U-Scaling modules are concatenated after cross-alignment, and a 1 × 1 convolution predicts target regions under binary cross-entropy and Dice losses.The resulting predictions are converted into foreground probabilities for prompt generation.
  • Adaptive Mask Generation: A learnable threshold filters sigmoid foreground probabilities, enabling intensive prompts to adapt to different object extraction tasks without manual annotation.The threshold is initialized at 0.5 and learned by the network.
  • Hierarchical Consistency Decoder: The decoder progressively fuses intermediate image-encoder and adapter outputs through skip connections, using the adapter's invariance learning as constraints and guidance.Hierarchical upsampling is used to preserve spatial details and class information.
  • Hierarchical Consistency Decoder: Four hierarchical features combine encoder, U-Scaling, mask-prediction, and decoder outputs before convolutional and transposed-convolutional operations produce the final segmentation.Spatial pointwise multiplication and dynamic token weights are also used in the decoding process.

D. Composite Loss Function

UrbanSAM jointly optimizes segmented predictions and mask prompts using pixel- and region-wise losses, with additional mask supervision and deep supervision in its composite objective.

  • UrbanSAM combines pixel- and region-wise losses to jointly optimize segmented predictions and mask prompts.
  • The loss uses binary cross-entropy and Dice components whose weighting parameters balance their contributions.
  • The composite objective incorporates mask supervision from mask cross-attention and mask prompt learning.
  • Deep supervision adds a prediction at 1/4 resolution to the final loss.

IV. EXPERIMENTS

The experiments evaluate UrbanSAM and comparison methods using standard segmentation metrics, broad geographic coverage, controlled implementations, and specified training configurations.

  • Evaluation metrics: Evaluation uses Overall Accuracy, Precision, Recall, F1, and Intersection over Union to assess segmentation performance.OA measures correctly classified pixels, Precision measures correct positive predictions among predicted positives, and Recall measures correctly identified positives.
  • Dataset coverage: The dataset spans Asia, Oceania, North America, Europe, and Africa, supporting evaluation across diverse geographic settings.The reported geographic distribution is presented as evidence of broad applicability across regions.
  • Training setup: Training uses a weighted cross-entropy and Dice loss with weights 0.2 and 0.8, respectively, optimized by SGD with momentum 0.9 and weight decay 0.0001.The experiments use PyTorch and four NVIDIA L40 GPUs with 48GB memory each.
  • Experimental protocol: All methods are evaluated under identical conditions using official implementations or pretrained weights and original domain-specific hyperparameters.This comparison protocol is intended to ensure a fair evaluation.

B. Urban Water

UrbanSAM is evaluated for water-body extraction on globally distributed, high-resolution imagery and achieves strong quantitative and qualitative performance across varying scales and complex landscapes.

  • Dataset and setup: The GLH-Water dataset contains 250 Google Earth satellite images covering every continent except Antarctica at roughly 0.3-meter resolution.Each image spans 3,686 km2 and measures 12,800 by 12,800 pixels.
  • Dataset and setup: The images were partitioned into non-overlapping 512×512 patches, producing 125,000 training samples and 7,319 testing samples.The reported partition follows the dataset’s original scheme.
  • Dataset and setup: UrbanSAM was trained with 512×512 inputs for 15 epochs without data augmentation and with an initial learning rate of 0.001.No learning-rate scheduling strategy was used.
  • Quantitative results: 83.81% IoU was achieved by UrbanSAM’s ViT-H without human prompts, exceeding the best baseline by 3.35%.The reported improvement is attributed to multiscale invariant features from the UrbanSAM prompting strategy.
  • Quantitative results: UrbanSAM combines multiscale invariant features with shape information rather than relying only on unstable color differences between water and adjacent surfaces.The passage links this feature fusion to improved generalization in large-scale water-body extraction.
  • Qualitative results: Qualitative results show accurate extraction of differently shaped and scaled water bodies across diverse geographic landscapes, with reduced omissions and false detections.The visualizations also highlight detailed water-body edges and robustness to similar land-cover interference.

C. Urban Road

UrbanSAM achieves state-of-the-art road extraction results on remote-sensing datasets, preserving road connectivity and structural details in visually and spatially complex scenes.

  • Datasets and setup: The road experiments use DeepGlobe and LSRV datasets, including 6,226 annotated DeepGlobe images from Thailand, Indonesia, and India.DeepGlobe provides 4,980 training images and 1,246 testing images at 1024×1024 pixels and 0.5-meter resolution.
  • Datasets and setup: Training uses 1024×1024 inputs, random rotation and flipping, 200 epochs, five warmup epochs, and an initial learning rate of 0.005.These settings are reported for the road extraction task.
  • Quantitative results: UrbanSAM’s three versions improve IoU over the best-performing Deeplabv3+ by 3.02%, 4.97%, and 5.27%.All three versions are reported to achieve state-of-the-art performance on the road extraction task.
  • Quantitative results: Compared with SAM and HQ-SAM, UrbanSAM’s learnable prompting reduces false detections for intricate and elongated road structures.The comparison specifically contrasts learnable prompting with traditional box- and point-based prompts.
  • Qualitative results: Qualitative results show continuous, non-fragmented road networks and preserved connectivity in topologically complex looped areas.The visualizations also cover similar-color backgrounds, blurred textures, and scenes with high spatial heterogeneity.

D. Urban Building

UrbanSAM is evaluated for building extraction across diverse datasets and compared with established segmentation methods. It achieves stronger quantitative and qualitative performance while capturing varied building structures in complex scenes.

  • Datasets: UrbanSAM is evaluated on Massachusetts, Inria, and WHU building datasets spanning different resolutions, cities, and building scales.The datasets include imagery from Boston, ten cities in the USA and Austria, and Christchurch, New Zealand.
  • Qualitative comparison: UrbanSAM qualitatively captures diverse building structures across dense urban and scattered suburban environments.The visualizations highlight regular-boundary understanding, reduced shadow interference, and discrimination from similar objects.
  • Experimental setup: The study uses 1024×1024 resampled inputs, 200 training epochs, rotation and flipping augmentation, and five warmup epochs.The initial learning rate is set to 0.005.
  • Quantitative comparison: UrbanSAM outperforms SAM and HQ-SAM by 7.5% and 6.83% IoU, respectively, for building extraction.The comparison is reported in Tab. III against state-of-the-art methods.
  • Quantitative comparison: UrbanSAM exceeds RSBuilding by 0.21% with ViT-B and 0.76% with ViT-L.The improvement is attributed to scale-invariant prompting that preserves building details and contours.

E. Ablation Analysis

The ablation analysis examines prompt robustness, fine-tuning choices, multiscale processing, cross-attention, and decoder design. Results consistently support the contribution of UrbanSAM’s learned and multiscale components.

  • Effectiveness of Learned U-Scaling Adapter: UrbanSAM matches a 70%-accurate mask prompt and a 90%-accurate 20-point prompt, while exceeding a 100%-accurate box prompt.The comparison evaluates learned prompts under simulated prompt-accuracy conditions.
  • Individual Components: LoRA improves zero-sample SAM transfer by 1.24% compared with omitting LoRA.The result is reported on the Massachusetts building dataset.
  • Individual Components: The multiscale cue increases IoU by 0.83% over a single-scale cue, while cross-attention improves baseline performance by 0.43%.The hierarchical decoder further improves extraction accuracy by nearly 2%.
  • Multiscale and cross-attention analysis: Multiscale UrbanSAM focuses more on key semantic regions and captures more small objects and boundary details than single-scale and non-interactive approaches.The comparison is based on class activation mapping visualizations.
  • LoRA configuration: Applying LoRA to both encoder and decoder outperforms decoder-only LoRA and full decoder fine-tuning despite using fewer learnable parameters.The paper characterizes the remote-sensing use of LoRA as largely empirical.
  • LoRA configuration: Rank 4 and applying LoRA only to q and v projections produce the best performance, whereas larger ranks degrade results.The findings suggest that excessive parameter updates increase adaptation complexity and training difficulty.

4) Complex Analysis:

UrbanSAM combines multiresolution adapters, hierarchical cross-attention, and LoRA to provide cue-independent segmentation with favorable accuracy–complexity trade-offs. Its reported experiments target scale-varying urban objects across a global dataset.

  • Complexity analysis: UrbanSAM’s ViT-Base variant reaches 75.80% segmentation accuracy with only 11.20M learnable parameters.This is reported as the highest accuracy among the compared cue-independent CNN, Transformer, and SAM frameworks.
  • Complexity analysis: UrbanSAM improves IoU over fully trained RSBuilding models by 2.21% with ViT-Base and 3.66% with ViT-Large.These versions use 11.1% and 5.8% of the total RSBuilding parameters, respectively.
  • Overall evaluation: The experiments demonstrate flexibility and superior segmentation performance for scale-varying urban objects on a global-scale dataset.The reported object categories are buildings, roads, and water bodies.
  • UrbanSAM design: The model integrates Uscaling-adapters inspired by MRA theory into the trunk encoder to combine domain-specific priors, multiresolution invariance, and pretrained foundation-model knowledge.The adapters are intended to capture invariant properties across scales in globally diverse urban environments.
  • UrbanSAM design: Hierarchical cross-attention and LoRA align and fuse multiscale adapter features with trunk-encoder features, yielding learnable prompts without manual interactive prompting.The resulting model is evaluated on scale-varying buildings, roads, and water bodies.
  • Future work: Future work targets broader training data spanning more types, modalities, and temporal sequences.The stated aim is to improve model versatility for real-world applications.
Loading 2502.15199v1…