PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-City Matters: A Multimodal Remote Sensing Benchmark Dataset for Cross-City Semantic Segmentation using High-Resolution Domain Adaptation Networks

Danfeng Hong, Bing Zhang, Hao Li, Yuxuan Li, Jing Yao, Chenyu Li, Martin Werner, Jocelyn Chanussot, Alexander Zipf, Xiao Xiang Zhu

arXiv:2309.16499v2cs.CVeess.IV

TL;DR

Remote-sensing AI performs well in individual urban environments but faces bottlenecks across cities because diverse information and generalizable solutions are limited. The paper introduces the C2Seg multimodal benchmark and HighDAN, a high-resolution domain-adaptation network, and reports superior segmentation performance and generalization against state-of-the-art competitors. The authors also identify large-scale hyperspectral imaging as an issue requiring further attention.

  • Problem

    Remote-sensing AI models face performance bottlenecks across cities or regions because diverse remote-sensing information and highly generalizable solutions are limited.

  • Method

    The paper builds C2Seg, a multimodal benchmark with hyperspectral, multispectral, and SAR data, and proposes HighDAN with high-to-low-resolution fusion, adversarial learning, and Dice loss.

  • Results

    HighDAN achieves superior segmentation performance and generalization ability compared with state-of-the-art competitors on C2Seg.

  • Takeaways & Limitations

    C2Seg and the associated toolbox support research on cross-city semantic segmentation and interpretation of urban environments across cities.

  • Takeaways & Limitations

    The development of hyperspectral remote sensing, particularly its large-scale application, remains an issue requiring urgent attention and exploration.

Abstract

from arXiv · show

Artificial intelligence (AI) approaches nowadays have gained remarkable success in single-modality-dominated remote sensing (RS) applications, especially with an emphasis on individual urban environments (e.g., single cities or regions). Yet these AI models tend to meet the performance bottleneck in the case studies across cities or regions, due to the lack of diverse RS information and cutting-edge solutions with high generalization ability. To this end, we build a new set of multimodal remote sensing benchmark datasets (including hyperspectral, multispectral, SAR) for the study purpose of the cross-city semantic segmentation task (called C2Seg dataset), which consists of two cross-city scenes, i.e., Berlin-Augsburg (in Germany) and Beijing-Wuhan (in China). Beyond the single city, we propose a high-resolution domain adaptation network, HighDAN for short, to promote the AI model's generalization ability from the multi-city environments. HighDAN is capable of retaining the spatially topological structure of the studied urban scene well in a parallel high-to-low resolution fusion fashion but also closing the gap derived from enormous differences of RS image representations between different cities by means of adversarial learning. In addition, the Dice loss is considered in HighDAN to alleviate the class imbalance issue caused by factors across cities. Extensive experiments conducted on the C2Seg dataset show the superiority of our HighDAN in terms of segmentation performance and generalization ability, compared to state-of-the-art competitors. The C2Seg dataset and the semantic segmentation toolbox (involving the proposed HighDAN) will be available publicly at https://github.com/danfenghong.

1. Introduction

Cross-city remote-sensing segmentation remains constrained by limited multimodal benchmarks and models optimized for individual urban environments. The paper addresses this gap with C2Seg and HighDAN, combining multimodal data, domain adaptation, high-resolution fusion, and Dice loss.

  • Cross-city studies progress slowly because high-quality multimodal benchmark datasets remain scarce and existing methods prioritize single-city performance.
  • Experiments report that domain-adaptation approaches outperform semantic-segmentation algorithms without cross-domain knowledge transfer.
  • C2Seg provides a multimodal benchmark for cross-city semantic segmentation across Berlin-Augsburg and Beijing-Wuhan.The datasets include hyperspectral, multispectral, and SAR data from satellite missions.
  • C2Seg is presented as the first benchmark dataset for cross-city multimodal remote-sensing image segmentation using three modalities.
  • HighDAN uses adversarial learning to bridge representation gaps between different urban environments and transfer knowledge across domains.Its architecture is based on parallel high-to-low-resolution subnetworks that capture multiscale representations.
  • Dice loss is embedded in HighDAN to reduce the impact of class-sample imbalance in multi-city studies.

2. Related Work

Related work has established strong remote-sensing segmentation and multimodal fusion methods, mainly for individual scenes. Cross-region research increasingly uses domain adaptation, but generalization across cities remains challenging and motivates combining multimodal data with adaptation methods.

  • Deep-learning remote-sensing methods have achieved accurate land-cover mapping, but many studies focus on individual scenes and single modalities.
  • Existing models often encounter performance bottlenecks in complex scenes because they lack diverse modality information.
  • Multimodal methods exploit complementary information from different platforms and sensors for semantic labeling and urban-scene understanding.
  • Prior multimodal segmentation research has explored deep features, multiscale FCNs, fusion strategies, CNNs, HR-Net, and transformer-based architectures.
  • Cross-city research applies domain adaptation through adversarial learning, transfer learning, distribution alignment, and consistency constraints.
  • The paper identifies a combined need for large-scale multimodal cross-city benchmarks and methods blending multimodal data with domain adaptation.

3. C2Seg: A Multimodal RS Dataset for Cross-City Semantic Segmentation

C2Seg is a multimodal benchmark for cross-city semantic segmentation, covering two city-pair scenes in Germany and China with hyperspectral, multispectral, and SAR data. The datasets provide aligned imagery and labeled land-use and land-cover categories for studying urban scene understanding across cities.

  • The benchmark integrates hyperspectral, multispectral, and SAR remote-sensing data across the studied city pairs.
  • C2Seg comprises two cross-city scenes: Berlin–Augsburg in Germany and Beijing–Wuhan in China.
  • C2Seg-AB: C2Seg-AB combines EnMAP hyperspectral, Sentinel-2 multispectral, and Sentinel-1 SAR imagery for Berlin and Augsburg.
  • C2Seg-BW: C2Seg-BW combines hyperspectral, multispectral, and SAR data from Gaofen-series satellites for Beijing and Wuhan.
  • The datasets maintain a 10-meter ground sampling distance and include 13 land-use and land-cover semantic categories.
  • C2Seg is presented as a large-scale benchmark tailored to cross-city multimodal remote-sensing semantic segmentation.

4. HighDAN: High-Resolution Domain Adaptation Network

HighDAN addresses cross-city semantic segmentation by combining a multimodal high-resolution network with unsupervised adversarial domain adaptation. Its design preserves high-resolution representations while aligning source and target domains using multimodal remote-sensing data.

  • Motivation: Conventional serial convolutional architectures can produce low-resolution feature maps that lose edge details, texture, and spatial information.
  • HR-Net: HR-Net maintains high-resolution representations through parallel high-to-low-resolution streams and exchanges information across resolutions.
  • Architecture: HighDAN combines a multimodal encoder, adversarial domain adaptation, and convolution decoder for cross-city semantic segmentation.
  • Adversarial Domain Adaptation: HighDAN aligns source and target representations through adversarial learning to transfer invariant semantic features across domains.
  • Training Objective: Dice loss is incorporated to weaken class-imbalance effects arising in cross-city settings.
  • Multimodal Encoder: The multimodal encoder processes hyperspectral, multispectral, and SAR inputs through modality streams and a shared high-resolution subnetwork.

16 end

HighDAN is trained by combining segmentation, feature-level adversarial, and category-level adversarial objectives. Its Dice component addresses imbalanced training samples, while adversarial losses align source and target domains.

  • Model training: HighDAN optimizes an overall objective combining segmentation, feature-level adversarial, and category-level adversarial losses.The penalty parameters λ and µ balance the objective terms.
  • Segmentation loss: The segmentation loss combines multi-class cross-entropy with Dice loss to alleviate effects from imbalanced training samples.LMCE weights pixels equally, whereas LDice addresses class imbalance.
  • Adversarial learning: Feature-level adversarial learning aligns source and target feature maps extracted by the multimodal high-resolution encoder.The generator extracts source and target representations, while a discriminator is optimized in the domain-adaptation task.
  • Feature stabilization: Target feature maps are updated with an attention map to improve their stability during optimization.The update is V_t^new = V_t + V_t ⊙ α, where α denotes the attention map.
  • Adversarial learning: Category-level adversarial learning operates on source and target decoder outputs to improve global adaptation ability.The adversary is applied to category-level representations rather than only feature maps.

5.1. Experimental Preparation

The experiments configure HighDAN as a high-resolution multimodal segmentation network and evaluate it against standard and domain-adaptation baselines on two cross-city datasets. Performance is measured with OA, mIoU, and mF1, alongside complexity statistics.

  • Implementation: Training uses PyTorch, Adam optimization, batch size 16, and learning rates of 0.0001 for both the segmentation network and discriminator.Training runs for 6000 epochs on C2Seg-AB and 10000 epochs on C2Seg-BW.
  • HighDAN architecture: HighDAN uses convolution and bottleneck blocks, three high-resolution encoding modules, adversarial fusion, and a convolution decoder.Each encoding module contains four basic HR blocks, and the decoder contains four decoding blocks.
  • Evaluation setup: The evaluation reports overall accuracy, mean intersection over union, and mean F1 score for cross-city semantic segmentation.The comparison also includes per-class F1 scores, FLOPs, and parameter counts.
  • Compared methods: The benchmark compares DeepLabv3, SegNet, FastFCN, AdaptSeg, DSAN, DualHR, SegFormer, and HighDAN.The compared methods include models without domain adaptation and models embedding domain-adaptation strategies.
  • Experimental protocol: Experiments use the same network configurations as the original literature, with slight parameter adjustments for multimodal remote-sensing data.This comparison protocol is applied across the segmentation experiments.

5.2. Quantitative Evaluation on C2Seg Datasets

Domain-adaptation methods generally outperform models that ignore cross-domain shifts, while HighDAN achieves the strongest quantitative performance across the two C2Seg datasets. SegFormer is consistently competitive, but HighDAN delivers larger gains over it.

  • Overall comparison: Models embedding domain adaptation generally outperform DeepLabv3 and SegNet, which do not account for data shifts between domains.The pattern holds across the reported cross-city comparisons.
  • Domain-adaptation comparison: FastFCN and AdaptSeg trail DSAN by average decreases of 3%–4% OA, 2%–3% mIoU, and 2%–4% mF1.Their per-category F1 scores remain broadly comparable, with differences concentrated in categories such as Pastures, Forests, and Shrub.
  • Backbone comparison: DualHR improves OA by 13% over DSAN on C2Seg-BW, while its performance is essentially identical to DSAN on C2Seg-AB.This indicates dataset-dependent variation among high-resolution architectures.
  • Overall comparison: SegFormer achieves the second-highest results across all evaluation indices and remains competitive and stable on both C2Seg datasets.This makes SegFormer the principal quantitative comparison for HighDAN.
  • Overall comparison: 4.26%, 4.60%, and 5.90% gains in OA, mIoU, and mF1 respectively distinguish HighDAN from SegFormer on C2Seg-AB.HighDAN also achieves over 6% OA increase on C2Seg-BW.
  • Dataset difficulty: C2Seg remains challenging because some categories are hardly identified, with certain class segmentation results at 0% or approximately 0%.The limitation is reported for the quantitative evaluation of the dataset.

5.3. Visual Comparison on C2Seg Datasets

Visual analyses show that HighDAN produces segmentation maps closer to ground truth while preserving fine-grained and small-scale object details across cities. Nonetheless, several land-cover categories remain difficult in particular sub-images.

  • Scene-wide comparison: HighDAN produces comparatively realistic maps closer to ground truth than SegFormer, with water bodies, urban areas, and forests nearly matching ground-truth profiles.The comparison covers the Berlin and Wuhan scenes.
  • Sub-region comparison: HighDAN and DualHR preserve small-scale features and intricate shapes in man-made objects without losing important details.This advantage is highlighted in local sub-region visualizations.
  • Sub-region comparison: DeepLab and SegNet lose substantial detail and struggle with tiny, irregular objects or structures.Other methods improve recognition of some shapes but retain limitations in accuracy and boundary segmentation.
  • Cross-city robustness: HighDAN adapts to imaging-condition changes and cross-city object-appearance variability, improving segmentation accuracy and robustness.This interpretation is based on the visual comparison of local results.
  • Remaining limitations: Some categories are almost entirely misclassified in certain sub-images, including Artificial vegetated areas, Open spaces with no vegetation, Inland wetlands, and Shrub.The authors identify continued need for improvement in high-resolution feature extraction and domain adaptation.
  • Class activation maps: HighDAN captures high-level semantic information with precise class activation, including for small classes such as Street Network.The CAM analysis is conducted on the C2Seg-AB datasets.

5.4. Ablation Study

The ablation study evaluates HighDAN’s multimodal HR-Net backbone, domain-adaptation modules, and Dice loss on C2Seg-AB. Results indicate gains from high-resolution feature extraction, domain adaptation, and class-balancing loss, alongside stable training behavior.

  • Component ablation: HR-Net improves over the SegNet baseline by 6.45% OA, 6.30% mIoU, and 8.21% mF1.The improvement follows integration of bottleneck and high-resolution feature extraction modules.
  • Component ablation: HighDAN’s adversarial domain adaptation bridges cross-city representation gaps from feature-level and category-level perspectives.Without domain adaptation, cross-city segmentation encounters a performance bottleneck.
  • Component ablation: Dice loss further improves cross-city semantic segmentation by at least 1.2% OA over the configuration without Dice loss.The loss balances samples from different categories to address class imbalance.
  • Data modality comparison: With hyperspectral data, OA, mIoU, and mF1 are 57.66%, 35.19%, and 24.76%, versus 53.91%, 31.81%, and the corresponding no-HS configuration.The comparison evaluates scenarios with and without hyperspectral data.
  • Robustness analysis: Training loss consistently decreases, while OA, mIoU, and mF1 show similar trends across individual C2Seg-AB datasets.The reported curves indicate stable convergence and robustness during learning.

6. Conclusion

The paper addresses the limited cross-city generalization of urban remote-sensing segmentation by introducing the C2Seg benchmark and HighDAN. Experiments report strong segmentation performance, while future work targets broader geographic coverage, hyperspectral scaling, and explicit knowledge embedding.

  • Motivation: Single-city segmentation methods often do not transfer effectively across cities or regions, hindering urban planning and management applications.The paper focuses on cross-city semantic segmentation of cross-modal satellite images.
  • Contributions: C2Seg is a multimodal benchmark dataset designed for cross-city land-cover segmentation using more diverse remote-sensing information.It addresses insufficient discriminative information from relying only on single-modality data.
  • Contributions: HighDAN embeds adversarial domain adaptation into HR-Net with Dice loss to improve cross-city accuracy and generalization while reducing class-imbalance effects.The architecture combines high-resolution processing, domain adaptation, and Dice loss.
  • Results: HighDAN achieves the best segmentation performance and beats other state-of-the-art competitors on almost all important indices.The experiments were conducted on the C2Seg datasets.
  • Resources and impact: The C2Seg benchmark datasets and corresponding source code are planned for public release to support research on cross-city interpretation of urban environments.The paper connects these resources with progress in semantic segmentation across cities.
  • Future work: Future work includes extending C2Seg nationally and globally, addressing large-scale hyperspectral imaging constraints, and incorporating explicit knowledge such as geometric and urban morphological priors.The authors also call for more advanced and accessible AI models.
Loading 2309.16499v2…