PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Content-enhanced Mask Transformer for Domain Generalized Urban-Scene Segmentation

Qi Bi, Shaodi You, Theo Gevers

arXiv:2307.00371v5cs.CV

TL;DR

Urban-scene segmentation must generalize across varied styles even when semantic categories remain similar. CMFormer combines mask-level and down-sampled features through content-enhanced mask attention; experiments report up to 14.00% mIoU improvement over existing CNN-based methods.

  • Problem

    Urban-scene segmentation faces varied landscapes, weather, and lighting, while models trained on one dataset may encounter unfamiliar scenes at inference.

    Source: Introduction · p. 1

  • Method

    CMFormer combines mask-level image features with down-sampled counterparts through content-enhanced mask attention and multi-resolution integration.

    Source: Introduction · p. 2

  • Results

    14.00% mIoU improvement over state-of-the-art CNN-based methods and up to 1.7% over HGFormer are reported for CMFormer.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Experiments across multiple settings support the feasibility of adapting mask Transformers to domain-generalized USSS.

    Source: Conclusion · p. 7

    The paper notes limited performance for existing approaches.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Domain-generalized urban-scene semantic segmentation (USSS) aims to learn generalized semantic predictions across diverse urban-scene styles. Unlike domain gap challenges, USSS is unique in that the semantic categories are often similar in different urban scenes, while the styles can vary significantly due to changes in urban landscapes, weather conditions, lighting, and other factors. Existing approaches typically rely on convolutional neural networks (CNNs) to learn the content of urban scenes. In this paper, we propose a Content-enhanced Mask TransFormer (CMFormer) for domain-generalized USSS. The main idea is to enhance the focus of the fundamental component, the mask attention mechanism, in Transformer segmentation models on content information. To achieve this, we introduce a novel content-enhanced mask attention mechanism. It learns mask queries from both the image feature and its down-sampled counterpart, as lower-resolution image features usually contain more robust content information and are less sensitive to style variations. These features are fused into a Transformer decoder and integrated into a multi-resolution content-enhanced mask attention learning scheme. Extensive experiments conducted on various domain-generalized urban-scene segmentation datasets demonstrate that the proposed CMFormer significantly outperforms existing CNN-based methods for domain-generalized semantic segmentation, achieving improvements of up to 14.00\% in terms of mIoU (mean intersection over union). The source code is publicly available at \url{https://github.com/BiQiWHU/CMFormer}.

Introduction

Domain-generalized urban-scene segmentation must handle varied scene styles even when semantics remain similar across domains. CMFormer addresses this by combining mask-level and down-sampled features, with experiments reporting gains over existing methods.

Introduction

  • Scene changes in landscape, weather, and lighting make urban-scene semantic segmentation challenging, while unfamiliar scenes can appear at inference.In domain-generalized USSS, styles vary substantially while semantics largely remain consistent.Source: Introduction · p. 1
  • Existing approaches use style normalization or whitening, or adverse-domain training, but the latter methods do not specifically focus on urban styles and have limited performance.Source: Introduction · pp. 1–2
  • CMFormer’s content-enhanced mask attention jointly uses high-resolution mask representations and down-sampled features with complementary content and style properties.The mechanism fuses the original image feature with its down-sampled counterpart to learn more robust content.Source: Introduction · p. 2
  • The proposed mechanism can be integrated into mask-level segmentation Transformers and adapted to multi-resolution features.The paper names the resulting domain-generalized USSS framework CMFormer.Source: Introduction · p. 2
  • 14.00% mIoU improvement over state-of-the-art CNN-based methods and up to 1.7% over HGFormer are reported for CMFormer.The experiments also report state-of-the-art performance on synthetic-to-real and clear-to-adverse generalization.Source: Introduction · p. 2

Related Work

Related work spans general domain generalization, domain-generalized segmentation, and mask-Transformer segmentation. For urban scenes, prior approaches chiefly manipulate style or incorporate external images and are described as CNN-based.

Related Work

  • General domain-generalization research includes unsupervised image retrieval, generalization to homogeneous domains, and single-source learning.Source: Related Work · p. 2
  • Domain-generalized segmentation studies generalization to unseen targets in settings including in-the-wild, scribble, and multi-source images.Source: Related Work · p. 2
  • Urban-scene methods use normalization or whitening on the training domain, while other segmentation approaches use external images for diverse styles or multi-scale content consistency.Source: Related Work · p. 2
  • The paper characterizes prior domain-generalized USSS methods as CNN-based and describes mask-Transformer segmentation approaches including Segmenter, MaskFormer, and Mask2Former.Source: Related Work · p. 2

Preliminary

The preliminary analysis frames domain generalization as a worst-case problem and describes urban scenes as having similar content but varied styles. It motivates combining mask-level and down-sampled features to stabilize content while handling style variation.

Preliminary

  • Domain generalization is formulated as a worst-case problem involving source and unseen target domains, with a domain-distance constraint threshold.The passage defines θ as model parameters, D as the source-target distance, and ρ as the threshold.Source: Preliminary · pp. 2–3
  • Across urban-scene domains, content patterns cluster closely while styles may separate into distinct clusters.The stated objective is stable content with styles uniformly distributed across domains.Source: Preliminary · p. 3, Figure 3 · p. 3
  • Mask-level representations capture pixel semantics but are more style-sensitive; their down-sampled counterparts are less capable of pixel-level content representation but more style-resilient.Source: Preliminary · p. 3
  • Unlike methods that focus on decoupling styles, the proposed approach combines mask-level content representation with the style-handling ability of down-sampled features.Source: Preliminary · p. 3

Methodology

The method extends mask attention with high- and low-resolution feature branches, then fuses their complementary representations. It integrates this mechanism into a CMFormer architecture built on inherited Mask2Former components.

Recap on Mask Attention

  • Mask attention uses query features to form segmentation masks, while a binary attention mask from the previous layer filters foreground regions.The previous mask is resized and thresholded at 0.5; the initial mask is binarized and resized from the input query features.Source: Recap on Mask Attention · p. 3

Exploiting High-Resolution Properties

Exploiting Low-Resolution Properties

  • The low-resolution branch is formed by average-pooling image features and uses self-attention to exploit greater robustness to style variation.Its representation is less capable of capturing pixel-level intricacies.Source: Exploiting Low-Resolution Properties · p. 4

Content-enhanced Fusion

  • Content-enhanced fusion concatenates the high- and low-resolution outputs and applies a linear layer to produce the decoder output.The joint representations are intended to help masks focus on scene content while reducing sensitivity to style variation.Source: Content-enhanced Fusion · p. 4

Network Architecture and Implementation Details

  • The framework uses a Swin Transformer backbone and an MSDeformAttn image decoder that upsamples features at ×32, ×16, ×8, and ×4, then fuses the 1/4-resolution map with Transformer-decoder features for dense prediction.The Transformer decoder is inherited directly from Mask2Former; loss weights are λce = λdice = 5.0 and λcls = 2.0, and training ends after 50 epochs.Source: Network Architecture and… · pp. 4–5

Experiment

Experiments evaluate CMFormer across source-to-unseen-domain settings using five datasets with shared semantic categories but differing scene styles. The reported comparisons show gains over CNN-based methods, while ablations and qualitative predictions examine the contribution of content enhancement.

Dataset & Evaluation Protocols

  • The experiments use five datasets with the same 19 semantic categories; models are trained on one source and validated on the other four, with mIoU averaged across three runs.The protocol includes G→C, B, M, S; S→C, B, M, G; and C→B, M, G, S.Source: Dataset & Evaluation Protocols · p. 5

Comparison with State-of-the-art

  • In the GTA5-source setting, CMFormer improves mIoU over existing state-of-the-art CNN methods by 10.66%, 9.45%, 14.00%, and 12.46% on real target domains C, B, M, and S, respectively.Source: Comparison with State-of-the-art · p. 5, Table 1 · p. 5
  • On SYNTHIA-source evaluation, CMFormer gains 5.67%, 8.73% and 11.49% mIoU over the best CNN methods, but SAW leads by 1.80% on BDD100K.With CityScapes as source and a Swin-Base backbone, CMFormer gains 6.32%, 10.43%, 9.50% and 12.11% mIoU on BDD100K, Mapillary, GTA5 and SYNTHIA; against Swin-Large HGFormer, gains are 1.1%, 1.5%, 1.3% and 1.7%.Source: Comparison with State-of-the-art · p. 5, Table 2 · p. 6, Table 3 · p. 6
  • When trained on synthetic domains G+S and validated on real-world domains B, C, and M, CMFormer significantly outperforms instance-normalization-, whitening-transformation-, and adversarial-domain-training-based methods by >10% mIoU.Source: Comparison with State-of-the-art · p. 6, Table 4 · p. 6
  • On ACDC adverse domains, CityScapes-pretrained CMFormer exceeds prior methods by up to 10.3% on fog, 0.5% on night, 11.6% on rain and 11.1% on snow.The model is directly evaluated on the four unseen conditions.Source: Comparison with State-of-the-art · p. 6, Table 5 · p. 6

Ablation Studies

  • Adding content enhancement at ×32 yields gains of 0.74%, 1.43%, 0.37% and 0.64% on B, M, G and S; adding it at ×16 brings further gains of 1.93%, 2.17%, 0.12% and 0.58%.Further enhancement at ×8 also significantly affects generalization; similar observations are reported for S→C, B, M, G.Source: Ablation Studies · pp. 6–7, Table 6 · p. 6

Quantitative Segmentation Results

Conclusion

The paper proposes content-enhanced mask attention and integrates it with multi-resolution features in CMFormer for domain-generalized USSS. Experiments across multiple settings report superior performance to existing domain-generalized USSS methods.

  • CMFormer combines content-enhanced mask attention with multi-resolution features to address style variation and robust content representation, and outperforms existing domain-generalized USSS methods across multiple settings.Source: Conclusion · p. 7

Related papers

Loading 2307.00371v5…