PaperlayerView exact revision on arXiv

Source-linked AI summary

HGFormer: Hierarchical Grouping Transformer for Domain Generalized Semantic Segmentation

Jian Ding, Nan Xue, Gui-Song Xia, Bernt Schiele, Dengxin Dai

arXiv:2305.13031v1cs.CV

TL;DR

Semantic segmentation models often face test data from a different distribution than their training data, motivating domain-generalization methods. HGFormer groups pixels into part-level and then whole-level masks and combines classifications at both scales. Across multiple cross-domain settings, it outperforms prior per-pixel and whole-level-mask methods.

  • Problem

    It is unclear whether explicit grouping in a segmentation decoder can improve robustness when models are tested on domains different from their training domain.

    Source: §1 · pp. 1–2

  • Method

    HGFormer groups pixels into part-level masks and then whole-level masks, combining mask-classification predictions across the two scales.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across multiple settings built from seven datasets, HGFormer is significantly better than prior per-pixel classification and whole-level mask segmentation models for out-of-distribution generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Part-level and whole-level mask classification provide complementary scales for robust semantic segmentation under domain shift.

    Source: §1 · p. 2

Abstract

from arXiv · show

Current semantic segmentation models have achieved great success under the independent and identically distributed (i.i.d.) condition. However, in real-world applications, test data might come from a different domain than training data. Therefore, it is important to improve model robustness against domain differences. This work studies semantic segmentation under the domain generalization setting, where a model is trained only on the source domain and tested on the unseen target domain. Existing works show that Vision Transformers are more robust than CNNs and show that this is related to the visual grouping property of self-attention. In this work, we propose a novel hierarchical grouping transformer (HGFormer) to explicitly group pixels to form part-level masks and then whole-level masks. The masks at different scales aim to segment out both parts and a whole of classes. HGFormer combines mask classification results at both scales for class label prediction. We assemble multiple interesting cross-domain settings by using seven public semantic segmentation datasets. Experiments show that HGFormer yields more robust semantic segmentation results than per-pixel classification methods and flat grouping transformers, and outperforms previous methods significantly. Code will be available at https://github.com/dingjiansw101/HGFormer.

1. Introduction

Domain generalization asks whether semantic segmentation trained on source data can remain reliable when test data come from another distribution. HGFormer addresses flat grouping’s cross-domain errors by grouping pixels into part-level and then whole-level masks, classifying both scales, and combining their predictions; experiments report stronger out-of-distribution performance than prior per-pixel and whole-level-mask methods.

1. Introduction

  • Domain generalization trains on source domains and tests on different target domains without requiring target data during training.This setting is described as practically useful for real-world applications, including safety-related ones.Source: §1 · p. 1
  • Flat grouping directly predicts whole-level masks from pixels, but those masks are susceptible to errors under cross-domain settings.Mask classification can aggregate features over regions of the same class, but reliable whole-level masks are challenging under distribution shift.Source: Figure 1 · p. 1, §1 · p. 2
  • HGFormer groups pixels into part-level masks first, then groups those masks into whole-level masks, and combines classification results from both scales.The two mask scales capture complementary information; part-level grouping is intended to make the grouping task less challenging.Source: §1 · p. 2
  • HGFormer is significantly better than prior per-pixel classification and whole-level mask segmentation models across multiple domain-generalization settings.The settings use seven semantic-segmentation datasets, training on one domain and testing on others.Source: §1 · p. 2

2. Related Work

Prior semantic-segmentation work largely uses pixel-level or whole-level mask classification, while part-level mask classification has been less explored. Domain-generalization research in segmentation is limited, and prior work had not studied explicit grouping in the decoder for that setting.

2.1. Semantic Segmentation

  • Deep-learning segmentation commonly uses pixel-level classification or whole-level mask classification, while part-level mask classification has received little attention.The cited part-level approaches either focus on learning masks, require extra masks as input, or were evaluated only under i.i.d. conditions.Source: §2.1 · p. 2

2.2. Domain Generalization

  • Earlier ensemble-based domain-generalization methods can increase runtime or training time, whereas HGFormer’s hierarchical grouping adds neither extra FLOPS nor training time.The passage distinguishes runtime-increasing ensembles from weight-averaging approaches that increase training time.Source: §2.2 · p. 3
  • Prior semantic-segmentation domain-generalization methods include domain randomization and normalization or whitening, while explicit decoder grouping had not been studied.Vision Transformer robustness in out-of-distribution settings had been linked to self-attention’s grouping property.Source: §2.2 · p. 3

3. Methods

HGFormer groups pixels into part-level masks and then whole-level masks, classifies masks at both scales, and adds the resulting segmentation outputs. Its method combines local similarity-based clustering, cross-attention, and losses for part- and whole-level learning.

3. Methods

  • HGFormer groups pixels into part-level masks and then whole-level masks, classifies masks at both scales, and adds the outputs for semantic segmentation.The part-level partition is proposed alongside whole-level masks to enhance robustness.Source: §3 · p. 3, §3.1 · p. 5

3.1. HGFormer

  • Part-level grouping initializes centers from regular grid cells, assigns pixels by similarity to nearby centers, iteratively updates centers, and extracts tokens from feature map V.The method limits comparisons to pixels and their nine nearby centers; the process is repeated L times.Source: §3 · p. 3, §3.1 · p. 4
  • Whole-level grouping uses transformer cross-attention to aggregate part-level tokens, then predicts whole-level masks and class probabilities.The final segmentation combines the part-level and whole-level classification outputs by addition.Source: §3.1 · p. 5
  • Because part-level ground-truth partitions are unavailable and non-unique, training combines part-level cross-entropy and pixel-cluster contrastive losses with matched whole-level dice, mask, and classification losses.The contrastive loss is intended to learn discriminative features for similarity-based part-level grouping.Source: §3.1 · p. 5

3.2. Implementation Details

  • The implementation uses six refinement stages per grouping level, output strides of 1/8 for K and V, and 20k training iterations with batch size 16.Training uses multi-scale augmentation and 512 × 1,024 crops; testing uses original-size images.Source: §3.2 · p. 5

4. Experiments

HGFormer is evaluated across cross-domain settings using seven semantic-segmentation datasets. Its results, ablations, and visualizations compare hierarchical part-to-whole grouping with per-pixel and whole-level approaches.

4.1. Datasets

  • The experiments use real-world, synthetic, and common-corruption datasets, including Cityscapes, BDD, Mapillary, ACDC, GTAV, and SYNTHIA.The common-corruption set expands Cityscapes validation with 16 corruption types at five severity levels.Source: §4.1 · pp. 5–6

4.2. Main Results

  • HGFormer outperforms prior methods across the reported domain-generalization settings, including by 1.5 points over Mask2Former on Cityscapes-to-other-datasets evaluation.The comparisons include Cityscapes-to-ACDC and Cityscapes-to-BDD, Mapillary, GTAV, and Synthia.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • HGFormer is consistently better than Mask2Former across backbones for Mapillary-to-other-datasets evaluation and significantly better on Cityscapes-C level-5 corruption.Cityscapes-C level 5 includes 16 types of artificial corruption.Source: §4.2 · pp. 6–7, Table 3 · p. 6, Table 4 · p. 7

4.3. Ablation Studies

  • Performance generally improves as part-level mask-classification iterations increase, saturating at iteration four; the final stage is lower than the second-last stage.The authors hypothesize that gradients from whole-level grouping affect the final stage because it uses the final part-level tokens; removing whole-level grouping during training slightly raises that stage’s result.Source: §4.3 · p. 7, Table 5 · p. 7
  • Part-level classification significantly outperforms whole-level classification, while combining both further improves performance, indicating complementary predictions.The passage reports this comparison across domain-generalization evaluations.Source: §4.3 · p. 7, Table 6 · p. 8

4.4. Visualization Analyses

  • Under increasing Gaussian-noise severity, part-level masks remain more stable than whole-level masks, which fail at level 5; visualizations also show fewer segmentation errors than Mask2Former.Reasonable part-level masks appear even with random or ImageNet-pretrained weights, while trained weights yield more accurate boundaries between categories; boundaries within a class need not be unique for correct semantic output.Source: §4.4 · pp. 7–8, Figure 6 · p. 8, Figure 7 · p. 8

5. Conclusion and Future work

HGFormer generates hierarchical image partitions and combines part-level and whole-level mask classification into an ensemble; the authors report robustness on out-of-distribution images. Future work includes exploring more complex scale fusion and automatically balancing classification loss weights.

5. Conclusion and Future work

  • HGFormer performs part-level and whole-level mask classification, with the final segmentation result formed as an ensemble of both results.The model generates image partitions in a hierarchical structure.Source: §5 · p. 8
  • The authors report that HGFormer is robust on out-of-distribution images.Source: §5 · p. 8
  • Future work includes exploring more complicated fusion across classification scales and automatically balancing their loss weights.The paper describes the model as a kind of multi-task learning.Source: §5 · p. 8

Related papers

Loading 2305.13031v1…