PaperlayerView exact revision on arXiv

Source-linked AI summary

Global and Local Texture Randomization for Synthetic-to-Real Semantic Segmentation

Duo Peng, Yinjie Lei, Lingqiao Liu, Pingping Zhang, Jun Liu

arXiv:2108.02376v2cs.CV

TL;DR

Synthetic-to-real segmentation faces texture bias and large synthetic–real texture differences, while target-domain data may be unavailable for Domain Generalization. The paper combines global and local texture randomization with consistency regularization and painting selection, and reports state-of-the-art performance across multiple transfer settings.

  • Problem

    CNN texture bias and synthetic–real texture differences challenge SRSS, while real-style transfer requires manually selected styles when target-domain data is unavailable.

    Source: §I · p. 2

  • Method

    For Domain Generalization SRSS, GTR and LTR randomize source textures with unreal paintings, CGL harmonizes the variants, and TCPS selects paintings by texture complexity.

    Source: §I · pp. 1–2

  • Results

    The method reports superior performance over state-of-the-art methods across GTA5/SYNTHIA-to-Cityscapes/BDDS/Mapillary settings.

    Source: §I · p. 3

  • Takeaways & Limitations

    The approach aims to reduce reliance on texture and promote learning domain-invariant shape and spatial-layout cues under global texture changes.

    Source: §I · p. 2

Abstract

from arXiv · show

Semantic segmentation is a crucial image understanding task, where each pixel of image is categorized into a corresponding label. Since the pixel-wise labeling for ground-truth is tedious and labor intensive, in practical applications, many works exploit the synthetic images to train the model for real-word image semantic segmentation, i.e., Synthetic-to-Real Semantic Segmentation (SRSS). However, Deep Convolutional Neural Networks (CNNs) trained on the source synthetic data may not generalize well to the target real-world data. In this work, we propose two simple yet effective texture randomization mechanisms, Global Texture Randomization (GTR) and Local Texture Randomization (LTR), for Domain Generalization based SRSS. GTR is proposed to randomize the texture of source images into diverse unreal texture styles. It aims to alleviate the reliance of the network on texture while promoting the learning of the domain-invariant cues. In addition, we find the texture difference is not always occurred in entire image and may only appear in some local areas. Therefore, we further propose a LTR mechanism to generate diverse local regions for partially stylizing the source images. Finally, we implement a regularization of Consistency between GTR and LTR (CGL) aiming to harmonize the two proposed mechanisms during training. Extensive experiments on five publicly available datasets (i.e., GTA5, SYNTHIA, Cityscapes, BDDS and Mapillary) with various SRSS settings (i.e., GTA5/SYNTHIA to Cityscapes/BDDS/Mapillary) demonstrate that the proposed method is superior to the state-of-the-art methods for domain generalization based SRSS.

I. INTRODUCTION

Domain Generalization based SRSS trains without target-domain data, addressing settings where that data is unavailable. The paper proposes global and local texture randomization, alongside mechanisms to coordinate and guide those augmentations.

I. INTRODUCTION

  • Domain Generalization trains using labeled source images without target-domain data, addressing scenarios where the target domain is unknown beforehand.Domain Adaptation instead assumes the target domain is known and uses its unlabeled data during training.Source: §I · p. 1, Figure 1 · p. 1
  • GTR replaces source-image textures with unreal paintings to reduce texture reliance and encourage learning domain-invariant cues such as shape and spatial layout.The paper describes unreal painting textures as irregular and indistinct, rather than regular and human-identifiable.Source: §I · p. 2
  • LTR targets cases where texture differences occupy only local regions, using randomly bounded regions to stylize source images partially.The paper notes local differences especially in vegetation, terrain, and buildings.Source: §I · p. 2, Figure 4 · p. 3
  • CGL is proposed to harmonize global and local texture randomization during training.The authors state that applying both mechanisms may direct the network toward distinct learning directions and impede convergence.Source: §I · pp. 2–3
  • TCPS measures texture complexity to select paintings, because performance is reported to be sensitive to complexity rather than painting style.The paper says different styles with the same texture complexity lead to similar performance.Source: §I · pp. 2–3
  • The method reports superior segmentation performance over state-of-the-art methods across GTA5/SYNTHIA-to-Cityscapes/BDDS/Mapillary settings.The introduction describes evaluation under various SRSS settings.Source: §I · p. 3

II. RELATED WORKS

Related work moves from traditional optimization methods to deep-learning segmentation architectures. Deep methods build on fully convolutional mapping and add mechanisms for preserving object details and combining features.

A. Semantic Segmentation

  • Traditional semantic segmentation used graph cuts, belief propagation, message passing, and random walks, while complex natural-image energy formulations motivated higher-order methods.Examples include maximizing quadratic submodular energy and minimizing explicit higher-order energy functions.Source: A. Semantic Segmentation · p. 3
  • Fully Convolutional Networks map RGB images end-to-end to semantic pixels, with later methods modifying FCNs or adding detail-preserving and feature-fusion mechanisms.Examples include dilated convolution, Markov and conditional random fields, large kernels, semantic enhancement, and KSAC.Source: A. Semantic Segmentation · p. 3

B. Domain Adaptation

Domain adaptation aligns synthetic and real data, whereas domain generalization relies on labeled source-domain images. Data-enhancement methods provide another route through image and regularization-based augmentation.

B. Domain Adaptation

  • Domain adaptation methods align source and target through features, image stylization, or both, using distribution-, class-, patch-, layer-, or label-level perspectives.Feature-level methods minimize feature distances or use adversarial learning; pixel-level methods stylize source images toward realistic target images.Source: B. Domain Adaptation · p. 3

C. Domain Generalization

  • Domain generalization uses labeled source-domain images alone, but semantic-segmentation work is limited and includes layer-normalized and data-enhanced approaches.Prior data-enhanced generalization randomizes synthetic images with real-image styles; this paper proposes randomized unreal images through data enhancement.Source: C. Domain Generalization · pp. 3–4

D. Data Enhancement

  • Data augmentation creates similar extra data through image operations, while regularization methods perturb or combine training samples and image regions.The proposed framework adds source, GTR, and LTR streams, with consistency regularization between the GTR and LTR streams.Source: D. Data Enhancement · p. 4, Figure 7 · p. 5

III. METHODOLOGY

The framework trains on source images alongside globally and locally stylized versions, using consistency between the stylized streams. TCPS scores paintings by unsmooth-area proportion and selects paintings in a reported favorable range.

A. Overview of the Proposed Framework

  • The framework feeds source, GTR, and LTR images to three segmentation streams, applies consistency between GTR and LTR predictions, and removes enhancement modules at test time.TCPS selects a painting subset, then one of 15 paintings is randomly chosen per iteration.Source: A. Overview of the Proposed Framework · p. 5

B. Texture Complexity based Painting Selection (TCPS)

C. Global Texture Randomization (GTR)

GTR transfers a randomly selected painting’s texture to a source image while retaining its content. It uses encoder features, AdaIN fusion, and a decoder to produce the stylized image.

C. Global Texture Randomization (GTR)

  • GTR uses a randomly sampled painting to replace source-image texture, feeding both images through a pretrained Siamese encoder for feature extraction.The architecture uses a pretrained style-translation network with an encoder, AdaIN fusion, and decoder.Source: C. Global Texture Randomization (GTR) · p. 6, Figure 10 · p. 6
  • AdaIN fuses source and painting feature maps using their spatial means and variances to form the transferred feature map Fu.The pretrained decoder inverts Fu to produce the GTR image.Source: C. Global Texture Randomization (GTR) · p. 6
  • GTR increases training-set texture variety and robustness to global texture change, but does not address cases where only local texture differences occur.The latter motivates the paper’s subsequent Local Texture Randomization mechanism.Source: C. Global Texture Randomization (GTR) · p. 6

D. Local Texture Randomization (LTR)

LTR creates diverse images with locally randomized textures by mixing the raw image with a GTR-generated image under a random mask. The mask is formed by smoothing noise and thresholding it to control the white-region proportion.

D. Local Texture Randomization (LTR)

  • LTR targets cases where texture differences occur in only some image areas, using random-shaped masks to create local texture randomization.The method uses a mask generator to create masks with diverse shapes across iterations.Source: D. Local Texture Randomization (LTR) · p. 6
  • Each iteration samples λ uniformly from 4 to 16, varying the mask generated from noise.The paper identifies λ as controlling the Gaussian-kernel standard deviation and illustrates masks at resolutions of 640×640.Source: D. Local Texture Randomization (LTR) · p. 6, Figure 12 · p. 7, Figure 11 · p. 7
  • A Gaussian-smoothed noise image is thresholded to form a mask whose white-region proportion is controlled by p.The threshold is computed from the smoothed image’s mean and standard deviation using the inverse error function, avoiding pixel ranking.Source: D. Local Texture Randomization (LTR) · pp. 6–7
  • LTR mixes the raw image and GTR-generated image pixel-wise under mask M to produce a locally randomized training image.Mask values select which source contributes at each pixel.Source: D. Local Texture Randomization (LTR) · p. 7

E. Consistency between GTR and LTR (CGL)

CGL applies an L1 consistency loss between GTR and LTR outputs to harmonize their training. Network training combines segmentation losses on raw, GTR, and LTR images with that consistency loss.

E. Consistency between GTR and LTR (CGL)

  • CGL uses L1 distance to regularize GTR and LTR training because the methods may fail to converge together or lead to unstable, divergent training directions.The compared feature maps come from the segmentation network’s last convolutional layer before softmax.Source: E. Consistency between GTR and LTR (CGL) · p. 7

F. Network Training

  • The segmentation loss sums weighted cross-entropy losses for the raw image, GTR-generated image, and LTR-generated image.Each image is paired with the same ground-truth labels.Source: F. Network Training · p. 7
  • The final loss trades off segmentation and consistency losses with β set to 1e-5, while testing uses only the segmentation network.GTR, LTR, and CGL are omitted at test time.Source: F. Network Training · p. 7

IV. EXPERIMENTAL SETUPS

The experiments use GTA5 and SYNTHIA as source datasets and Cityscapes, BDDS, and Mapillary as target datasets. The six settings pair each source with each target.

A. Datasets Description

  • GTA5 contains 24,966 labeled images across 19 classes, while the SYNTHIA-RANDCITYSCAPES subset contains 9,400 images across 16 categories.GTA5 images are 1914 × 1052; SYNTHIA images are 1280 × 760.Source: A. Datasets Description · p. 7
  • The target datasets are Cityscapes, BDDS, and Mapillary, with 19, 34, and 19/16 used overlapping classes, respectively.Cityscapes has 5,000 images; BDDS has 7,000 training, 1,000 validation, and 2,000 test images; Mapillary has 18,000 training and 2,000 validation images.Source: A. Datasets Description · pp. 7–8
  • The six SRSS settings pair GTA5 or SYNTHIA with Cityscapes, BDDS, or Mapillary as the target.These are GTA5→Cityscapes, GTA5→BDDS, GTA5→Mapillary, SYNTHIA→Cityscapes, SYNTHIA→BDDS, and SYNTHIA→Mapillary.Source: A. Datasets Description · p. 8

B. Implementation Details

Implementation uses VGG-16, ResNet-50, or ResNet-101 segmentation networks and evaluates performance with mIoU. Training uses cropped and resized images, ImageNet initialization, and SGD for 200,000 iterations.

B. Implementation Details

  • The study evaluates VGG-16, ResNet-50, and ResNet-101 using IoU and its category mean, mIoU.A higher mIoU is positively correlated with segmentation performance.Source: B. Implementation Details · p. 8
  • GTA5 panoramas are cropped into two 1052 × 1052 patches and resized to 640 × 640; SYNTHIA images are cropped into two 640 × 640 patches.Random mirroring and Gaussian blur are also applied before training.Source: B. Implementation Details · p. 8
  • Models use ImageNet-pretrained initialization and SGD with batch size 2 for 200,000 iterations.Momentum is 0.9, weight decay is 5e-4, and the initial learning rate is 1e-5 with poly power 0.9.Source: B. Implementation Details · p. 8

A. Comparison with State-of-Arts

The framework outperforms DRPC across the reported source–target settings and is competitive with domain-adaptation methods, while comparisons with IBN and SW also favor the proposed method on reported mIoU results.

Comparison with DRPC

Comparison with IBN

Qualitative comparison

  • Qualitative results show progressively improved segmentation as proposed modules are added under TCPS.The figure covers GTA5 to Cityscapes, Mapillary, and BDDS.Source: Figure 13 · p. 9

Comparison with IBN

Comparison with SW

Comparison with Domain Adaptation methods

B. Ablation Studies

The ablations report gains from GTR, LTR, CGL, and TCPS, while also identifying texture complexity, raw-style proportion, and residual texture sensitivity as relevant conditions.

Effects of CGL

Effects of GTR

  • GTR raises results from 34.0% to 39.2%, 28.1% to 35.4%, and 28.6% to 32.1% on G→C, G→B, and G→M, respectively.The authors attribute the improvements to using diverse texture styles.Source: Table VIII · p. 10, B. Ablation Studies · p. 10

Effects of LTR

  • 41.8%/36.1%/33.7% mIoU is reported for LTR on G→C/G→B/G→M, with gains of 7.8%/8%/5.1% over baseline.LTR outperforms GTR in all three settings, and the combined GTR-LTR model improves further.Source: B. Ablation Studies · p. 10, Table VIII · p. 10

LTR style combinations and proportions

  • LTR performance depends chiefly on the proportion of raw style: one painting style performs nearly as well as multiple styles at the same raw-style proportion.Adding painting styles alone does not improve results and can degrade them; the best reported configuration uses equal proportions of raw and painting styles.Source: Table IX · p. 10, B. Ablation Studies · p. 11, Table X · p. 11

Unreal paintings vs. real or synthetic images

  • Using paintings outperforms using real or synthetic images for GTR and LTR in all reported SRSS settings.The authors associate paintings’ irregular, indistinct textures with learning domain-invariant cues.Source: Table XII · p. 11, B. Ablation Studies · p. 11

Effects of TCPS

  • 0.5%–1.3% gains are reported for TCPS over random painting selection in each ablation setting.The comparison averages five experiments, each using an independent random selection of 15 paintings when TCPS is not used.Source: Table VIII · p. 10, B. Ablation Studies · p. 11

Parameter analysis

  • 200,000 training iterations perform best; the reported favorable ranges are λ=4–16 and Texc∈[0.55, 0.65], with β=1e−5 best.Neighboring λ results in the stated range are within 1%; λ is sampled from per iteration to increase painting diversity while maintaining performance.Source: B. Ablation Studies · pp. 11–12, Table XIII · p. 11, Table XIV · p. 12

Qualitative results and failures

  • The framework produces clearer boundaries and less noisy road regions, but may misclassify dark-green sidewalk regions as terrain.The authors state that GTR and LTR do not completely eliminate texture effects.Source: Figure 14 · p. 11, B. Ablation Studies · p. 12

VI. CONCLUSION

The paper proposes GTR and LTR for domain-generalization SRSS, with CGL to harmonize them and TCPS to select paintings. Experiments report state-of-the-art results across multiple synthetic-to-real settings and backbones.

Conclusion

  • GTR randomizes texture to address CNN texture bias, while LTR targets cases where texture differences occur only in local regions.The paper presents both mechanisms for domain generalization in synthetic-to-real semantic segmentation.Source: §VI · p. 12
  • CGL is introduced to harmonize GTR and LTR, and TCPS selects paintings considered reliable for those mechanisms.Source: §VI · p. 12
  • Experiments report state-of-the-art results across multiple synthetic-to-real settings and different network backbones.Source: §VI · p. 12

Related papers

Loading 2108.02376v2…