PaperlayerView exact revision on arXiv

Source-linked AI summary

FSDR: Frequency Space Domain Randomization for Domain Generalization

Jiaxing Huang, Dayan Guan, Aoran Xiao, Shijian Lu

arXiv:2103.02370v1cs.CV

TL;DR

Domain generalization needs models that transfer from known source domains to unknown targets without relying on target data, while existing spatial randomization can alter invariant features. FSDR decomposes images into frequency components, randomizes domain-variant components, and preserves domain-invariant ones; experiments report superior segmentation and consistent gains when incorporated into related networks.

  • Problem

    Existing domain-randomization methods alter the full spatial image spectrum and can undesirably modify domain-invariant features, while domain adaptation requires target-domain data during training.

  • Method

    FSDR transforms images into frequency space, identifies domain-variant and domain-invariant frequency components, randomizes the former, and preserves the latter using spectrum analysis and iterative spectrum learning.

  • Results

    FSDR achieves superior semantic segmentation across multiple domain-generalization tasks and consistently improves existing domain-adaptation and domain-generalization networks when incorporated.

  • Takeaways & Limitations

    Explicit frequency-component isolation provides more controllable randomization with minimal effects on image semantics and domain-invariant features.

Abstract

from arXiv · show

Domain generalization aims to learn a generalizable model from a known source domain for various unknown target domains. It has been studied widely by domain randomization that transfers source images to different styles in spatial space for learning domain-agnostic features. However, most existing randomization uses GANs that often lack of controls and even alter semantic structures of images undesirably. Inspired by the idea of JPEG that converts spatial images into multiple frequency components (FCs), we propose Frequency Space Domain Randomization (FSDR) that randomizes images in frequency space by keeping domain-invariant FCs (DIFs) and randomizing domain-variant FCs (DVFs) only. FSDR has two unique features: 1) it decomposes images into DIFs and DVFs which allows explicit access and manipulation of them and more controllable randomization; 2) it has minimal effects on semantic structures of images and domain-invariant features. We examined domain variance and invariance property of FCs statistically and designed a network that can identify and fuse DIFs and DVFs dynamically through iterative learning. Extensive experiments over multiple domain generalizable segmentation tasks show that FSDR achieves superior segmentation and its performance is even on par with domain adaptation methods that access target data in training.

1. Introduction

Semantic segmentation suffers when models trained on synthetic or source-domain images encounter real-world domain shifts. FSDR addresses this by randomizing domain-variant frequency components while preserving domain-invariant components, improving domain generalization.

  • Synthetic-image segmentation models can experience drastic performance drops on real-world images because of domain bias and shift.
  • Unsupervised domain adaptation uses unlabelled target data, whereas domain generalization learns domain-invariant features without target-domain data.
  • Existing spatial-space domain randomization modifies whole images and can undesirably alter domain-invariant features.
  • FSDR transforms images into frequency space, randomizes domain-variant frequency components, and keeps domain-invariant components unchanged.
  • FSDR includes spectrum-analysis and spectrum-learning approaches for identifying domain-variant and domain-invariant frequency components.
  • Experiments across multiple domain-generalization tasks report consistently superior semantic segmentation, while combining FSDR with spatial-space generalization further improves generalization.

2. Related Works

Related work addresses domain generalization through multi-source or single-source strategies, with domain randomization especially common when only one source domain is available. FSDR differs from prior randomization by modifying frequency components selectively rather than altering the full spatial image spectrum.

  • Domain generalization includes multi-source and single-source settings that learn domain-agnostic features through invariant representations, domain-specific modules, or randomized data and domains.
  • Single-source domain generalization is more challenging because training uses only one source domain.
  • Domain randomization methods range from manually controlled image synthesis to gradient-based, adversarial, GAN-based, and other learned augmentations.
  • Manual randomization has limited scalability and constrained variation, while learning-based methods offer better scalability but may modify domain-invariant features.
  • FSDR explicitly separates domain-invariant and domain-variant frequency components, modifying only the latter to minimize effects on semantic structures.
  • Domain adaptation methods use unlabelled target data and include adversarial-training and image-translation approaches.

3. Method

FSDR analyzes frequency components to preserve domain-invariant information while randomizing domain-variant information for domain generalization. It supports static and iterative spectrum learning approaches, including histogram matching and bidirectional updates.

  • Spectrum Analysis: Low- and high-frequency components largely capture domain-variant colors and styles, whereas middle-frequency components largely capture domain-invariant structures and shapes.The frequency bands shown are low-pass [0, 0.04), middle-pass [0.04, 0.16) and [0.16, 0.64), and high-pass [0.64, 1].
  • Spectrum Analysis: FSDR identifies frequency components by applying DCT, decomposing each image into 64 frequency components, and evaluating filtered-source models on target images.A binary mask records preserved and discarded components; improved or degraded target performance indicates variant or invariant components, respectively.
  • Spectrum Analysis: Removing low- and high-frequency components improves model generalization, while visualizations show these bands capture more domain variation than middle-frequency components.The analysis evaluates synthetic-to-real classification using source SYNTHIA images and target ImageNet images.
  • FSDR-SA: FSDR-SA randomizes DVF histograms to match reference images while leaving DIFs unchanged, adding little extra computation compared with GAN-based translation.The source and reference images are transformed into frequency space, and histogram matching is applied only to components selected by the domain-invariant mask.
  • FSDR-SL: FSDR-SL iteratively identifies DIFs and DVFs from FC activations and prediction entropy, then uses the learned mask to randomize the remaining components.It represents decomposed frequency components spatially, records the learned information in ISL, and reduces overall prediction entropy by preserving low-entropy components.
  • FSDR-SL: FSDR-SL alternates spectrum learning on real-image batches with FSDR using spectrum-learned reference images, forming a bidirectional learning framework.Randomized synthetic images are transformed back to spatial representation and used with segmentation loss to update the model.

4. Experiments

Experiments evaluate FSDR on synthetic-to-real domain generalization for semantic segmentation and object detection, including ablations, comparisons, sensitivity analysis, and integration with existing methods. FSDR consistently improves generalization by preserving domain-invariant frequency components while randomizing domain-variant components.

  • Evaluation setup: FSDR is evaluated on GTA5- and SYNTHIA-to-real domain generalization tasks involving Cityscapes, BDD, and Mapillary.GTA5 contains 24,966 synthetic images and SYNTHIA contains 9,400; the tasks share 19 and 16 classes, respectively, with the real target datasets.
  • Ablation studies: The ablation compares Baseline, FSDR-SA, FSDR-SL, and full FSDR using Lorig, LSA, and LSL.FSDR-SA uses spectrum analysis, FSDR-SL uses spectrum learning, and full FSDR combines both approaches.
  • Ablation studies: FSDR-SA and FSDR-SL substantially outperform Baseline, while full FSDR performs best consistently across the three target domains.The results attribute the advantage of FSDR-SL to adaptive iterative spectrum learning and describe the two approaches as complementary.
  • State-of-the-art comparisons: FSDR outperforms state-of-the-art domain generalization methods across GTA5 and SYNTHIA tasks with both ResNet101 and VGG16 backbones.The reported advantage is attributed to preserving DIFs and randomizing DVFs only, avoiding over-randomization that can degrade image structures and semantics.
  • State-of-the-art comparisons: FSDR is on par with state-of-the-art UDA while remaining more stable across Cityscapes, Mapillary, and BDD evaluations.The UDA methods access Cityscapes target data during training, whereas FSDR is evaluated as domain generalization without target-domain training data.
  • Integration and efficiency: Adding FSDR to existing domain adaptation and generalization networks consistently improves segmentation with little extra inference cost.The integration adds a few losses without changing network structures, so trained models incur little extra parameters and computation at inference.
  • Sensitivity analysis: FSDR is relatively insensitive to p between 1/6 and 5/6, but performance drops near p=0 or p=1.Here p controls DIF and DVF selection sensitivity; p=0 randomizes all frequency components, while p=1 applies no randomization.
  • Other tasks: FSDR also outperforms domain adaptation and generalization methods consistently in a SYNTHIA-to-real object detection test.The detection evaluation uses Cityscapes, Mapillary, and BDD as target datasets and reports metric mAP.

5. Conclusion

The conclusion presents FSDR as a controllable frequency-space randomization method that changes domain-variant components while preserving domain-invariant ones. Its spectrum-analysis and spectrum-learning variants support effective identification of these components, and experiments show strong segmentation performance and consistent gains when integrated with existing networks.

  • Conclusion: FSDR decomposes images into DIFs and DVFs, enabling explicit manipulation and more controlled randomization with minimal effects on semantics.The method keeps DIFs unchanged while randomizing DVFs.
  • Conclusion: FSDR-SA and FSDR-SL identify DIFs and DVFs through spectrum analysis and spectrum learning, respectively.The two approaches provide complementary mechanisms for frequency-space randomization.
  • Conclusion: FSDR achieves superior segmentation performance and can be incorporated into state-of-the-art adaptation and generalization networks with consistent improvement.The conclusion characterizes FSDR as broadly compatible with existing networks.
Loading 2103.02370v1…