Source-linked AI summary
FDA: Fourier Domain Adaptation for Semantic Segmentation
Yanchao Yang, Stefano Soatto
TL;DR
Unsupervised domain adaptation must transfer models trained on annotated source data to differently distributed target data without target annotations. FDA aligns low-level image statistics without additional alignment training, and in semantic segmentation it surpasses a more involved state-of-the-art adversarial model.
Problem
Unsupervised domain adaptation transfers models from annotated source data to unannotated target data, but covariate shift can substantially reduce target performance.
Method
FDA aligns source and target low-level statistics using a Fourier-based preprocessing operation, integrated into semantic segmentation without additional alignment training.
Results
FDA surpasses the state-of-the-art adversarial model in semantic segmentation despite its simplicity.
Takeaways & Limitations
Low-level distributional misalignment can sometimes be captured with a fast Fourier transform instead of sophisticated architectures or laborious data augmentation.
Takeaways & Limitations
The authors do not claim FDA is a final solution and identify applications to other domain adaptation tasks as future work.
Abstract
from arXiv · showhide
We describe a simple method for unsupervised domain adaptation, whereby the discrepancy between the source and target distributions is reduced by swapping the low-frequency spectrum of one with the other. We illustrate the method in semantic segmentation, where densely annotated images are aplenty in one domain (synthetic data), but difficult to obtain in another (real images). Current state-of-the-art methods are complex, some requiring adversarial optimization to render the backbone of a neural network invariant to the discrete domain selection variable. Our method does not require any training to perform the domain alignment, just a simple Fourier Transform and its inverse. Despite its simplicity, it achieves state-of-the-art performance in the current benchmarks, when integrated into a relatively standard semantic segmentation model. Our results indicate that even simple procedures can discount nuisance variability in the data that more sophisticated methods struggle to learn away.
1. Introduction
Unsupervised domain adaptation addresses performance loss when models trained on annotated source data encounter unlabeled target data with different low-level statistics. The paper proposes aligning these statistics through a simple Fourier-based transformation rather than additional adversarial training.
- Motivation: Unsupervised domain adaptation transfers models from annotated source data to unlabeled target data, such as synthetic images to real images.Models trained only on source data can perform poorly on the target because of covariate shift.
- Contribution: Unlike many contemporary domain-adaptation methods, the proposed alignment requires no training beyond the primary semantic-segmentation task and avoids adversarial optimization.The related approaches use auxiliary objectives or discriminators to encourage domain invariance.
- Approach: Fourier Domain Adaptation swaps low-frequency spectral components from target images into source images while preserving source annotations and semantic content.The transformed images are reconstructed with the inverse Fourier transform for segmentation training.
- Approach: The method requires selecting the size of the spectral neighborhood to swap and also evaluates averaging results across multiple domain sizes.This parameter is represented by the green square in Figure 1.
- Motivation: Low-level image statistics can vary without changing high-level semantics, yet neural networks may fail to transfer across such statistics.The paper identifies sensor, illumination, and other low-level variability as nuisance factors affecting generalization.
- Evaluation setting: The method is evaluated for semantic segmentation in a setting where dense annotations are difficult to obtain for real images but available for synthetic data.The paper places this challenge within broader domain-adaptation and image-translation research.
2. Method
The method aligns source and target domains by replacing low-frequency source amplitudes with target amplitudes, then trains segmentation models using adapted source images and target regularization. Multi-scale FDA and self-learning further regularize adaptation through multiple spectral bandwidths and pseudo-label predictions.
- Fourier alignment: FDA computes Fourier representations and swaps the source image’s low-frequency amplitude region with the target image’s corresponding region before inverse transformation.The source phase remains unchanged, preserving source content while adapting image appearance toward the target domain.
- Fourier alignment: β controls the swapped spectral neighborhood: larger values reduce the domain gap but introduce visible artifacts, so the method restricts β and can average multiple values.The paper states β ≤0.15 for standard use and describes multi-scale averaging across choices of β.
- Segmentation training: The adapted source dataset trains a semantic segmentation network with cross-entropy loss while retaining the original source annotations.Adapted images are generated online because the full Cartesian product of source and target datasets is large.
- Segmentation training: FDA reframes UDA as semi-supervised learning and regularizes target predictions by minimizing entropy with a robust Charbonnier weighting function.For η > 0.5, the penalty assigns greater weight to high-entropy predictions than to low-entropy predictions.
- Self-supervised training: Multi-band Transfer trains multiple models with different β values, averages their softmax predictions, and uses the resulting pseudo-labels for self-supervised training.The full procedure uses an initial round of training from scratch followed by two rounds of self-supervised training.
3. Experiments
Experiments evaluate FDA on GTA5→CityScapes and SYNTHIA→CityScapes using two segmentation backbones, ablations, and benchmark comparisons. Multi-band transfer with self-supervised training achieves the strongest reported results, including improvements over prior methods.
- Evaluation setup: Experiments use GTA5 and SYNTHIA as labeled synthetic source domains and CityScapes as an unlabeled real target domain.The two adaptation scenarios are GTA5→CityScapes and SYNTHIA→CityScapes.
- Evaluation setup: FDA is evaluated with DeepLabV2-ResNet101 and FCN-8s-VGG16, with the segmentation network as the only trained network.The experiments use standard segmentation architectures and apply FDA before mean subtraction on training images.
- Ablation study: Across β values, single-scale FDA networks maintain similar mIOU on GTA5→CityScapes, indicating robustness to the tested spectral-neighborhood sizes.The single-scale ablation trains separate models with β values 0.01, 0.05, and 0.09.
- Ablation study: 4.54%: FDA with β=0.09 and no entropy loss surpasses the Cycada baseline on GTA5→CityScapes.This comparison uses the single-scale FDA model trained with λent = 0.
- Multi-band Transfer (MBT): 3.9%: averaging predictions across β values with MBT improves relative performance over the best first-round single-scale performer.The improvement is also observed after subsequent self-supervised training rounds.
- Benchmarks: FDA-MBT achieves top performance on GTA5→CityScapes, including a 4.0% improvement over BDL, and leads across the tested backbones.Single-scale FDA also outperforms most adversarial-training methods, while FDA-ENT achieves similar performance to methods using spatial priors or structured-output adversarial training.
- Benchmarks: On SYNTHIA→CityScapes, the method achieves top performance with both backbones, outperforming BDL by 2.1% with VGG16 and 3.9% with ResNet101.Evaluation uses 16 classes for VGG16 and 13 classes for ResNet101.
- Qualitative results: FDA-MBT predictions are visually smoother, preserve fine structures such as poles, and perform well on rare classes including trucks and bicycles.The qualitative comparison is against BDL using the same segmentation-network backbone.
4. Discussion
The paper proposes learning-free domain alignment that integrates into a learning system, while showing that simple Fourier-based processing can surpass more involved state-of-the-art methods. It also frames the method as a limited response to nuisance variability rather than a final solution to domain robustness.
- The proposed domain-alignment method requires no learning and can transform unsupervised domain adaptation into semi-supervised learning within a learning system.
- The method improves over the baseline and surpasses current state of the art despite being considerably simpler.
- Fourier-based adaptation keeps transformed images in the real image space because the inverse Fourier transform cancels the imaginary component for real signals.
- The authors position the method as a possible preprocessing alternative to sophisticated architectures or laborious data augmentation for some nuisance-variability cases.
- The authors do not claim the method is a final solution to robustness against nuisance variability and identify other domain-adaptation tasks as future applications.