PaperlayerView exact revision on arXiv

Source-linked AI summary

A Fourier-based Framework for Domain Generalization

Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, Qi Tian

arXiv:2105.11120v1cs.CV

TL;DR

The paper addresses domain-shift degradation by proposing a Fourier-based domain-generalization framework that emphasizes phase information while augmenting amplitude information. FACT combines amplitude perturbation with co-teacher regularization, and experiments on three benchmarks report state-of-the-art domain-generalization performance.

  • Problem

    Deep neural networks generalize poorly when testing data come from distributions different from training data, motivating generalization to arbitrary unseen target domains from multiple source domains.

  • Method

    FACT uses Fourier-based amplitude augmentation to emphasize phase information and applies dual-form co-teacher regularization between predictions for original and augmented images.

  • Results

    Experiments on three benchmarks demonstrate state-of-the-art performance for domain generalization.

  • Takeaways & Limitations

    The results support focusing on spectral phase information as a way to capture domain-invariant semantic concepts for domain generalization.

  • Takeaways & Limitations

    The reported comparison with RSC is limited because the authors could not reproduce its original results and therefore compare under their own hardware environment.

Abstract

from arXiv · show

Modern deep neural networks suffer from performance degradation when evaluated on testing data under different distributions from training data. Domain generalization aims at tackling this problem by learning transferable knowledge from multiple source domains in order to generalize to unseen target domains. This paper introduces a novel Fourier-based perspective for domain generalization. The main assumption is that the Fourier phase information contains high-level semantics and is not easily affected by domain shifts. To force the model to capture phase information, we develop a novel Fourier-based data augmentation strategy called amplitude mix which linearly interpolates between the amplitude spectrums of two images. A dual-formed consistency loss called co-teacher regularization is further introduced between the predictions induced from original and augmented images. Extensive experiments on three benchmarks have demonstrated that the proposed method is able to achieve state-of-the-arts performance for domain generalization.

1. Introduction

Domain generalization addresses the poor out-of-distribution generalizability of deep networks by learning from multiple source domains for arbitrary unseen targets. FACT uses Fourier-based amplitude perturbation and co-teacher regularization to emphasize phase information as a potentially domain-robust semantic signal.

  • Motivation: Domain generalization trains on multiple source domains to generalize to arbitrary unseen target domains, unlike domain adaptation, which uses a specific target domain.Collecting data from every possible target domain and training every source-target pair is described as expensive and impractical.
  • Fourier perspective: Fourier phase information is described as preserving high-level semantics, whereas amplitude information contains low-level statistics.Phase-only reconstruction reveals recognizable visual structures, while amplitude-only reconstruction makes the exact object difficult to identify.
  • Proposed framework: FACT introduces Fourier-based data augmentation that linearly interpolates image amplitudes while keeping phase information unchanged.The augmentation distorts amplitude information without access to target-domain data, addressing the limitation of amplitude transfer for domain generalization.
  • Proposed framework: FACT combines an implicit Fourier-augmentation constraint with an explicit co-teacher regularization constraint.The framework is presented as a Fourier Augmented Co-Teacher approach for learning more from phase information.
  • Evaluation: FACT is evaluated on Digits-DG, PACS, and OfficeHome, where experiments report performance exceeding several state-of-the-art domain-generalization methods.The paper also reports ablation studies and analyses of the rationale behind the framework.

2. Related Work

Prior work on domain generalization includes distribution alignment, adversarial learning, meta learning, and other techniques, while Fourier methods connect spectral components to semantics and domain statistics. FACT builds on these directions by combining Fourier augmentation with dual consistency regularization.

  • Domain generalization: Domain generalization seeks knowledge from multiple source domains that generalizes to arbitrary unseen target domains.Earlier approaches include kernel methods, domain-adversarial learning, and meta learning.
  • Fourier methods: Fourier-spectrum studies associate phase with high-level semantics and amplitude with low-level statistics in signals.Prior Fourier domain-adaptation work replaces part of a source image’s amplitude spectrum with a target image’s spectrum to generate target-like training images.
  • Consistency regularization: Consistency regularization aligns outputs from differently perturbed models, with momentum-updated teachers providing targets for consistency alignment.Related methods also use adversarial perturbations or MixUp samples for consistency.
  • FACT: FACT designs a dual-formed consistency loss that aligns predictions across original and Fourier-augmented images using a momentum-updated teacher.The framework’s two key components are Fourier-based data augmentation and co-teacher regularization.

3. Method

FACT is a Fourier-based domain-generalization framework that perturbs amplitude information while preserving phase, then enforces prediction consistency between original and augmented views. Its objective combines classification with dual-form co-teacher regularization using a momentum-updated teacher.

  • Fourier-based perspective: FACT models domain generalization with Fourier representations, using phase information as the presumed carrier of high-level semantics and amplitude as low-level statistics.The framework is motivated by emphasizing phase while perturbing amplitude across source images.
  • Fourier-based data augmentation: Amplitude mix linearly interpolates the amplitude spectrums of two images from arbitrary source domains and recombines the mixed amplitude with the original phase.The resulting Fourier representation is transformed back into an augmented image, with λ sampled from U(0, η).
  • Fourier-based data augmentation: The augmented images are trained with their original labels using standard cross entropy, while the original images also contribute a classification loss.The framework therefore retains supervised learning on both augmented and original inputs.
  • Fourier-based data augmentation: Amplitude mix differs from target-domain spectral transfer because it swaps or mixes amplitude spectrums between source images without requiring target data.This makes the operation compatible with domain generalization, where unseen target data are unavailable during training.
  • Co-teacher regularization: Co-teacher regularization aligns student predictions across augmented and original images in both directions, using softened outputs from a momentum-updated teacher.The dual alignment treats the original image and augmented counterpart as two views and encourages shared learning focused on invariant phase information.
  • Co-teacher regularization: The full objective combines the classification losses and co-teacher regularization, with β controlling their trade-off.The regularizer addresses possible differences in categorical relations learned from original and augmented inputs.

4. Experiment

FACT is evaluated on three domain-generalization benchmarks using leave-one-domain-out testing, where it generally improves over established baselines and competing methods. Ablations attribute its performance to amplitude-mix augmentation and dual-form co-teacher regularization, while the PACS gains are strongest on domains with larger discrepancies.

  • Datasets and settings: FACT is evaluated on Digits-DG, PACS, and OfficeHome using leave-one-domain-out classification accuracy averaged over three runs.Models train on source-domain training splits, use source validation splits for selection, and test on the held-out domain.
  • Evaluation on Digits-DG: FACT exceeds the second-best Digits-DG method by more than 3% on average, with margins of 4% on SVHN and 7% on SYN over L2A-OT.SVHN and SYN are described as the hardest target domains because of cluttered digits and low image quality.
  • Evaluation on PACS: On PACS, FACT improves over DeepAll by 7.52% on art-painting, 3.52% on cartoon, and 11.41% on sketch, while slightly decreasing on photo.The paper attributes the photo-domain decrease to ignoring potentially useful low-level cues, yet reports better overall performance.
  • Evaluation on PACS: FACT outperforms adversarial-augmentation and meta-learning methods, including MASF, DDAIG, and L2A-OT, without additional adversarial or episodic training steps.It also exceeds RSC by 2.11% on average under the authors’ comparison setup.
  • Evaluation on OfficeHome: On OfficeHome, FACT consistently improves over DeepAll on all held-out domains and surpasses DDAIG and L2A-OT in average performance.DeepAll is a strong baseline on OfficeHome because of smaller domain discrepancy and similarity to ImageNet.
  • Evaluation on PACS: The RSC comparison is qualified because the authors could not reproduce its reported results and therefore reran its code under their own environment.The paper conjectures that hardware differences may explain the discrepancy.
  • Ablation studies: Removing amplitude mix causes a large performance drop, while adding vanilla consistency helps slightly and the momentum-teacher variant performs best among the ablations.Using only one direction of co-teacher alignment also fails to outperform full FACT, supporting both original-to-augmented and augmented-to-original consistency.
  • Ablation studies: Amplitude mix outperforms the AS-partial alternative, which swaps only a small centralized amplitude-spectrum area with ratio r = 0.09.The comparison directly evaluates alternative Fourier-based augmentation choices.

5. Discussion

The discussion argues that Fourier phase information carries useful semantics for domain generalization, while amplitude perturbation and co-teacher regularization encourage models to rely more on phase-derived features. Single-domain evaluations and theoretical analysis support this view, while also indicating that amplitude information can remain useful in some settings.

  • Phase information: Phase-only training matched or exceeded original-image baselines in 11 of 16 cases, supporting phase information as useful semantics for unseen-domain generalization.All models were trained from scratch without ImageNet pretraining.
  • Phase information: Amplitude-only training caused large performance degradation in almost all cases, indicating that amplitude information alone provides few meaningful semantics.
  • Phase information: Phase-only models sometimes underperformed when transferring from cartoon or art domains to photos, suggesting that photo-domain performance may also require amplitude information.The authors therefore retain amplitude information while perturbing it rather than removing it completely.
  • Amplitude perturbation: Amplitude perturbation increases attention to phase-derived features because high-variance amplitude features receive weights approaching zero under the analyzed objective.The analysis considers a linear softmax classifier with a feature extractor and a second-order expansion of augmented risk.
  • Why phase matters: Phase records event locations and spatial relationships, allowing models to aggregate contours and edges into object-related cues.

6. Conclusions

The paper presents Fourier phase information as a basis for domain generalization and combines amplitude-based augmentation with co-teacher regularization. Experiments and reconstruction examples support the framework’s state-of-the-art domain-generalization performance and the semantic role of phase information.

  • Conclusions: FACT introduces a Fourier-based domain-generalization perspective centered on learning domain-invariant semantic concepts from spectral phase information.
  • Conclusions: The framework combines an implicit Fourier-based augmentation constraint with an explicit co-teacher regularization constraint.
  • Conclusions: Extensive experiments on three benchmarks demonstrate state-of-the-art performance for domain generalization.
  • Conclusions: Phase-only reconstructions preserve general visual structures, whereas amplitude-only reconstructions mainly contain low-level statistics without clear semantic meanings.

B.1. Experiments on three DG benchmarks

The benchmark experiments evaluate FACT across Digits-DG, PACS, and OfficeHome using dataset-specific backbones, optimization settings, and augmentation strengths. The supplied passages also describe single-domain phase-only and amplitude-only evaluation procedures.

  • Experimental setup: FACT is evaluated on Digits-DG, PACS, and OfficeHome using the corresponding benchmark protocols and backbone implementations.Digits-DG follows the backbone used by [52], while PACS and OfficeHome use ImageNet-pretrained ResNet18 and ResNet50.
  • Experimental setup: Training uses Nesterov-momentum SGD with momentum 0.9 and weight decay 5e-4, with 50 epochs for Digits-DG and PACS and 30 for OfficeHome.
  • Experimental setup: Standard spatial and color augmentations are used, with 32 × 32 inputs for Digits-DG and 224 × 224 inputs for PACS and OfficeHome.
  • Experimental setup: The consistency-loss weight is 2 for Digits-DG and PACS and 200 for OfficeHome, while AM strength is 1.0 for Digits-DG and PACS and 0.2 for OfficeHome.
  • Single-domain evaluation: Single-domain evaluations train ResNet18 separately on original, phase-only, and amplitude-only reconstructed images before evaluating across all domains.

C. Complete results on DG benchmarks

Complete benchmark results are reported as mean±std on Digits-DG, PACS, and OfficeHome. For OfficeHome, the paper additionally reports results under a best-target-domain accuracy protocol for fair comparison with prior work.

  • Complete benchmark results: Complete results are reported in mean±std format for Digits-DG, PACS, and OfficeHome.
  • Complete benchmark results: Because Dou et al. report best accuracy on the target domain, the paper also reports OfficeHome results under that protocol for fair comparison.

D. Additional results of AlexNet

FACT with AlexNet remains competitive on PACS, exceeding EISNet and MetaVIB by around 1% on average. The largest gains occur on art and sketch, which have large distribution shifts from ImageNet.

  • FACT with AlexNet exceeds EISNet and MetaVIB by around 1% in average performance on PACS.The experiments use an ImageNet-pretrained AlexNet backbone with standard augmentation and SGD training.
  • The largest gains occur on the art and sketch domains, which have large distribution shifts from ImageNet.The authors present this as evidence of effectiveness on unknown out-of-domain data.

E. Variants of Fourier data augmentation

The paper evaluates several Fourier-based augmentation variants beyond amplitude swap and amplitude mix. Amplitude mix performs best on average, while eliminating amplitude information performs relatively poorly.

  • Fourier augmentation variants: The paper introduces Amplitude CutMix, which mixes amplitude components of two images using a pixel-level CutMix strategy.Amplitude CutMix samples a binary mask and linearly combines the amplitude components.
  • Fourier augmentation variants: Amplitude Jittering perturbs an image’s amplitude information with Gaussian noise controlled by σ.The noise is sampled from N(0, σ).
  • Fourier augmentation variants: Amplitude Elimination uses phase-only reconstructions as augmented images, completely removing the original amplitude information.This variant directly emphasizes phase information rather than perturbing amplitude components.
  • Results: Amplitude mix performs best in average performance among the evaluated augmentation types.The comparison is reported for both DeepAll and FACT on PACS.
  • Results: Amplitude Elimination performs relatively worse, potentially because phase-only images differ substantially from the original and ImageNet-pretrained distributions.The authors also note that photo domains may require amplitude information because they contain rich low-level details.
  • Limitations: The evaluated augmentation types are only some possible Fourier-based instantiations, leaving other variants and compositions for future study.

F.3. Sensitivity to the perturbation strength η

The perturbation strength η affects datasets differently: larger values work best on PACS, while smaller values work best on OfficeHome. The authors relate this difference to domain discrepancy.

  • Sensitivity to η: On PACS, increasing η improves performance, with the best result at η = 1.0.Here η controls the strength of amplitude perturbation; η = 0 applies no Fourier-based augmentation.
  • Sensitivity to η: On OfficeHome, smaller η values perform better, with the best result at η = 0.2.The authors attribute the contrasting behavior to different levels of domain discrepancy.
  • Interpretation: A larger η is more suitable for PACS, whereas a smaller η is more suitable for OfficeHome.The authors describe larger η as more aggressive and smaller η as more conservative augmentation.

F.4. Sensitivity to the consistency loss weight β

FACT’s consistency-loss weight β has dataset-dependent effects and interacts with the perturbation strength η. The reported settings favor smaller β with larger η on high-discrepancy datasets and larger β with smaller η on low-discrepancy datasets.

  • Sensitivity to β: On PACS, average performance is stable for β from 1.0 to 20, with the best performance at β = 2.0.Values outside this range can impose either too weak or too strong a consistency constraint.
  • Sensitivity to β: On OfficeHome, performance is stable for β from 20 to 500, with the best performance at β = 200.The authors report better performance with larger β on OfficeHome.
  • Interaction between η and β: A larger η paired with a smaller β works better for PACS, while a smaller η paired with a larger β works better for OfficeHome.The authors interpret η and β as compensating for each other in regularization strength.
  • Recommended settings: For large-discrepancy datasets, the suggested setting is η = 1.0 with β = 2.0; for small-discrepancy datasets, it is η = 0.2 with β = 200.PACS and Digits-DG are given as examples of large discrepancy, while OfficeHome represents small discrepancy.
Loading 2105.11120v1…