PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain-invariant Feature Exploration for Domain Generalization

Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, Xing Xie

arXiv:2207.12020v2cs.LGcs.CV

TL;DR

Because training and test distributions may be non-IID, domain generalization seeks models that work on unseen target domains. DIFEX combines Fourier-phase distillation for internal features with cross-domain correlation alignment and feature exploration, and experiments report it outperforming recent baselines across datasets.

  • Problem

    How to generalize from non-IID training data to an unseen target distribution remains a domain-generalization challenge, since the IID assumption may not hold in practice.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    DIFEX distills high-level semantics from Fourier phase for internal features, aligns feature distributions across domains for mutual features, and regularizes divergence to encourage exploration.

    Source: §1 · p. 2, §3.3.1 · p. 5

  • Results

    DIFEX outperforms other recent baselines in all datasets across experiments on image data and time-series datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents DIFEX as a domain-generalization approach for learning both internally- and mutually-invariant features across image and time-series classification.

    Source: §5 · p. 12

    The authors identify combining more internally- and mutually-invariant features and improving optimization stability as future work, including normalization or other distances.

    Source: §5 · p. 12

Abstract

from arXiv · show

Deep learning has achieved great success in the past few years. However, the performance of deep learning is likely to impede in face of non-IID situations. Domain generalization (DG) enables a model to generalize to an unseen test distribution, i.e., to learn domain-invariant representations. In this paper, we argue that domain-invariant features should be originating from both internal and mutual sides. Internal invariance means that the features can be learned with a single domain and the features capture intrinsic semantics of data, i.e., the property within a domain, which is agnostic to other domains. Mutual invariance means that the features can be learned with multiple domains (cross-domain) and the features contain common information, i.e., the transferable features w.r.t. other domains. We then propose DIFEX for Domain-Invariant Feature EXploration. DIFEX employs a knowledge distillation framework to capture the high-level Fourier phase as the internally-invariant features and learn cross-domain correlation alignment as the mutually-invariant features. We further design an exploration loss to increase the feature diversity for better generalization. Extensive experiments on both time-series and visual benchmarks demonstrate that the proposed DIFEX achieves state-of-the-art performance.

1 Introduction

DIFEX treats domain-invariant features as both intrinsic within-domain semantics and transferable cross-domain information. It combines these feature types and reports outperforming recent baselines across its evaluated datasets.

1 Introduction

  • DG targets generalization to an unseen domain, unlike domain adaptation settings where target data is available for training.The motivation is that training and test data may not be identically distributed, as when activity data from children and adults differ.Source: §1 · pp. 1–2
  • The paper distinguishes internal invariance, which captures intrinsic data semantics, from mutual invariance, which captures transferable information across domains.It argues that combining these sources can support generalization to unseen domains.Source: §1 · p. 2
  • DIFEX outperforms recent baselines across all datasets in experiments spanning visual and time-series benchmarks.The paper presents these experiments as demonstrating the method’s superiority and universality.Source: §1 · p. 2
  • DIFEX distills high-level Fourier-phase semantics for internal features and aligns cross-domain correlations for mutual features.It also adds a regularization to maximize feature divergence and encourage exploration.Source: §1 · p. 2
  • The authors identify simultaneous exploration of inter- and intradomain features as a promising direction for future DG research.They report comprehensive experiments across visual and time-series settings as evidence of DIFEX’s superiority and universality.Source: §1 · p. 2

2 Related Work

The related work places DIFEX among domain-generalization methods, distinguishes its two invariant feature types from common disentanglement approaches, and connects Fourier phase to semantic content.

2.1 Domain Generalization

  • DG work includes data augmentation, domain-invariant feature learning, and meta-learning, while the paper focuses on domain-invariant methods applicable across tasks.The authors describe DIFEX as the first DG work, to their knowledge, to propose internally- and mutually-invariant representations and use inheritance and exploration simultaneously.Source: §2.1 · pp. 2–3

2.2 Domain-invariant Representation Learning

  • Domain-invariant representations are learned through alignment, adversarial learning, or disentanglement, but DIFEX learns two invariant feature types rather than invariant and domain-specific features.The paper shares with recent work the concern that simple alignment may be insufficient, and seeks greater feature diversity.Source: §2.2 · p. 3

2.3 Fourier Features

  • Prior work links Fourier phase to high-level semantics, while Fourier-amplitude methods address style or diversity and are largely designed for visual tasks.The authors suggest phase may provide universal domain-invariant features because phase retains much high-level semantic information in original signals.Source: §2.3 · p. 3

3 Methodology

DIFEX targets domain generalization by combining within-domain Fourier-phase features with cross-domain correlation features. It distills Fourier-phase information, aligns domain correlations, and encourages diversity between the two feature types.

3.1 Problem Formulation

  • DG learns a predictor from multiple labeled source domains with differing joint distributions to minimize error on an unseen target with an unknown distribution.The source and target domains share input and label spaces.Source: §3.1 · p. 3

3.2 Motivation

  • Fourier phase retains high-level semantics and is less affected by domain shifts, but phase features alone miss commonness across domains.In the walking examples, phases were similar across two people while amplitudes differed; phase-based restoration also lost some raw-data periodicity.Source: §3.2 · p. 4

3.3 DIFEX

  • DIFEX concatenates internally- and mutually-invariant features from multiple training domains for classification and maximizes their divergence with an exploration loss.The network uses a common feature extractor before producing the two feature types.Source: §3.3 · p. 4

3.3.1 A distillation framework to learn internally-invariant features

  • A distillation framework uses Fourier-phase information to train a teacher for classification, then guides a student using raw data to learn those features.The teacher is trained with Fourier-phase inputs and class labels; feature distillation brings student features closer to teacher features.Source: §3.3.1 · p. 5

3.3.2 Explore mutually-invariant features

  • For mutually-invariant features, DIFEX aligns second-order statistics between domains using covariance matrices and encourages divergence from internal features.The exploration loss uses negative L2 distance between the two feature types.Source: §3.3.2 · p. 6

3.3.3 Summary of DIFEX

  • DIFEX optimizes the teacher first, then a four-objective model for classification, internal features, mutual features, and exploration; inference uses a forward pass.The implementation tunes hyperparameters to balance the objectives, with heuristic automatic selection left for future work.Source: §3.3.3 · p. 6

3.4 Discussions

  • Distillation avoids an extra feature extractor and Fourier input at inference, while virtual domain labels offer a training option when only one domain is available.The authors report that random domain labels for alignment training on one domain seem effective in experiments.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate DIFEX across visual and time-series domain-generalization benchmarks, including image classification, activity and gesture recognition, single-domain tests, and component analyses.

4.1 Image classification

  • DIFEX exceeds the second-best average accuracy by 1.08%, 0.92%, and 3% across three image benchmarks.The paper reports these as average improvements across Digits-DG, PACS, and VLCS.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 7

4.2 Sensor-based human activity recognition

  • DIFEX achieves the best average performance across all three settings, beating the second-best method by about 2.8%, 5.5%, and 1.5% on DSADS, USC-HAD, and PAMAP in Cross-Person, about 2.3% in Cross-Position, and about 7.6% in Cross-Dataset.The authors report larger improvements for more difficult benchmarks with fewer raw channels, including USC-HAD in Cross-Person, which has 6 channels.Source: §4.2 · pp. 9–10

4.3 EMG gesture recognition

  • On EMG Cross-Person recognition, DIFEX has the best average performance and is 7.25% better than the second-best method.The authors report that simple domain alignments can sometimes hurt, while DIFEX performs best on all tasks.Source: §4.3 · p. 10, Table 7 · p. 10

4.4 Single domain generalization

  • For single-domain generalization, DIFEX improves USC-HAD performance by over 9% against four methods and PACS performance by over 9% on average against five methods.The USC-HAD test uses two randomly chosen subjects as domains; PACS trials use one selected dataset as source and another as target.Source: §4.4 · pp. 10–11, Figure 4 · p. 10, Table 8 · p. 11

4.5 Analysis

  • On PACS, combining internally- and mutually-invariant features performs much better than either feature type alone; all three Eq. 7 components contribute to the best result.Mutual features alone slightly improve over ERM, while internal features alone match ERM; removing any Eq. 7 component still improves over ERM.Source: §4.5 · p. 11, Figure 6 · p. 11, Table 9 · p. 12
  • In image examples, Fourier phase retains most classification information, whereas amplitude may be unhelpful; phase-only reconstructions remain less clear than the originals.The examples vaguely reveal dogs in phase-only reconstructions, while the original images show them more clearly.Source: Figure 5 · p. 11, §4.5 · p. 12
  • DIFEX is somewhat insensitive to hyperparameter choices, but the exploration-loss weight λ3 is relatively sensitive and may need attention in applications.The three parameters correspond to Fourier-phase teacher distillation, CORAL alignment, and exploration loss.Source: §4.5 · p. 12, Figure 7 · p. 12

5 Conclusion

DIFEX combines internally invariant features distilled from a teacher using Fourier phase information with mutually invariant features from domain alignment. Experiments on image and time-series classification support its reported superiority, while the authors identify further feature combinations and more stable optimization as future directions.

5 Conclusion

  • DIFEX combines internally invariant features distilled from a Fourier-phase teacher with mutually invariant features obtained through domain alignment.The teacher first learns classification information; its filtered features and a classification loss help reduce the influence of noisy or unhelpful information.Source: §5 · p. 12
  • Experiments on image and time-series classification demonstrate DIFEX’s superiority.The conclusion reports broad evaluation but supplies no quantitative comparison in these passages.Source: §5 · p. 12
  • The authors plan to combine more internal and mutual features and consider normalization or other distances to stabilize optimization.They also note that performance drops when λ3, the exploration-loss weight, is set to zero.Source: §5 · p. 12

A.1 Novelty

DIFEX’s novelty claim centers on separating domain-invariant representations into internal and mutual aspects, then combining their exploration with Fourier-phase-guided feature learning. The authors distinguish this design from existing disentanglement and Fourier-based approaches.

A.1 Novelty

  • DIFEX claims novelty in splitting domain-invariant representations into internally and mutually invariant aspects, rather than in the general idea of disentanglement.The authors contrast their method with existing disentanglement work, which they say is mostly based on variational autoencoders.Source: §A.1 · p. 16
  • Unlike Fourier-based DG work the authors discuss, DIFEX uses Fourier phase to guide internally invariant feature learning through knowledge distillation.They distinguish this from Xu et al. (2021), which uses Fourier amplitude for data augmentation, and from Xingchen et al. (2021), which they say is not a traditional DG setting.Source: §A.1 · p. 16
  • DIFEX also combines internal and mutual feature learning with exploration to increase diversity between feature types.The authors state that few methods attend to exploration among different feature types for diversity.Source: §A.1 · p. 17

A.2 Compare to other Fourier-based methods, e.g. FACT

DIFEX is presented as outperforming FACT, especially on time-series benchmarks, while differing in how it uses Fourier phase information. Its feature extractor splits the last layer into internal and mutual representations, guided by distillation, alignment, and exploration.

A.2 Compare to other Fourier-based methods, e.g. FACT

  • FACT uses amplitude Mixup for augmentation and teacher-student consistency alignment, whereas DIFEX’s teacher extracts Fourier phase for feature learning.The authors say other Fourier-based DG methods they discuss do not directly use Fourier phase to guide internally invariant feature learning.Source: §A.2 · p. 17
  • DIFEX achieves better results than FACT, especially on time-series benchmarks.Figure 8 presents the comparison; the supplied passages do not report numerical values.Source: §A.2 · p. 17

A.3 How the mutually-invariant and the internally-invariant features are extracted.

  • DIFEX splits the feature network’s last layer into internal and mutual branches, sharing the feature extractor before that layer.Distillation, alignment, and exploration guide the two branches; their outputs are concatenated.Source: §A.3 · p. 17

B Implementation details

The authors seek fair comparisons by holding several training choices constant while tuning hyperparameters for each method. They also explain why they do not rely on DomainBed or original-paper results for every comparison.

B Implementation details

  • Comparisons use the same backbone, optimizer, data splits, and transformations across methods.The authors say the epoch count is large enough for each method to converge.Source: B Implementation details · p. 17
  • The authors tune hyperparameters for each method and select its best result for comparison.They also tune some baseline hyperparameters, such as learning rate, while reporting mainstream baseline settings.Source: B Implementation details · p. 17
  • The authors avoid relying on DomainBed or original-paper results because its random setup may not suit some methods and differing experimental choices can affect results.They state that different tricks, splits, and initial states can have large influences on final results.Source: B Implementation details · p. 17

C Limitations

The discussion motivates norm-L1 as an alternative because L2 exploration can be difficult to optimize, and reports gains from norm-L1, especially on difficult Cross-Dataset9.

C Limitations

  • L2 exploration may be difficult to optimize because L2 can be infinite.The authors therefore try a normalized distance, norm-L1.Source: C Limitations · p. 19
  • Norm-L1 brings improvements, especially on the difficult Cross-Dataset9 problem.Table 10 compares classification accuracy across exploration-distance computation methods.Source: C Limitations · p. 19, Table 10 · p. 19
  • The authors plan to try other distances, such as cosine, in future work.Source: C Limitations · p. 19

D Visualization study

The visualization study uses t-SNE to compare learned training and test features across methods. It reports that DIFEX further alleviates distribution shifts, while internal and mutual features can be weighted differently for performance in different situations.

D Visualization study

  • DIFEX further alleviates distribution shifts compared with traditional domain-alignment methods.Figure 9(d) illustrates the method's learned features; the paper links this to better performance.Source: D Visualization study · p. 19
  • ERM leaves test features with a different distribution from training features, and the target data are hard to classify.Figure 9(a) shows this t-SNE visualization.Source: D Visualization study · p. 19, Figure 9 · p. 20
  • Alignments alleviate the distribution issue, but some classes remain affected, especially with CORAL; the paper reports a similar pattern for mutual features in Figure 9(f).Source: D Visualization study · p. 19
  • The last fully connected layer can assign different weights to internally- and mutually-invariant features for performance in different situations.Source: D Visualization study · p. 19
  • The paper says its L2 version achieved the best results compared with other methods, while normalization techniques are mainly for performance and easier optimization.Source: D Visualization study · p. 19
  • Figure 9 encodes classes by color, domains by shape, test data as circles, and training data as plus signs.Source: Figure 9 · p. 20
Loading 2207.12020v2…