PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Challenging Improves Cross-Domain Generalization

Zeyi Huang, Haohan Wang, Eric P. Xing, Dong Huang

arXiv:2007.02454v1cs.CVcs.LG

TL;DR

CNNs often perform poorly when tested on distributions different from training data, especially when domain information is unavailable. The paper introduces Representation Self-Challenging (RSC), which suppresses dominant high-gradient features so CNNs learn from remaining label-correlated representations. Theoretical analysis and domain-agnostic experiments support improved cross-domain generalization without extra model architecture.

  • Problem

    CNNs struggle to generalize from training distributions to unseen domains, motivating robust training without requiring domain information.

  • Method

    RSC iteratively mutes high-gradient feature representations and forces CNNs to predict using remaining information.

  • Results

    Theoretical and empirical analyses support RSC as an effective cross-domain training heuristic, including state-of-the-art improvement over baseline CNNs in standard domain-generalization settings.

  • Takeaways & Limitations

    RSC can be applied to almost any CNN architecture with no extra model architecture and almost no additional computing effort.

  • Takeaways & Limitations

    The theoretical analysis is restricted to a simplified setting that considers only the upper layer while treating the feature representation Z as fixed.

Abstract

from arXiv · show

Convolutional Neural Networks (CNN) conduct image classification by activating dominant features that correlated with labels. When the training and testing data are under similar distributions, their dominant features are similar, which usually facilitates decent performance on the testing data. The performance is nonetheless unmet when tested on samples from different distributions, leading to the challenges in cross-domain image classification. We introduce a simple training heuristic, Representation Self-Challenging (RSC), that significantly improves the generalization of CNN to the out-of-domain data. RSC iteratively challenges (discards) the dominant features activated on the training data, and forces the network to activate remaining features that correlates with labels. This process appears to activate feature representations applicable to out-of-domain data without prior knowledge of new domain and without learning extra network parameters. We present theoretical properties and conditions of RSC for improving cross-domain generalization. The experiments endorse the simple, effective and architecture-agnostic nature of our RSC method.

1 Introduction

Cross-domain image classification remains difficult because models often rely on dominant features that do not transfer to unseen distributions. RSC addresses this by suppressing highly predictive representations during training and forcing CNNs to use additional label-correlated features.

  • Motivation: CNN performance degrades on out-of-domain samples whose distributions differ from training data.Domain adaptation and domain generalization address this distributional mismatch, but the paper targets settings without domain information.
  • Representation Self-Challenging: RSC is illustrated as muting the highest-gradient feature representations so the network predicts labels through other features.After training, the model is expected to leverage more predictive features than conventionally trained models.
  • Motivation: Without domain information, the goal is to train models that generalize robustly to arbitrary unseen domains.The paper builds on a setup where source-domain partitions and corresponding domain information may be unavailable.
  • Representation Self-Challenging: RSC discards representations with higher gradients each epoch, forcing prediction through remaining information rather than dominant feature subsets.The heuristic targets highly predictive cues such as frequent colors, edges, or shapes.
  • Analysis and Evaluation: The paper analyzes RSC mathematically and evaluates it with domain-agnostic cross-domain experiments and ablations.The ablations examine whether empirical performance aligns with the method’s intuition and inform the choice of an extra hyperparameter.

2 Related Work

Prior domain-generalization methods learn invariant features, augment source data, or use specialized regularization and meta-learning strategies. RSC instead selectively mutes gradient-identified predictive features without domain partitions and is designed as a model-agnostic training algorithm.

  • Learning Domain-Invariant Features: Feature-invariance methods minimize discrepancies among source domains to obtain representations that generalize to unseen target distributions.Examples include MMD constraints, multi-domain reconstruction, and adversarial training.
  • Learning Domain-Invariant Features: Meta-learning methods alternate domain-specific components or learn regularization functions for domain-generalization networks.The cited approaches include episodic training and MetaReg’s weighted ℓ1 regularization for the classification layer.
  • Learning Domain-Invariant Features: Other methods learn robust representations from mixed training data without source-domain partitions, while RSC activates more features directly.RSC follows this path without knowledge of source-domain partitions.
  • Augmenting the Source Domain: Source-augmentation methods widen the training-data span through domain perturbations, adversarial fictitious targets, or related transformations.These methods aim to increase coverage of possible target-domain data.
  • Key Differences: RSC is model-agnostic and does not use source- or target-domain partition knowledge, unlike approaches relying on target-domain assumptions.The paper presents this as the general real-world setting it addresses.
  • Generic Model Regularization: RSC locates and mutes the most predictive feature-map parts using gradients rather than randomness, activation, or prediction-divergence maximization.This selective process is presented as important to improving convergence.

3 Method

RSC trains CNNs by identifying and muting highly predictive representations, then updating the model using the remaining features. Its analysis connects this procedure to smaller generalization bounds, while emphasizing assumptions and a trade-off controlled by the discard percentage p.

  • RSC Update Algorithm: RSC requires one extra scalar hyperparameter, p, representing the percentage of representations discarded during training.The algorithm also takes ordinary training settings such as learning rate and maximum epochs.
  • RSC Update Algorithm: The method uses the same standard loss as other neural networks but changes how the model generates the representations used for prediction.RSC applies the perturbation before computing the updated gradients and parameters.
  • RSC Update Algorithm: RSC calculates representation gradients, identifies the most predictive features, masks them, and updates the full model using the perturbed representation.The procedure consists of locating high-gradient features, muting them, and performing the usual model update.
  • Theoretical Evidence: The theoretical analysis studies only the task component h with fixed representations Z, rather than the full CNN feature-learning process.This simplified setting is used to argue for the method’s theoretical strength.
  • Theoretical Evidence: Under stated assumptions, smaller ξ(p) yields a tighter generalization bound, and ξ(p)=0 recovers the classical i.i.d. bound.The analysis also relates ξ(p) to the empirical loss difference, which decreases across iterations under the stated conditions.
  • Theoretical Evidence: Choosing p balances training and cross-domain generalization error: larger p can reduce the latter but may prevent learning predictive features.The paper therefore treats p as a trade-off rather than selecting the largest possible discard percentage.

4 Experiments

Experiments evaluate RSC across several cross-domain datasets, ablations, dropout comparisons, and competing domain-generalization methods. Results favor gradient-based feature dropping, with consistent gains on PACS and broader evaluations.

  • Cross-Domain Evaluation: RSC was evaluated on PACS, VLCS, Office-Home, and ImageNet-Sketch using cross-domain protocols.PACS trains on three domains and tests on the remaining domain; ImageNet-Sketch trains on ImageNet and tests on sketches.
  • Ablation Study: Top-Gradient feature dropping outperformed Top-Activation, and both outperformed random selection.Top-Gradient is the default strategy used in subsequent ablations.
  • Ablation Study: 33.3% feature dropping produced the highest average PACS accuracy, although the optimal percentage is data-specific.The results support choosing neither an excessively large nor excessively small dropping percentage.
  • Ablation Study: Spatial+Channel RSC improved over Spatial-wise RSC, indicating further activation of features beneficial to target domains.The comparison used Top-Gradient, 33.3% feature dropping, and 33.3% batch percentage.
  • Ablation Study: RSC outperformed the compared dropout methods, including Cutout, DropBlock, and AdversarialDropout.The authors attribute the distinction to gradient-based selection of predictive features rather than random dropping or divergence maximization.
  • Cross-Domain Evaluation: Across comparisons with Hex, PAR, JiGen, and MetaReg, RSC consistently outperformed competing methods.The comparisons covered commonly used AlexNet, ResNet18, and ResNet50 configurations.
  • Cross-Domain Evaluation: On PACS, RSC improved over the latest baselines by 4.5 points with AlexNet, 5.2 with ResNet18, and 4.5 with ResNet50.Gains were especially pronounced on the sketch domain and marginal in the photo domain.

5 Discussion

The ImageNet experiments examine whether RSC benefits models with different capacities. RSC narrows performance gaps between same-family networks of different sizes, suggesting earlier performance saturation.

  • Standard ImageNet Benchmark: ImageNet experiments evaluated RSC with ResNet50, ResNet101, and ResNet152 backbones.The models shared the same architectural design while differing in capacity.
  • Standard ImageNet Benchmark: The authors report that RSC could induce faster performance saturation than increasing model size.They suggest scaling down deployed networks while maintaining comparable performance.

6 Conclusion

RSC is a simple, architecture-agnostic training heuristic that expands the features CNNs use for prediction and improves cross-domain generalization across standard and larger-scale settings.

  • 6 Conclusion: RSC forces CNNs to activate less-dominant features that remain correlated with labels, expanding the training-domain feature distribution.The method iteratively challenges dominant representations rather than adding model architecture.
  • 6 Conclusion: RSC applies to almost any CNN architecture with no extra model architecture and almost no additional computing effort.
  • 6 Conclusion: RSC achieves state-of-the-art improvement over baseline CNNs under standard domain-generalization settings with small networks and datasets.
  • 6 Conclusion: RSC also remains effective at larger scales, including the ImageNet database and networks up to ResNet152.

A1 Assumptions

The analysis assumes a shared optimal predictor across distributions, persistent label-relevant features, and perturbed-domain samples, while recognizing that multiple training-optimal hypotheses can generalize differently.

  • A1 Assumptions: The framework assumes a global optimum hypothesis θ⋆ matches the distribution-specific optimum θ⋆(D) for every distribution D.This can hold when the conditional distribution P(Y(D)|Z(D)) is identical across domains.
  • A1 Assumptions: The cross-domain challenge arises because multiple hypotheses may be optimal on one distribution while only some remain applicable to other distributions.A model can reduce training error by learning only one predictive signal even when additional signals are present.
  • A1 Assumptions: Assumption A2 interprets a shared set of features z_i as appearing across all distributions, with z_i identically distributed while other features may vary arbitrarily.The paper illustrates z_i with cat-related features such as ears, paws, and fur, while z_-i includes context such as sitting by a window.
  • A1 Assumptions: Assumption A3 models each distribution A as a perturbed version of O, retaining the ith-feature distribution while arbitrarily sampling the remaining features.The paper notes that train and test can use different subsets of the same underlying distribution, creating cross-domain variation.

A2.1 Corollary 1

The first corollary analyzes convergence for RSC and bounds the resulting empirical and population-risk terms using concentration arguments under the stated assumptions.

  • A2.1 Corollary 1: The RSC-related term has expectation 0 under Assumption A4 and is bounded by [0, ξ(p)].
  • A2.1 Corollary 1: For a fixed RSC hypothesis, the empirical risk over n i.i.d. samples is bounded within [0, 1] and has expectation L(θRSC).
  • A2.1 Corollary 1: The proof combines the terms with Hoeffding’s inequality before extending the argument from a fixed hypothesis to uniform convergence.The stated random variable has expectation L(θRSC) and bound [0, 1 + 2ξ(p)].
  • A2.1 Corollary 1: The corollary follows by rearranging the resulting uniform-convergence terms using standard arguments.

A2.2 Corollary 2

The second corollary’s proof decomposes the change between successive RSC hypotheses and controls higher-order terms through Taylor expansion and Assumption A6.

  • A2.2 Corollary 2: The proof compares predictions under successive RSC iterates on both original and perturbed representations.
  • A2.2 Corollary 2: The argument invokes the definition of RSC before applying Taylor expansion to h(θ̂RSC(t+1), ·) around θ̂RSC(t).
  • A2.2 Corollary 2: The higher-order terms are denoted by σ, and Assumption A6 permits dropping terms involving η^2 or higher orders.
  • A2.2 Corollary 2: The proof then evaluates the resulting expression for both z and its perturbed counterpart z̃ before rearranging terms to establish the conclusion.
Loading 2007.02454v1…