PaperlayerView exact revision on arXiv

Source-linked AI summary

A Closer Look at Smoothness in Domain Adversarial Training

Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, R. Venkatesh Babu

arXiv:2206.08213v1cs.LGcs.CV

TL;DR

Domain Adversarial Training combines task and adversarial losses, but the optimization role of smoothness in this setting has been insufficiently analyzed. The paper studies loss-surface curvature, introduces SDAT to smooth only task loss, and finds more stable training and improved target-domain performance, while adversarial-loss smoothing is harmful.

  • Problem

    DAT combines task and adversarial losses, yet the effects of explicitly improving its optimization smoothness have not been sufficiently analyzed.

  • Method

    The paper analyzes DAT loss-landscape curvature and introduces SDAT, which enhances smoothness only with respect to task loss.

  • Results

    SDAT improves existing domain-adversarial methods across classification and object-detection settings, while smoothing adversarial loss leads to suboptimal target-domain generalization.

  • Takeaways & Limitations

    Task-loss smoothness stabilizes DAT and supports better target-domain generalization, whereas adversarial-loss smoothness should be avoided in practice.

  • Takeaways & Limitations

    SDAT has no automatic way to select ρ, and its analysis assumes the discriminator is L-smooth with a small η.

Abstract

from arXiv · show

Domain adversarial training has been ubiquitous for achieving invariant representations and is used widely for various domain adaptation tasks. In recent times, methods converging to smooth optima have shown improved generalization for supervised learning tasks like classification. In this work, we analyze the effect of smoothness enhancing formulations on domain adversarial training, the objective of which is a combination of task loss (eg. classification, regression, etc.) and adversarial terms. We find that converging to a smooth minima with respect to (w.r.t.) task loss stabilizes the adversarial training leading to better performance on target domain. In contrast to task loss, our analysis shows that converging to smooth minima w.r.t. adversarial loss leads to sub-optimal generalization on the target domain. Based on the analysis, we introduce the Smooth Domain Adversarial Training (SDAT) procedure, which effectively enhances the performance of existing domain adversarial methods for both classification and object detection tasks. Our analysis also provides insight into the extensive usage of SGD over Adam in the community for domain adversarial training.

1. Introduction

The paper analyzes how smoothness-targeting optimization affects Domain Adversarial Training and proposes smoothing only the task loss to stabilize training and improve target-domain generalization.

  • Motivation: Domain Adversarial Training learns domain-invariant neural representations and supports unsupervised adaptation from labeled source data to unlabeled target data.It has been applied across recognition, fairness, object detection, domain generalization, and image-to-image translation.
  • Motivation: Directly applying smooth-minima techniques to DAT does not significantly improve target-domain generalization.DAT combines adversarial-loss and task-loss objectives, but its optimization behavior has received limited explicit analysis.
  • Findings: The analysis finds that smoothing task loss stabilizes DAT, whereas smoothing adversarial loss produces suboptimal performance.The paper attributes the difference to the distinct effects of smoothness on the task and adversarial components.
  • Method: SDAT enhances smoothness only with respect to task loss and is theoretically motivated through a target-error generalization bound.The procedure can be combined with existing methods and requires one additional gradient computation step.
  • Evaluation: With a ViT backbone, SDAT achieves an effective average gain of 3.1% over the baseline without an additional module or pre-training data.The reported setup uses only a 12 GB GPU and produces state-of-the-art domain-adaptation performance.

2. Related Work

Related work covers unsupervised domain adaptation, DAT-based alignment, smooth loss landscapes, and Hessian-based evidence connecting optimizer-induced smoothness with target accuracy.

  • Unsupervised Domain Adaptation: Unsupervised Domain Adaptation learns representations from labeled source data that generalize to unseen target-domain data.DAT is a prominent approach that uses a discriminator and Gradient Reversal layer to encourage domain-invariant features.
  • DAT Methods: CDAN improves DANN by using multi-linear conditioning to align source and target distributions more effectively.The paper uses an implementation of CDAN as its default DAT method.
  • Smoothness of Loss Landscape: Smooth-minima methods aim to improve generalization in overparameterized neural networks, with SAM providing a practical approach through additional computations.The related-work discussion notes that low training error alone does not always imply better generalization.
  • Hessian Analysis: Hessian eigenvalue spectra compare the loss landscapes of Adam, SGD, and SDAT using maximum eigenvalue and trace as smoothness indicators.Lower Tr(H) and λmax indicate a smoother loss surface; SGD reaches a smoother minimum than Adam in the reported comparison and has higher target accuracy.

3. Background

The background formulates unsupervised adaptation through source and target risks, a domain discrepancy, and a min-max adversarial objective implemented with a feature extractor, classifier, discriminator, and Gradient Reversal layer.

  • Problem Setup: Unsupervised adaptation uses labeled source samples and unlabeled target samples to learn a hypothesis with low target risk.Source and target samples are drawn from distributions P_S and P_T, with source risk defined through a task loss.
  • Domain Discrepancy: The discrepancy is defined through a variational f-divergence formulation over a hypothesis space H.The formulation uses the Fenchel conjugate φ* of a lower semi-continuous convex function φ satisfying φ(1)=0.
  • Generalization Bound: The target-risk bound combines source risk, domain discrepancy, and an irreducible joint-classifier error.Reducing source risk and discrepancy therefore reduces the bound on target risk under the stated framework.
  • DAT Objective: DAT composes a feature extractor, classifier, and domain discriminator, optimizing task loss and adversarial loss through a min-max procedure.The discriminator distinguishes source and target features, while the adversarial term is optimized with a Gradient Reversal layer; CDAN is the default method.

4. Analysis of Smoothness

The analysis links task-loss smoothness to stable domain adversarial training and better target-domain generalization, while smoothing the adversarial loss worsens discrepancy estimation and target performance.

  • Task-loss smoothness: Low λmax and Tr(H) indicate smoother task-loss landscapes, which the paper uses to analyze DAT optimization.The Hessian eigen-spectrum provides the curvature-based smoothness measures.
  • Training stability: Increasing task-loss curvature destabilizes training, whereas SDAT keeps λmax low across epochs and maintains more stable validation accuracy.The comparison is reported for adaptation from Infographic to Clipart and supported by additional adaptation tasks.
  • Task-loss smoothness: SGD reaches smoother task-loss minima than Adam and is associated with better target-domain performance in DAT.The paper connects this observation to the widespread use of SGD over Adam for DAT.
  • Adversarial-loss smoothness: Smoothing the adversarial loss lowers domain classification accuracy, indicating ineffective discrepancy estimation and sub-optimal target-domain generalization.Further increasing adversarial-loss smoothness decreases target accuracy.
  • SDAT formulation: SDAT smooths only the task-loss minimum while preserving the original discrepancy term, and its objective is supported by a target-error generalization bound.The bound includes sharpness-aware source risk, source-target discrepancy, a weight-norm term, and an ideal-hypothesis term.
  • SDAT formulation: The proposed SDAT objective is reported to yield significantly lower generalization error than the original domain-adaptation objective.The method can modify any DAT baseline with a few lines of code.

5. Adaptation for classification

SDAT is evaluated with CDAN and CDAN+MCC across Office-Home, VisDA-2017, and DomainNet, including ResNet and ViT backbones, and improves classification adaptation performance.

  • Experimental setup: SDAT is combined with CDAN and CDAN+MCC and evaluated on Office-Home, VisDA-2017, and DomainNet.Experiments include ResNet backbones and ViT results on Office-Home and VisDA-2017.
  • Office-Home: On Office-Home, CDAN+MCC with SDAT achieves state-of-the-art adversarial adaptation performance with a ResNet-50 backbone.SDAT improves CDAN and CDAN+MCC across the majority of source-target domain pairs.
  • DomainNet: On DomainNet, SDAT improves CDAN across all reported source-target pairs, including a 4.5% gain for inf → real and nearly 1.8% overall.The paper attributes the gains to stabilized adversarial training.
  • VisDA-2017: On VisDA-2017, CDAN with SDAT improves overall performance by more than 1.5% with ResNet and 4.9% with ViT.With CDAN+MCC, SDAT yields 89.8% accuracy across classes and improves minority classes including bicycle, car, and truck.

6. Adaptation for object detection

For object detection, SDAT is tested on two domain shifts and improves adaptation most when smoothness is applied to classification loss alone rather than to combined classification and regression loss.

  • Experimental setup: Object-detection experiments use DA-Faster as the baseline and mAP at 0.5 IoU as the evaluation metric.The study evaluates Pascal → Clipart and Cityscapes → Foggy Cityscapes domain shifts.
  • Method variants: SDAT-Classification smooths the classification loss, whereas SDAT smooths the combined classification and regression loss.These are the two smoothness-enhancement variants evaluated for detection.
  • Results: On average, SDAT-Classification gains 2.0% mAP over SDAT and 2.8% over the DA-Faster baseline.The comparison is reported across the two domain shifts and varying training batch sizes.
  • Results: SDAT-Classification improves mAP by 1.3% on Pascal → Clipart and 2.8% on Cityscapes → Foggy Cityscapes.The improvement remains consistent at batch size 8, reaching 4.3% mAP improvement.
  • Results: Smoothing both classification and regression still improves DA-Faster by 0.9% on average, but is less effective than smoothing classification alone.The results support SDAT’s applicability to adaptation methods with adversarial components.

7. Discussion

The discussion finds that task-loss smoothing improves domain-adversarial stability and target performance, whereas adversarial-loss smoothing is harmful. SDAT shows consistent gains across noise settings, smoothing methods, and domain-adaptation algorithms.

  • How much smoothing is optimal?: ρ = 0.02 performs best on Office-Home Ar→Cl, outperforming the baseline by at least 1.5%.The optimal ρ is generally shared across domains within a dataset but varies between datasets.
  • Which components benefit from smooth optima?: Smoothing the adversarial loss reduces performance to 51.0%, below even the DAT baseline.The comparison evaluates smoothing task loss, adversarial loss, or both using SAM.
  • Is it Robust to Label Noise?: SDAT remains robust under injected label noise, with smoother minima producing models that generalize well on the target domain.Figure 4B compares SGD and SDAT across different percentages of label noise.
  • Is it better than other smoothing techniques?: Label Smoothing, SAM, and VAT lack significant consistent gains, while SDAT outperforms SWAD by an average margin of 1.2%.The comparison tests smoothing techniques originally proposed for empirical risk minimization.
  • Does it generalize well to other DA methods?: DANN with SDAT gains an average of 3.3% on Office-Home with a ViT backbone, and SDAT also improves GVB-GD and other DAT methods.The reported results cover CDAN, CDAN+MCC, DANN, and GVB-GD across multiple datasets and backbones.

8. Conclusion

The conclusion states that task-loss smoothness stabilizes domain-adversarial training and improves target-domain generalization, while adversarial-loss smoothing is sub-optimal. SDAT operationalizes this finding and achieves state-of-the-art performance, but requires manual selection of ρ.

  • 8. Conclusion: Task-loss smoothing stabilizes DAT and improves generalization on the target domain, whereas adversarial-component smoothing produces sub-optimal results.This conclusion is supported theoretically and empirically for unsupervised domain adaptation.
  • 8. Conclusion: SDAT increases smoothness only with respect to task loss and improves existing adversarial domain-adaptation methods on benchmark datasets.The method is presented as practical and effective for current adversarial DA methods.
  • 8. Conclusion: SDAT has no automatic procedure for selecting ρ, the extent of smoothness.The authors identify automatic ρ selection as a direction for future work.

C. Proof of Theorems

The paper’s theoretical appendix develops smoothness-based bounds and analyzes how gradient steps affect discrepancy estimation. It also documents Hessian measurements, GAN evidence, and implementation settings supporting the empirical claims.

  • Proof assumptions: The proofs assume L-smoothness and analyze one alternating gradient-ascent step for estimating the domain discrepancy.The single-step analysis reflects the reverse-gradient procedure used to measure discrepancy.
  • Theoretical implication: The discrepancy estimate can be sub-optimal when the optimality gap is large, because differences between gradient directions then become potentially large.The appendix connects the bound on directional differences to the distance from the minimum.
  • Generalization bound: The generalization proof combines a source-risk bound with a domain-adaptation bound involving source and target risk.The derivation uses a PAC-Bayes-style source-risk result and an f-divergence adaptation bound.
  • Hessian analysis: Hessian experiments use PyHessian and subsets of source data because full Hessian computation is computationally expensive.Office-Home uses 50% of source data, while DomainNet uses 10% for Hessian calculation.
  • Additional GAN evidence: Smoothing the discriminator in SNGAN also yields sub-optimal performance, measured by higher FID across both datasets.This experiment extends the adversarial-loss smoothness analysis beyond domain-adversarial training.
  • Implementation details: SDAT experiments evaluate multiple backbones, datasets, and adaptation methods under specified SGD-based training configurations.The appendix reports settings for Office-Home, VisDA-2017, DomainNet, ViT experiments, reproducibility controls, and object detection.

F.2. Additional Implementations Details for DA for Object detection

For object detection, SDAT smooths classification losses in Faster R-CNN’s RPN and ROI components using source data. Smoothing the regression loss was less effective.

  • F.2. Additional Implementations Details for DA for Object detection: SDAT smooths the RPN and ROI classification losses in Faster R-CNN during source-data training for domain-adaptive object detection.The implementation uses Detectron2 and SGD with momentum 0.9 for 70k iterations.
  • F.2. Additional Implementations Details for DA for Object detection: Smoothing the regression loss is less effective than smoothing the classification loss.

G. Additional Results

Additional experiments show that SDAT improves existing domain-adaptation methods across datasets, backbones, and smoothing comparisons, including classification and object detection settings.

  • Classification results: CDAN w/ SDAT improves overall DomainNet accuracy by 1.8% and improves all three evaluated domain-adaptation methods.The evaluated methods are DANN, CDAN, and CDAN+MCC.
  • Classification results: CDAN w/ SDAT outperforms CDAN by 1.7% with ResNet-101 and 4.4% with ViT on VisDA-2017 overall accuracy.The results concern the large-scale Synthetic →Real shift.
  • Smoothing comparisons: On Office-Home, SDAT averages 61.6% accuracy, a +2.4% gain over DAT, compared with SWA’s 60.4% accuracy and +1.2% gain.The paper attributes the difference to SDAT selectively smoothing minimization components, whereas SWA averages weights affected by both losses.
  • Smoothing comparisons: SDAT significantly outperforms other evaluated smoothing techniques, while those techniques also improve DAT performance relative to unsmoothed training.This comparison supports smoothing as useful for adaptation, while distinguishing SDAT’s stronger performance among the tested techniques.
  • Hyperparameter analysis: A smoothing parameter of ρ = 0.02 works robustly across experiments and is proposed as a rule of thumb, although it is not always best.The result is reported for DomainNet and VisDA-2017 parameter studies.

K. Significance and Stability of Empirical Results

Repeated-seed experiments and validation-accuracy analyses support the reliability and stability of SDAT’s improvements over domain-adaptation baselines.

  • Experimental reliability: 36 DomainNet experimental runs across three random seeds compare 18 CDAN w/ SDAT runs with 18 CDAN baseline runs.The experiments cover a subset of splits representative of different source domains because each experiment takes approximately 20 hours.
  • Stability analysis: SDAT consistently improves validation accuracy across six DomainNet splits and later-epoch confidence intervals do not overlap with the baseline.The Infographic-source split also shows significant training stabilization.
  • Experimental reliability: Median validation accuracy over the last five epochs shows gains similar to those reported in the main results.The median reduces sensitivity to outlier checkpoints.
  • Experimental reliability: Across four Office-Home splits and three seeds, CDAN w/ SDAT produces consistent improvement despite variance in baseline CDAN results.The dataset contains 44 images per class, contributing to baseline variance.
  • Implementation: SDAT can be integrated into DAT with an additional forward pass and gradient computation, followed by separate updates for the feature-task model and domain classifier.The pseudocode applies a sharpness-aware update using task loss, then backpropagates task and domain losses together before updating the domain classifier.
Loading 2206.08213v1…