PaperlayerView exact revision on arXiv

Source-linked AI summary

PAPT++: Risk-Aware Adversarial Tuning and Generation for Single Domain Generalization

Zhipeng Xu, De Cheng, Xinyang Jiang, Lingfeng He, Huaijie Wang, Dongsheng Li, Nannan Wang, Xinbo Gao

arXiv:2609.04837v1cs.CV

TL;DR

SDG must generalize from one labeled source domain to unseen target domains, yet diversity-oriented augmentation may not expose classifier-specific weaknesses. PAPT++ searches a semantically constrained class-conditional T2I space for high-loss samples, alternates synthesis with classifier training, and reports strong benchmark performance with effective component ablations.

  • Problem

    SDG has only one labeled source domain, while robust generated samples must capture difficult variations beyond the source distribution rather than diversity alone.

  • Method

    PAPT++ learns diverse semantic references and uses them to constrain classifier-guided diffusion synthesis within a class-conditional semantic ambiguity set, alternating generation with classifier training.

  • Results

    PAPT++ achieves state-of-the-art performance on SDG and multi-source DG benchmarks, with ablations verifying the effectiveness of its main components.

  • Takeaways & Limitations

    Challenging yet semantically reliable synthesized samples are important for improving generalization to unseen domains.

Abstract

from arXiv · show

Single domain generalization (SDG) aims to learn a model from one labeled source domain that generalizes to unseen target domains. A common strategy is to enrich the source distribution with augmented or generated samples, and recent text-to-image (T2I) diffusion models provide a strong generative prior for this purpose. However, diversity alone is insufficient for robust generalization, because useful generated samples should also capture variations that the current classifier finds difficult. Motivated by distributionally robust optimization (DRO), we define a semantic ambiguity set in the class-conditional generative space of a pretrained T2I model and search it for samples with high classification loss under the current classifier. To this end, we introduce PAPT++, a risk-aware adversarial generation-training framework for SDG. PAPT++ first learns diverse semantic reference images for each class through image-text alignment and intra-class diversity regularization. These references then serve as denoising targets during classifier-guided diffusion synthesis, reducing semantic drift while guiding generation toward challenging variations. The generated samples are combined with the source data to update the classifier, and the updated classifier guides the next synthesis round in return. In this way, PAPT++ progressively exposes the classifier to challenging yet semantically consistent variations. Extensive experiments on standard SDG benchmarks demonstrate the superiority of the proposed PAPT++ method and the effectiveness of its main components.

I. INTRODUCTION

SDG must generalize from one labeled source domain to unseen domains, but conventional augmentation may miss complex shifts and diversity alone does not target classifier weaknesses. PAPT++ addresses this by searching a semantically constrained T2I generative space for challenging samples and alternating their generation with classifier training.

  • Single-domain generalization trains with one labeled source domain while targeting multiple unseen domains.
  • Predefined augmentation spaces may fail to cover complex shifts in style, background, viewpoint, or image quality.
  • T2I diffusion models provide broader visual and semantic variations, but useful samples must also expose variations the current classifier handles poorly.
  • PAPT++ searches a class-conditional T2I semantic ambiguity set for high-loss variations while semantic references constrain generation against drift.
  • CSRL learns aligned and diverse class references, while CADS uses them as denoising targets during classifier-guided synthesis and then retrains the classifier on generated challenges.
  • PAPT++ alternates challenging sample generation with classifier training and connects its reference denoising objective to DRO-based risk bounds.

B. Class-Level Semantic Reference Learning

PAPT++ learns class-level semantic references that preserve prompt-aligned class meaning while covering diverse within-class appearances. These fixed references later constrain risk-guided generation against semantic drift.

  • Reference Learning: Semantic reference learning combines image–text alignment with intra-class diversity regularization to produce class-consistent, varied images.The alignment objective encourages prompt correspondence, while DINOv2 feature similarity regularization promotes variation in shape, structure, and appearance.
  • Reference Learning: HPSv2 scores guide candidates toward better correspondence with class prompts and higher perceptual quality.The alignment loss is minimized so generated candidates better match the class-level textual condition.
  • Reference Learning: Diversity regularization yields richer eyeglass variations in viewpoint, frame shape, lens color, background, and type than generation without it.Without regularization, generated images tend toward similar front-view layouts, frame shapes, and blurred backgrounds.
  • Reference Learning: DINOv2 feature similarity is reduced among same-class candidates to expand the reference set beyond a small set of high-scoring visual patterns.The resulting variation covers object shape, structure, and appearance rather than only low-level pixel differences.
  • Reference Learning: After optimization, the adapted diffusion model samples fixed class reference sets that serve as denoising targets during classifier-guided synthesis.These targets constrain generation while preserving class-discriminative semantics during changes in domain style.

C. Classifier-Guided Adversarial Diffusion Synthesis

CADS searches the T2I generative space for samples that incur high classification loss while semantic references constrain the generator. It then combines generated samples with source data to update the classifier and repeats the process.

  • Adversarial Synthesis: CADS replaces PAPT’s diversity-driven domain generation with classifier-guided adversarial diffusion synthesis targeting difficult samples.Classification loss provides feedback for searching high-risk variations rather than only novel domain styles.
  • Risk-Guided Generation: At each round, a frozen classifier evaluates generated samples and its classification-risk signal guides the generator toward high-loss within-class variations.The classifier is warmed up on source data before iterative synthesis begins.
  • Semantic Constraint: Semantic-reference denoising regularizes the generation parameters rather than directly supervising each generated high-risk sample.This prevents risk maximization from moving the generator away from semantic references and discourages artifacts and distorted textures.
  • Semantic Constraint: The risk and denoising terms jointly seek samples with high classification loss while keeping the class-conditional generated distribution close to semantic references.Their balance produces high-risk yet semantically consistent samples for classifier training.
  • Classifier Update: Each round updates the classifier using both source-domain data and a class-balanced set of generated samples.The updated classifier then provides feedback for the next synthesis round.

D. LoRA Optimization and Sampling Strategy

PAPT++ replaces PAPT’s one-step denoising approximation with progressive trajectory guidance, while using semantic references and classifier feedback to support risk-aware synthesis and its theoretical analysis.

  • LoRA Optimization and Sampling Strategy: The training pipeline learns class-level reference images, adapts the diffusion model with LoRA, generates class-balanced samples, and alternates adversarial synthesis with classifier updates.The classifier is fixed during each synthesis phase, while generated samples form the training set for the subsequent classifier update.
  • LoRA Optimization and Sampling Strategy: PAPT++ uses a progressive denoising schedule that gradually adds earlier reverse-diffusion steps to back-propagation, extending guidance beyond the final step.The schedule stabilizes early optimization and later guides a longer denoising trajectory, producing images with fewer artifacts.
  • Theoretical Analysis: Reference-based denoising constrains generated distributions through a semantic ambiguity set, while classifier-guided synthesis searches for high-risk samples within that constraint.The reference denoising objective controls an upper bound on KL divergence between generated and smoothed reference distributions.
  • Theoretical Analysis: The target-domain risk is bounded by worst-case risk over the semantic ambiguity sets plus coverage gaps, class-wise search gaps, and finite-sample estimation error.Theoretical results account for ambiguity-set coverage, imperfect CADS search, and estimation from generated samples.
  • Theoretical Analysis: For a fixed hypothesis class, the generated-data estimation gap decreases as the number of generated samples per class increases.The bound treats the generated-data loss as a finite-sample estimate of classifier risk under CADS-generated class-conditional distributions.

A. Datasets and Evaluation Protocols

Experiments evaluate SDG across PACS, VLCS, OfficeHome, and CIFAR-10-C under source-only training protocols, using diffusion and backbone settings specified for reproducibility.

  • Datasets: Evaluation covers PACS, VLCS, OfficeHome, and corruption generalization from clean CIFAR-10 to CIFAR-10-C.PACS, VLCS, and OfficeHome use DomainBed benchmarks; CIFAR-10-C measures robustness to common corruptions.
  • Evaluation Protocol: Each PACS, VLCS, and OfficeHome domain is used once as the sole labeled source, with remaining domains held out as unseen targets.Target-domain data are excluded from training and model selection, and results are averaged across source-domain settings.
  • Evaluation Protocol: CIFAR-10-C evaluation trains only on clean CIFAR-10 and reports accuracy across corruption types and severity levels without adaptation.CIFAR-10-C images are not used during training or model selection.
  • Implementation Details: PAPT++ uses Stable Diffusion v1.5 with frozen backbone components and optimized LoRA parameters for semantic reference and high-risk generation.The reported setup uses LoRA rank 4, AdamW, and 80 total generation-related optimization steps.
  • Implementation Details: Downstream classifiers are ResNet-18 for PACS, VLCS, and CIFAR-10 and ResNet-50 for OfficeHome, trained with SGD during classifier-update stages.The generated-data loss weight is set to 1.0.

C. Comparison with State-of-the-Art Methods

PAPT++ improves domain-generalization performance across single-source, multi-source, and corruption benchmarks, while ablations attribute gains to risk guidance, semantic references, denoising, and iterative synthesis. The method also achieves strong accuracy with partial-parameter diffusion adaptation, though at higher GPU use than PAPT.

  • Single-Domain Results: PAPT++ reaches 73.43%, 74.21%, and 65.36% average accuracy on PACS, VLCS, and OfficeHome, improving over PAPT by 0.71, 0.84, and 3.33 pp.Improvements are especially pronounced for difficult source-domain settings and remain consistent across OfficeHome source domains.
  • Single-Domain Results: 79.37% average PACS accuracy with ResNet-50 surpasses PAPT and UDIM by 2.53 and 4.47 pp, respectively.The result indicates that PAPT++ gains persist when scaling from ResNet-18 to ResNet-50.
  • Corruption Generalization: 82.34% average CIFAR-10-C accuracy exceeds PAPT and UDIM with GAM by 8.23 and 7.04 pp, with PAPT++ best at every severity level.The reported gains are consistent across corruption severities.
  • Multi-Source Domain Generalization: On multi-source DG, PAPT++ achieves 55.18%, 72.36%, and 81.52% on TerraInc, OfficeHome, and VLCS, respectively.It improves over PAPT by 0.91, 1.15, and 0.68 pp on those benchmarks.
  • Ablation Study: Risk-aware generation improves over diversity-oriented PAPT by 0.71 pp on PACS and 0.84 pp on VLCS.The ablation compares Source Only, PAPT, and PAPT++ and attributes the difference to classifier-guided high-risk sample search.
  • Ablation Study: CSRL improves accuracy by 2.72 pp on PACS and 2.60 pp on VLCS, while CADS improves it by 5.94 and 2.79 pp, respectively.The results support complementary contributions from semantic references and classifier-guided synthesis.
  • Ablation Study: Adding classifier-risk guidance improves PACS and VLCS accuracy by 0.76 and 1.96 pp, while reference denoising adds 2.78 and 2.50 pp.The two objectives balance sample difficulty against class consistency.
  • Ablation Study: Accumulating samples across CADS rounds yields a maximum 4.06 pp gain on VLCS LabelMe over the first round.Later classifier feedback finds new high-risk variations while retaining earlier challenging examples.

APPENDIX A THEORETICAL ANALYSIS

The appendix formalizes PAPT++ as a model-restricted, DRO-inspired search over semantically feasible class-conditional distributions and relates this search to target-domain risk. It also defines coverage and search gaps that quantify mismatch between unseen target distributions and the generated ambiguity sets.

  • Reference and generated distributions: Gaussian smoothing makes the KL comparison between empirical reference latents and continuous generated distributions well-defined.The smoothing bandwidth controls the spread around reference latents and converges weakly to the empirical reference measure as it decreases.
  • Generator-induced semantic ambiguity set: The semantic ambiguity set contains distributions induced by LoRA parameters satisfying a class-wise semantic tolerance.CADS uses a shared LoRA parameter across classes, with a nonempty joint feasible set assumed.
  • Generator-induced semantic ambiguity set: PAPT++ maps semantically feasible LoRA parameters to a generator-induced family of class-conditional latent distributions.Semantic feasibility is imposed through the reference-based denoising objective.
  • Target-domain risk and ambiguity-set coverage: The target-domain risk is bounded through class-wise worst-case risks over the semantic ambiguity sets and their coverage of target distributions.The residual coverage gap ρ_k measures the remaining mismatch; smaller values indicate better coverage of the corresponding target-domain shift.
  • Regularity and search gaps: The class-wise search gap is zero when CADS reaches the worst-case risk and positive when finite shared-parameter optimization does not.The gap can arise because one shared LoRA parameter need not maximize every class risk simultaneously, and because optimization is finite-step and non-convex.

B. Proof of Theorem 1

Theorem 1 is proved by relating trajectory-level KL divergence to the reference-denoising loss under fixed-variance Gaussian diffusion assumptions, then applying data processing to obtain the marginal bound.

  • KL control: A technical lemma upper-bounds trajectory-level KL divergence by the semantic-reference denoising loss.The bound uses constants independent of the trainable LoRA parameters.
  • KL decomposition: Fixed covariance makes each reverse-transition KL term a coefficient-weighted squared noise-prediction error plus a parameter-independent term.The reconstruction term at the first timestep is handled separately through its negative log-likelihood.
  • KL decomposition: The forward-noising relation expresses the Gaussian transition means through the noise predictor, enabling the KL terms to be rewritten in denoising-loss form.The timestep coefficients depend on the diffusion schedules rather than on the LoRA parameters.
  • From trajectories to latents: Marginalizing intermediate diffusion states cannot increase KL divergence, so the trajectory-level bound yields a bound for the clean-latent distributions.This is obtained using the data-processing inequality under the projection onto z_0.

C. Proof of Theorem 2

Theorem 2 connects target-domain risk with worst-case risks over the semantic ambiguity sets by using the residual coverage gaps for each class.

  • Coverage-based risk bound: For each class, the residual coverage gap identifies an ambiguity-set distribution arbitrarily close to the unseen target class-conditional distribution.The proof applies the definition of ρ_k and then lets the approximation parameter tend to zero.
  • Coverage-based risk bound: Applying the class-wise bound across classes yields the target-domain risk relation in Theorem 2.The final step combines the class-wise inequalities with the class priors.

D. Proof of Proposition 1

Proposition 1 aggregates the class-wise CADS search gaps into bounds on the overall adversarial risk, including the uniform-class-prior case.

  • Class-wise search gaps: The class-wise search-gap definition bounds the discrepancy between CADS risk and the corresponding class-wise worst-case risk in a synthesis round.The proof fixes a synthesis round and applies the gap definition separately to each class.
  • Aggregated risk bound: Weighting the class-wise inequalities by class priors and summing produces the overall risk bound.This establishes the proposition for the general class-prior setting.
  • Uniform-prior specialization: Setting each class prior to 1/C gives the uniform-class-prior specialization of the aggregated bound.The proof derives the corresponding expression from the general weighted result.

E. Proof of Proposition 2

The proof establishes Proposition 2 through conditional sampling, symmetrization, and concentration arguments. It uses bounded loss changes under sample replacement and derives the stated equations with high probability.

  • Conclusion: The proof concludes by deriving the target equations, including Eq. 48, Eq. 51, and the displayed high-probability bound.The round-wise feasibility condition supplies the starting point for the proposition’s class-wise argument.
  • Finite-sample analysis: The finite-sample argument conditions on the round-specific representation and treats the class-balanced latent-label sample as conditionally independent.Subsequent probabilities and expectations are taken under this conditional sampling distribution.
  • Finite-sample analysis: Because losses are bounded, replacing one sample changes the empirical quantity by at most Lmax/(CMg), enabling McDiarmid’s inequality.The resulting concentration statement holds with probability at least 1 −δ conditional on ϕ(r).
  • Symmetrization: A standard symmetrization argument introduces a conditionally independent ghost sample and independent Rademacher variables.The proof then uses coordinate-wise exchangeability, the triangle inequality, and Eq. 100.

APPENDIX B FURTHER ANALYSIS

The appendix analyzes CADS sample quality across synthesis rounds and guidance settings. Results indicate that classifier feedback produces increasingly difficult but class-consistent variations, while CFG++ with λ = 0.6 balances semantic fidelity and sample difficulty.

  • CADS synthesis analysis: ImageReward trajectories vary across classes and are often non-monotonic because classifier updates change which variations remain difficult.The analysis uses λrisk = 0.5 to strengthen classifier feedback and create a more challenging synthesis setting.
  • Guidance comparison: CFG++ with λ = 0.6 helps CADS balance semantic fidelity and sample difficulty while generating class-consistent samples for unseen-domain generalization.This conclusion is supported by the quantitative tables and ImageReward analyses on PACS and VLCS.
  • Notation: The theoretical appendix groups notation by semantic ambiguity sets, target-domain risk, adversarial search, and finite-sample generalization.The symbols index classes, diffusion timesteps, synthesis rounds, total diffusion steps, and target domains.
  • Round-wise evolution: Across eight dog-class synthesis rounds, early samples are clean and canonical, whereas later rounds explore less typical variations in style, texture, shape, and composition.The later samples are accompanied by a general decrease in mean ImageReward.
Loading 2609.04837v1…