PaperlayerView exact revision on arXiv

Source-linked AI summary

AdvFD: Boosting Visual Generation via Adversarial Fr'echet Distance Loss

Mingju Gao, Jingkai Zhou, Kun Gai, Changqian Yu, Hao Tang

arXiv:2608.11205v1cs.CV

TL;DR

Static pretrained feature spaces can overlook discrepancies, allowing Fréchet hacking as target metrics improve while visual quality or other-feature alignment stagnates. AdvFD adds an adversarially learned representation with real-feature whitening, and consistently improves one-step generation across backbones and model scales.

  • Problem

    Static pretrained representations provide incomplete, fixed supervision, allowing target Fréchet metrics to improve while visual quality or other-feature alignment stagnates or worsens.

  • Method

    AdvFD adds an adversarially learned representation to static Fréchet targets and uses real-feature whitening to stabilize the adaptive comparison space.

  • Results

    AdvFD consistently improves one-step generation across JiT and pMF backbones and model scales, including optimized and held-out Fréchet metrics.

  • Takeaways & Limitations

    AdvFD improves the robustness of Fréchet-based distribution matching beyond the static feature spaces used during training while preserving one-step sampling.

  • Takeaways & Limitations

    Without whitening, the adversarial representation could increase raw Fréchet distance by enlarging feature norms rather than exposing additional distributional differences.

Abstract

from arXiv · show

Fréchet distance has recently emerged as an effective distribution-level objective for generator post-training, complementing the conventional sample-level diffusion and flow-matching losses. However, directly optimizing Fréchet objectives can cause Fréchet hacking. The target metrics keep improving, but visual quality and Fréchet alignment in other feature spaces may stagnate or deteriorate. We attribute this failure to the static pretrained feature spaces used by existing Fréchet losses. These feature spaces provide incomplete and fixed views of the differences between real and generated distributions. To address this limitation, we propose Adversarial Fréchet Distance (AdvFD), which complements the static representation targets in FD-Loss with a calibrated adversarially learned representation. AdvFD augments the original static Fréchet objective with a learnable representation that adversarially maximizes the Fréchet discrepancy between real and generated samples, while the generator minimizes the same discrepancy in the resulting adaptive feature space. To prevent the adversarial representation from trivially increasing the objective through feature amplification, we further introduce real-feature whitening, which normalizes its scale and covariance geometry and stabilizes the min--max optimization. Extensive experiments show that AdvFD consistently improves one-step generator post-training across both JiT and pMF backbones and across different model scales.

1 INTRODUCTION

The paper identifies Fréchet hacking as a limitation of static representation targets: optimized metrics can improve while visual quality and alignment under other encoders deteriorate. It proposes AdvFD, which adds an adversarially learned representation and real-feature whitening to expose residual discrepancies and stabilize training.

  • The Limits of Static Representation Targets: FD-Loss matches real and generated first- and second-order feature statistics using pretrained encoders without paired targets.The encoders include Inception, SigLIP, and MAE.
  • The Limits of Static Representation Targets: Fréchet hacking occurs when generators specialize to selected feature spaces, reducing target FD while preserving human-visible or cross-encoder errors.A universal perturbation can introduce visible high-frequency artifacts while reducing Inception FID.
  • Adversarial Fréchet Distance: AdvFD complements static FD targets with a learnable representation that adversarially exposes residual discrepancies between real and generated distributions.The generator minimizes the Fréchet objective defined in the adaptive feature space.
  • Adversarial Fréchet Distance: Real-feature whitening stabilizes adversarial representation learning by preventing trivial feature amplification during Fréchet maximization.The method complements adversarial learning with normalization of the representation’s scale and covariance geometry.
  • Experiments: AdvFD improves optimized and unseen-encoder Fréchet metrics while reducing visual artifacts across ImageNet class-conditional generators, backbones, and model scales.The experiments compare AdvFD with FD-loss and alternative remedies.

2 RELATED WORK

Prior work established Fréchet distances as evaluation tools and incorporated feature-statistic discrepancies into adversarial training objectives. Objective hacking describes improving an imperfect proxy through unpenalized artifacts rather than intended visual quality, motivating mitigation strategies across training stages.

  • Fréchet Distance as Evaluation: Fréchet-based evaluation compares real and generated image distributions using feature statistics, with FID relying on first- and second-order statistics from pretrained Inception features.The passage lists Fréchet-based distances as standard image-generation evaluation tools and identifies FID as a representative metric.
  • Fréchet Distance as Training: Prior training methods match or optimize feature-statistic discrepancies through norm-based IPMs, normalized critics, constrained encoders, or discriminator-feature Fréchet distance.The cited methods are McGAN, Fisher GAN, MMD-GAN, and Fréchet GAN.
  • Objective Hacking: Objective hacking occurs when generators over-optimize an imperfect proxy, improving measured objectives by exploiting unpenalized artifacts instead of improving intended visual quality.Existing mitigation strategies are grouped into pretraining-stage and post-training approaches, with adaptive feedback cited for pretraining-stage distribution matching.

3 FR´ECHET HACKING UNDER STATIC REPRESENTATION MATCHING

Static pretrained feature matching can reduce Fréchet distance while overlooking visual and semantic discrepancies, enabling Fréchet hacking. The section motivates adaptive representations because fixed encoders provide incomplete coverage of generator–real distribution differences.

  • Static representation matching: FD-Loss minimizes Fréchet distance between real and generated feature distributions, aligning first- and second-order statistics without teacher distillation.It provides a direct representation-space training objective without paired reconstruction targets.
  • Fréchet hacking: 3.31 to 2.56: Inception FID decreased for a visibly artifact-inducing universal perturbation, exposing an exploitable blind direction in the static representation.The perturbation was optimized solely for Inception FID while the pMF-B generator remained frozen.
  • Fréchet hacking: 29.4%: FD-r-Inception decreased from 50k-75k steps, while FD-r-CLIP increased by 8.5% during JiT-B post-training.The opposing metric trends demonstrate that improving the target static representation can coincide with worsening alignment in another feature space.
  • Fréchet hacking: Static feature spaces constrain only discrepancies their representations detect, allowing important visual structures and semantic details to degrade while optimized FD decreases.Visual quality and alignment in other feature spaces, such as CLIP, may stagnate or worsen.
  • From Static to adversarial representations: Adding frozen encoders broadens feature coverage but increases computational and memory costs and can still miss discrepancies that emerge during generator adaptation.These limitations motivate extending Fréchet training beyond a fixed encoder set toward learnable representations.

4 METHOD

AdvFD augments frozen Fréchet objectives with a trainable representation that adversarially exposes residual real–generated discrepancies while the generator minimizes them. Real-feature whitening prevents the adversarial representation from increasing the discrepancy through feature-scale amplification, enabling alternating bounded updates.

  • Adaptive Fréchet objective: AdvFD combines frozen representations with a pretrained-initialized trainable representation that adapts the comparison space as the generator evolves.Frozen spaces provide stable supervision but may overlook residual discrepancies; ψω adapts to the current generator.
  • Alternating optimization: The generator minimizes static and adaptive Fréchet discrepancies, while the adversarial representation maximizes the adaptive discrepancy to reveal remaining distribution differences.These alternating updates form a GAN-like game in feature space.
  • Alternating optimization: During the G-step, representations are frozen and gradients pass only through generated samples; during the D-step, the generator is frozen and the representation takes a bounded local response.The D-step uses gradient-norm clipping with threshold τ and AdamW, alternating throughout post-training.
  • Preventing Feature-Scale Explosion: Raw feature-space FD admits a trivial scale direction, allowing the adversarial representation to enlarge feature norms without exposing additional distributional differences.This feature-scale explosion motivates real-feature whitening.
  • Preventing Feature-Scale Explosion: Real-feature whitening normalizes the adaptive discrepancy by the real distribution’s scale and covariance geometry, so global rescaling alone cannot increase Dadv.Under exact whitening, affine-equivalent representations differ only by an orthogonal transformation, under which FD is invariant.
  • Relation to adversarial training: Unlike WGAN’s fixed input-space geometry with a learned scalar transport potential, AdvFD learns the feature geometry while retaining a fixed Gaussian transport functional.AdvFD compares mean and covariance statistics of representation vectors using a fixed closed-form W2-based functional.

5 EXPERIMENT

AdvFD is evaluated on ImageNet-1K across JiT and pMF backbones and scales, consistently improving FD-loss while preserving one-step generation. Ablations show that adversarial-feature whitening and pretrained Inception initialization are important for stable, effective optimization, with qualitative gains across architectures.

  • Datasets and Training Protocol: AdvFD is evaluated on ImageNet-1K 256 × 256 using JiT and pMF generators across B-, L-, and H-scale variants.All methods share pretrained checkpoints, training data, optimization budgets, and sampling configurations.
  • Quantitative Results: 21.0%, 29.1%, and 28.6% relative improvements on JiT-B reduce FID from 1.00 to 0.79, FD-r6 from 5.53 to 3.92, and FD-r3 from 8.45 to 6.03.AdvFD consistently outperforms FD-loss across JiT and pMF scales while preserving one-step generation.
  • Quantitative Results: FD-r3 reductions across architectures and scales show that AdvFD generalizes to representations excluded from training rather than only optimized SIM spaces.The evaluation reports FID, FD-r6, and held-out FD-r3, with FD-r6 aggregating six visual representations.
  • Qualitative Results: AdvFD produces more coherent object structures, cleaner textures, and fewer local artifacts than FD-Loss on representative JiT-L and pMF-L comparisons.These qualitative improvements are consistent across both generator architectures.
  • Ablation on Whitening Location: Without whitening, FID, FD-r6, and FD-r3 deteriorate to 10.69, 58.50, and 40.54, respectively, while whitening the adversarial representation stabilizes optimization.Real-feature whitening suppresses feature-scale inflation and removes the incentive to increase adversarial Fréchet discrepancy through global scaling.
  • Ablation on Adversarial Representation Backbones and Initialization: Pretrained Inception performs best across all metrics because its well-conditioned feature space focuses adversarial updates on residual real–generated discrepancies.Random initialization more readily maximizes FD through low-level shortcuts, while SigLIP and MAE underperform due to sensitivity to fine-grained textures and local artifacts.

6 CONCLUSION · A FINITE-HORIZON BOUNDEDNESS OF REPRESENTATION ADAPTATION

AdvFD adaptively exposes discrepancies missed by static feature spaces, improving one-step ImageNet generation and Fréchet metrics without increasing sampling cost. The appendix establishes finite-horizon boundedness of representation adaptation while noting that whitening remains necessary to address raw Fréchet scale degeneracy.

  • 6 CONCLUSION: AdvFD improves the robustness of Fréchet-based distribution matching under static representation targets.It uses calibrated adversarial representation training to expose residual discrepancies overlooked by static feature spaces.
  • 6 CONCLUSION: AdvFD improves one-step ImageNet generation across different backbones while preserving the same 1-NFE sampling cost.The conclusion attributes this improvement to adaptively exposing residual discrepancies missed by static feature spaces.
  • 6 CONCLUSION: AdvFD consistently enhances both optimized Fréchet metrics and metrics measured in unused feature spaces.This indicates gains beyond the feature spaces directly optimized during training.
  • A FINITE-HORIZON BOUNDEDNESS OF REPRESENTATION ADAPTATION: Finite, gradient-clipped AdamW updates in AdvFD remain within a bounded neighborhood of their initialization.Clipped gradients and bias-corrected moments yield bounded update directions under the stated assumptions.
  • A FINITE-HORIZON BOUNDEDNESS OF REPRESENTATION ADAPTATION: The first K iterates form a compact reachable set, and continuity implies that the regularized adaptive Fréchet discrepancy attains a finite maximum.This conclusion assumes compact input support and a continuous representation network.
  • A FINITE-HORIZON BOUNDEDNESS OF REPRESENTATION ADAPTATION: The boundedness result applies only to the implemented optimizer trajectory over a finite horizon.It does not establish boundedness for unrestricted optimization or eliminate raw Fréchet scale degeneracy.
  • A FINITE-HORIZON BOUNDEDNESS OF REPRESENTATION ADAPTATION: Real-feature whitening addresses the scale degeneracy of raw Fréchet distance that finite-horizon trajectory boundedness does not remove.Whitening is therefore required separately from the optimizer boundedness argument.

B DERIVATION AND AFFINE INVARIANCE OF REAL-FEATURE WHITENING · C TRANSPORT-BASED RELATION TO GANS

Real-feature whitening makes the adaptive Fréchet discrepancy invariant to affine reparameterizations, preventing feature-scale inflation while introducing controlled regularization and EMA-related approximations. In transport terms, AdvFD learns the feature geometry for a fixed Gaussian second-order transport functional, unlike WGAN’s learned dual potential under fixed geometry.

  • B DERIVATION AND AFFINE INVARIANCE OF REAL-FEATURE WHITENING: Raw Fréchet distance is not affine-invariant, so expanding feature coordinates can increase discrepancy without revealing additional distributional differences.The scaling transformation A = cI demonstrates the coordinate-dependent failure that motivates whitening.
  • B DERIVATION AND AFFINE INVARIANCE OF REAL-FEATURE WHITENING: Real-feature whitening centers adversarial features and normalizes their covariance to identity, with the symmetric whitener selected because common orthogonal rotations preserve Fréchet distance.After whitening, affine-equivalent representations become orthogonally equivalent, satisfying the desired invariance.
  • B DERIVATION AND AFFINE INVARIANCE OF REAL-FEATURE WHITENING: Global coordinate scaling cannot increase the whitened discrepancy, because for A = cI with c > 0 the induced orthogonal transformation is the identity.Thus, whitening removes the adversarial representation’s trivial scale-amplification direction.
  • B DERIVATION AND AFFINE INVARIANCE OF REAL-FEATURE WHITENING: Covariance regularization makes minibatch whitening well-defined for ill-conditioned or rank-deficient estimates but approximates exact whitening by O(ϵ/λmin) on well-conditioned directions.Small-eigenvalue directions incur larger error and are intentionally regularized to avoid unstable inverse scaling.
  • B DERIVATION AND AFFINE INVARIANCE OF REAL-FEATURE WHITENING: The implementation uses detached EMA real-feature statistics, so affine invariance holds only up to controlled covariance-regularization and statistic-drift errors.The stated setting has ηD ≤2 × 10−6, with the adversarial representation updated every two generator steps.
  • C TRANSPORT-BASED RELATION TO GANS: WGAN fixes a sample-space ground cost and learns a scalar dual transport potential, whereas AdvFD learns the representation that defines the feature-space comparison geometry.WGAN’s critic solves the dual problem without changing inter-sample distances; AdvFD changes the representation-induced distances.
  • C TRANSPORT-BASED RELATION TO GANS: For a fixed representation, AdvFD evaluates a closed-form squared Wasserstein-2 transport between Gaussian models matched by feature means and covariances.The mean term measures center translation, while the covariance term measures scale and orientation alignment.
  • C TRANSPORT-BASED RELATION TO GANS: AdvFD maximizes this Gaussian transport discrepancy over reachable clipped representations and then has the generator minimize the resulting discrepancy.Its structure is learned feature geometry plus a fixed Gaussian transport functional.

D MORE EXPERIMENT RESULTS

Experiments show that AdvFD benefits from a properly calibrated adversarial loss weight and from adaptively updating its learned representation. Under equal wall-clock budgets, these gains exceed those from freezing the representation or extending optimization of static FD objectives.

  • Ablation on Adversarial Loss Weight: All nonzero λadv values improve over the FD-loss baseline, while λadv=0.10 achieves the best FD-r6 and FD-r3 results.A weight of 0.01 provides limited gains, particularly on FD-r3, whereas 0.05 and 0.10 perform substantially better.
  • Wall-clock-aligned comparison: Freezing the adversarial representation substantially degrades all metrics because it cannot adapt to the evolving generated distribution.The frozen variant retains the additional representation branch, isolating the effect of adversarial representation learning.
  • Wall-clock-aligned comparison: Extending FD-loss to match AdvFD’s wall-clock cost provides no improvement, showing that longer optimization of static objectives is insufficient.This comparison indicates that the gains are not attributable to additional computation alone.
  • Wall-clock-aligned comparison: 0.79 FID, 3.92 FD-r6, and 6.03 FD-r3 are achieved by AdvFD under the same wall-clock budget.The results confirm that adaptively updating the representation exposes residual real–generated discrepancies.

E IMPLEMENTATION AND EVALUATION DETAILS · E.1 IMPLEMENTATION DETAILS

The implementation evaluates post-training on ImageNet-1K at 256 × 256 using the original pMF and JiT architectures, primarily combining FD-SIM with FD-Adv. It specifies representation, statistic-tracking, optimization, and evaluation procedures for the generator and adversarial critic.

  • E.1 IMPLEMENTATION DETAILS: Experiments use ImageNet-1K at 256 × 256, released base checkpoints, unchanged pMF and JiT architectures, and primarily the FD-SIM plus FD-Adv objective.Training hyperparameters are summarized in Table 6.
  • E.1 IMPLEMENTATION DETAILS: pMF and JiT use 2D rotary and learned positional embeddings, while JiT retains the legacy time convention and pMF disables its auxiliary velocity head.Large-scale experiments enable gradient checkpointing; generator weights use EDM-style EMA, but evaluation uses online generator weights.
  • E.1 IMPLEMENTATION DETAILS: Each iteration samples ImageNet classes uniformly and generates one-step images with num sampling steps=1.FD-SIM matches SigLIP, MAE, and InceptionV3 feature distributions at target resolutions 224, 224, and 299, with unit weight for each term.
  • E.1 IMPLEMENTATION DETAILS: FD reference statistics come from center-cropped ImageNet images, while training statistics initialize from 50,000 generated samples and update with β = 0.999 EMA.Generated features are gathered across GPUs before mean and covariance computation, and this statistic EMA is independent of generator-weight EMA.
  • E.1 IMPLEMENTATION DETAILS: The FD-Adv formulation uses stop-gradient notation through sg(·), alongside detached real-feature gradients during adversarial training.These gradient controls accompany the adversarial FD optimization procedure.
  • E.1 IMPLEMENTATION DETAILS: FD-Adv jointly updates a trainable representation-backbone copy and the generator, using AdamW critic learning rates of 2 × 10−6 for pMF and 1 × 10−6 for JiT.In SIM+Adv, the critic updates every two generator steps; the adversarial term starts at step 1,000, warms up linearly for 4,000 steps, and uses real-reference whitening with ϵ = 10−3.

E.2 EVALUATION DETAILS

The evaluation uses 50,000 generated images per checkpoint with model-specific sampling configurations and a six-encoder Fréchet-distance suite. Results include aggregate metrics averaging all six spaces or the DINOv2, CLIP, and ConvNeXt spaces.

  • Checkpoint evaluation: Each checkpoint is evaluated on 50,000 generated images using online generator weights and an evaluation batch size of 128 images per GPU.The released-checkpoint evaluation uses the online generator weights.
  • Sampling configurations: Sampling configurations are CFG 8.5 with interval [0.1, 0.7] for pMF-B, CFG 7.0 with interval [0.2, 0.7] for pMF-L, and CFG 7.0 with interval [0.2, 0.6] and noise scale 2.0 for pMF-H.pMF-B and pMF-L use noise scale 1.0.
  • Sampling configurations: JiT-H uses CFG 2.2 and interval [0.1, 1.0].No noise scale is specified for JiT-H in the supplied passage.
  • Representation evaluation: The default suite contains InceptionV3, ConvNeXt, DINOv2-L, MAE-L, SigLIP-SO400M, and CLIP-L, with images scaled to [0, 1] and resized per encoder.Fréchet distance is computed against corresponding ImageNet reference statistics in each representation space.
  • Aggregate metrics: FDr-6 is the arithmetic mean across all six representation spaces, whereas FD-r3 averages DINOv2, CLIP, and ConvNeXt and excludes Inception, SigLIP, and MAE.The reported normalized representation Fréchet distances use validation-set normalizers for each encoder.

F LIMITATIONS

The evaluation is limited to class-conditional ImageNet generation at 256×256 resolution. AdvFD’s effectiveness in higher-resolution, text-conditioned, and video generation remains unvalidated, and the method adds training-time computation.

  • Evaluation currently focuses on class-conditional ImageNet generation at 256×256 resolution.
  • AdvFD’s effectiveness on higher-resolution, text-conditioned, and video generation remains to be validated.
  • AdvFD introduces extra training-time computation for updating the adversarial representation and computing the whitening transformation.
Loading 2608.11205v1…