PaperlayerView exact revision on arXiv

Source-linked AI summary

Representation Fréchet Loss for Visual Generation

Jiawei Yang, Zhengyang Geng, Xuan Ju, Yonglong Tian, Yue Wang

arXiv:2604.28190v1cs.CV

TL;DR

FD has traditionally been used to evaluate generative models because reliable estimation requires populations much larger than practical training batches. This paper introduces FD-loss by decoupling population and gradient batch sizes, consistently improving visual quality and reaching 0.72 FID on ImageNet 256 × 256 for a one-step generator.

  • Problem

    Directly optimizing FD requires large effective populations while keeping computation at the scale of affordable training batches.

  • Method

    FD-loss decouples population size from gradient batch size using queued features or exponential moving averages of feature moments.

  • Results

    FD-loss consistently improves visual quality across generator families, sizes, resolutions, and representation spaces, reaching 0.72 FID on ImageNet 256 × 256.

  • Takeaways & Limitations

    FD-loss links training and evaluation: it improves generators while exposing single-representation FID’s blind spots, motivating the multi-representation metric FDr^k.

  • Takeaways & Limitations

    FDr^k depends on the chosen representations and inherits Fréchet Distance’s Gaussian moment-matching assumption.

Abstract

from arXiv · show

We show that Fréchet Distance (FD), long considered impractical as a training objective, can in fact be effectively optimized in the representation space. Our idea is simple: decouple the population size for FD estimation (e.g., 50k) from the batch size for gradient computation (e.g., 1024). We term this approach FD-loss. Optimizing FD-loss reveals several surprising findings. First, post-training a base generator with FD-loss in different representation spaces consistently improves visual quality. Under the Inception feature space, a one-step generator achieves0.72 FID on ImageNet 256x256. Second, the same FD-loss repurposes multi-step generators into strong one-step generators without teacher distillation, adversarial training or per-sample targets. Third, FID can misrank visual quality: modern representations can yield better samples despite worse Inception FID. This motivates FDr$^k$, a multi-representation metric. We hope this work will encourage further exploration of distributional distances in diverse representation spaces as both training objectives and evaluation metrics for generative models.

1 Introduction

This work makes Fréchet Distance practical as a training loss by decoupling estimation population size from gradient batch size, improving generators while exposing limitations of single-representation FID. FD-loss achieves strong post-training results, repurposes multi-step generators into one-step models, and motivates multi-representation evaluation.

  • Motivation: Although FD is differentiable, direct optimization was considered impractical because reliable statistics require populations such as 50k and gradients through all samples would be prohibitive.Historically, FID [17] functioned as an evaluator rather than a training loss, especially on benchmarks such as ImageNet.
  • Method: FD-loss decouples the large population needed for reliable FD estimation from the smaller batch used for gradient computation.It is implemented with either an online feature queue or exponential moving averages, enabling backpropagation through only the current batch while estimating FD over a population such as 50k.
  • Results: 0.72 FID on ImageNet 256 × 256 is achieved by a one-step generator fine-tuned with FD-loss under Inception features.Across pixel-space and latent-space generator families, model sizes, and resolutions, FD-loss consistently improves visual quality using pre-computed real-data feature statistics.
  • Results: From around 300 FID to 0.72, FD-loss repurposes a pre-trained multi-step generator into a one-step generator without teacher distillation, adversarial training, or per-sample targets.This supports FD as a distribution-level objective rather than only an evaluation metric, and the approach is validated on class-conditioned models.
  • Results: 1.89 FDr6 is achieved after FD-loss post-training, while models optimized with modern representations can look better despite worse Inception FID.These findings motivate evaluating generators across multiple representation spaces rather than relying on a single FD.
  • Conclusion: FD-loss links training and evaluation by improving generators while revealing that any single FD representation can misjudge perceptual quality.The authors position FD as useful on both sides of the generative-modeling pipeline.

2 Related Work

Prior work established Fréchet Distance as a widely used but imperfect evaluation metric and explored distributional matching as a training objective. This work addresses the statistical-scale limitation of batchwise FD optimization and applies it to improving one-step generators.

  • Fréchet Distance as an evaluation metric: FID [17] dominates image-generation evaluation, but documented limitations [21] [19] have motivated precision–recall [21], CKA, and MMD [19].The paper makes FD directly optimizable to probe FID reliability and motivate FDr^k.
  • Fréchet Distance as a training loss: Prior training objectives include adversarial, MMD-based [25] [6], sliced-Wasserstein [7], feature-moment, and feature-space FD matching [34] [8].Existing FD optimization methods minimize estimates computed within a single batch, limiting their statistical scale.
  • Optimizing over broader sample windows: This work extends large-effective-sample optimization ideas from memory banks, feature queues [16], and EMA population statistics [18] by computing FD over a queue.The queue is intended to provide a broader sample window than a single batch.
  • One-step generators and post-training: For one-step generation, FD optimization in capable representation spaces complements straightening ODEs [26], consistency models, score distillation, identity-based methods [11] [30], and drifting models [6].The broader context is that modern high-quality generators often rely on multi-step denoising [9] [24].

3 Method

FD-loss makes Fréchet Distance practical for generator post-training by decoupling large-population statistics from small-batch gradients. The method supports queue- or EMA-based estimation, multi-representation objectives, and repurposes multi-step generators into one-step generators.

  • 3.1 Challenges and FD-loss: FD-loss decouples a large effective population for stable distribution statistics from the current batch used for gradient computation.Reliable estimation may require roughly 50k samples, while typical training batches contain 64–1024 samples; small batches are especially unstable for high-dimensional covariances.
  • 3.2 Estimators: The queue estimator aggregates features over N samples while back-propagating only through the current B-sample batch, with B ≪ N.A representative configuration uses N = 100,000 and B = 1,024; queued features are detached for statistics and gradient computation.
  • 3.2 Estimators: The EMA estimator replaces feature storage with exponentially smoothed first and second moments, providing a more scalable and recent-sample-weighted alternative.It computes covariance from the running moments and back-propagates only through the current batch; the method achieves 0.81 FID in the reported post-training setting.
  • 3.3 Multi-representation FD-loss: Multi-representation FD-loss normalizes each representation-specific FD by its stop-gradient magnitude plus a stability constant, then combines the unit-scale terms with equal weights.This prevents feature spaces whose raw FD values differ by orders of magnitude from dominating the objective.
  • 3.4 Post-training settings: FD-loss post-trains existing one-step generators and can repurpose multi-step generators into one-step generators by evaluating them once at the terminal timestep.The method is demonstrated for pixel- and latent-space one-step generators, while multi-step models use their terminal output as the clean-image prediction.
  • 3.5 Evaluation metric: FDr^K averages normalized FD ratios across K representation models, preserving per-model interpretability while reducing the blind spots of single-representation FID.Validation images score 1.0 by definition, and the metric is presented as a more robust automatic measure rather than a replacement for human evaluation.

4 Experiments

Experiments on ImageNet show that FD-loss scales across generator families, model sizes, resolutions, and representation spaces, while also converting multi-step generators into effective one-step models. The results show that large but not stale population estimates and multi-representation training are central to improving both FID and broader quality.

  • Representation model: Optimizing Inception gives FID 3.31→0.81, whereas ConvNeXt gives FID 3.31→1.64 but improves FDr6 13.70→8.46, showing that FID can misrank visual quality.Modern ViT representations worsen FID but improve FDr6 more substantially, and visual comparisons favor their object structure despite higher FID.
  • Multiple representation models: Combining representation models is generally more effective: SigLIP with Inception recovers FID to 0.89, while adding MAE further improves FDr6 with negligible FID trade-off.The resulting FD-SIM configuration is used as the default for subsequent experiments.
  • One-step repurposing: FD-loss repurposes a multi-step JiT-L/16 model into sensible 1-NFE generators, with FD-SIM achieving FDr6 3.29 and FID 0.85.The conversion requires no distillation, adversarial loss, or per-sample regression target, although repurposed models may exhibit artifacts.
  • Scalability: FID-Inception reaches 0.72–0.79 and FD-SIM reaches FDr6 1.81–5.56 across generator families, model sizes, and resolutions using the same hyperparameters.The results span pMF, iMF, and JiT on ImageNet 256×256, plus pMF on 512×512; per-model tuning could improve them further.
  • Text-to-image extension: FD-loss also extends to SD3.5 Medium, converting its 2.5B-parameter multi-step latent denoiser into a 1-NFE text-to-image generator that preserves recognizable prompt content.Despite a 56× NFE reduction, the model inherits the stylized look of the reference distribution, demonstrating extension beyond class-conditioned ImageNet generation.

5 Conclusion · Appendix for Representation Fréchet Loss for Visual Generation

The work argues that distributional distances can become practical training objectives when optimized at scale, shifting attention toward the representation space defining them. It aims to broaden distribution-level post-training beyond image generation, including other modalities and restricted-data settings.

  • 5 Conclusion: Distributional distances, including Fréchet Distance, have primarily served as evaluators because reliable estimation requires populations far larger than training batches.The separation between sample-level training losses and distributional evaluation is described as practical rather than principled, since Fréchet Distance is differentiable.
  • 5 Conclusion: The paper positions scalable distributional optimization as a way to reconnect generative-model training with generation’s fundamentally distributional nature.This conclusion follows from the stated contrast between generation as a distributional problem and training dominated by sample-level objectives.
  • 5 Conclusion: The work makes distribution-level post-training broadly applicable by enabling optimization of distributional distances at scale.This goal extends beyond the specific image-generation setting studied in the paper.
  • 5 Conclusion: The proposed direction targets applications in other modalities, settings with scarce or restricted real-data access, and paradigms beyond image generation.These are presented as intended areas for broader applicability rather than demonstrated results in the supplied passage.
  • 5 Conclusion: Once a distributional distance is optimizable at scale, the central design question becomes which representation space should define it.The conclusion frames representation choice as the key follow-up problem after optimization practicality is addressed.
  • 5 Conclusion: The work begins addressing representation choice by exploring several existing representations.The passage characterizes this exploration as an initial attempt rather than a completed resolution of the design question.

A Additional Design Attempts and Observations … B.2 Configurations

The appendix shows that narrow representation-based objectives can be over-optimized without improving perceptual quality, motivating diverse representations and human checks. It also specifies the frozen feature-extraction choices and ImageNet post-training configurations used in evaluation.

  • A Additional Design Attempts and Observations: These exploratory failures are presented as inverse gradients for avoiding overly narrow representations or objectives, rather than as negative results of FD-loss.The appendix frames proxy misalignment through Goodhart’s law and related reward-model over-optimization in RLHF [4].
  • A Additional Design Attempts and Observations: Optimizing Inception FD improves pMF-B/16 to 0.81 FID, but further optimization can exploit representation blind spots without matching generators with FID around 1.0–1.5 perceptually.The proxy improves the base generator while remaining incomplete as a measure of visual quality.
  • A Additional Design Attempts and Observations: FDr6 combines six feature spaces spanning supervised, self-supervised, reconstructive, and vision-language objectives, yet representation coverage remains an open-ended perceptual problem.The six representations reveal gaps that Inception misses but are unlikely to capture every image imperfection.
  • A Additional Design Attempts and Observations: 660 IS and 2.09 FID coincide with artifacts and FDr6 degradation to 50.66 after using a 100× larger learning rate, exposing metric gaming.Figure A.1 presents this stress test of deliberately over-optimizing Inception-based metrics.
  • A Additional Design Attempts and Observations: FDr^k reduces single-model bias but remains provisional, so human preference studies test whether gains over strong multi-step systems such as RAE reflect genuine visual superiority.The broader metric may still be hackable despite using multiple representations.
  • B.1 Representation Models for FDr6: FDr6 uses supervised, self-supervised, and vision-language models across CNN and ViT architectures.Table B.1 summarizes the representation-model families used for evaluation.
  • B Implementation and Evaluation Details: CNN features are extracted after final spatial pooling, ViT features from the final-layer CLS token, and all representation models remain frozen during training.These choices define the feature extraction procedure for FDr6.
  • B.2 Configurations: Table B.2 summarizes ImageNet class-conditional post-training configurations for the pMF, iMF, and JiT generator families.These configurations cover the three generator families evaluated in the appendix.

B.3 Training Details

Training initializes FD statistics from generated samples so the estimate is meaningful from the first step, while computing the covariance square-root term efficiently via eigendecomposition.

  • Initialization: The EMA variant generates 50k images before training to initialize its feature statistics, while the queue variant generates N images to fill its feature queue.Both procedures provide a warm start for FD estimation.
  • Configurations: ImageNet class-conditional post-training uses shared configurations across ablation and final runs unless noted, with base models used as released and only post-training changed.These settings are summarized in Table B.2.
  • Matrix square root: The implementation precomputes Σ_r^1/2 and uses torch.linalg.eigvalsh on a symmetric product to avoid explicitly computing a matrix square root at every training step.eigvalsh was adopted because it was significantly faster than eigvals while performing similarly in early exploration.

B.4 Text-to-Image Post-Training · B.5 Evaluation Protocol

The study post-trains Stable Diffusion 3.5 Medium for one-step 256×256 text-to-image generation using FD-loss in the SIM representation set. It compares two caption-image sources under matched training settings and evaluates official checkpoints from uniformly sampled images against fixed ImageNet reference statistics.

  • B.4 Text-to-Image Post-Training: SD3.5 Medium’s 2.5B-parameter MMDiT is post-trained with FD-loss for one-step 256×256 generation, using the unchanged SD3.5 VAE tokenizer and SIM representations.SIM combines SigLIP2, Inception, and MAE features.
  • B.4 Text-to-Image Post-Training: Reference statistics are precomputed from all real images in each set: 3M images for variant (i) and 60k for variant (ii).The configurations are summarized in Table B.3.
  • B.4 Text-to-Image Post-Training: Training runs for 15,000 steps with global batch size 1024, peak lr=10^-5, and 2,500 warmup steps under a cosine schedule.EMA feature statistics are warm-started with 50k base-model images.
  • B.4 Text-to-Image Post-Training: Each training step generates one image per caption using one denoising step without classifier-free guidance, with CFG=1.EMA statistics use β=0.999, and the matrix square root is computed with eigvalsh.
  • B.4 Text-to-Image Post-Training: The two variants differ only in data source: BLIP3o-Pretrain-Long-3M [3] uses 3M randomly sampled web pairs, whereas BLIP3o-GPT4o-60k [3] uses 60k curated GPT-4o-distilled images.The former is photographic, while the latter has a stylized, illustration-leaning aesthetic; all other hyperparameters are identical.
  • B.5 Evaluation Protocol: Evaluation samples 50,000 images from each official checkpoint using the official code, manually verifies correctness, and runs evaluation on the sampled images.The protocol primarily uses coding agents such as Claude Code and Codex to follow each codebase’s instructions.
  • B.5 Evaluation Protocol: For class-conditional models, evaluation samples uniformly across all 1,000 ImageNet classes, with 50 images per class.Reference statistics are computed once from the full ImageNet training set.

C Human Preference Study

The study uses anonymized pairwise comparisons of 3×3 image grids to measure human visual-fidelity preferences, with randomized presentation and hidden model identities. It aggregates 2,929 valid votes from 17 participants using a tie-adjusted preference score.

  • Human Preference Study: Each trial compares anonymized 3×3 grids, randomizes left–right order, hides model identities, and lets voters choose higher fidelity or tie.Post-trained versus Base comparisons use matched initial noise for direct visual comparison.
  • Human Preference Study: 2,929 valid votes from 17 participants are aggregated into preference scores, counting each tie as half a win.For FD-loss, the preference score is W + T/2, where W, T, and L are win, tie, and loss rates; Figure 6 reports the aggregates.
  • Interface: The interface presents uncurated side-by-side grids from the same ImageNet class, with Left, Tie, Right, Skip, Back, and image-zoom controls.Classes are sampled uniformly from all 1000 ImageNet classes, and the voting page is shown in Figure C.1.
  • Sampling: Each model contributes 50,000 samples, or 50 images per class, with replacement used when grids cannot be formed evenly.Generators use their best-FID inference settings, and trials are sampled uniformly across the specified model pairs.

D Limitations and Broader Impact

The study is limited to image generation, primarily on ImageNet, and FD-loss depends on representation spaces and reference statistics. Its quality improvements carry dual-use risks, while its diagnostic finding challenges reliance on FID alone.

  • Limitations: FD-loss is evaluated primarily for image generation on ImageNet, with text-conditioned generation as an additional demonstration.
  • Limitations: FD-loss behavior depends on the chosen representation spaces and reference statistics, so different domains may require different feature sets or weighting schemes.
  • Broader impact: Higher-quality image generators pose dual-use risks, including misuse for disinformation or deceptive content.
  • Broader impact: The work’s diagnostic finding that FID alone is insufficient may improve the field’s evaluation and understanding of generative models.

E Additional Qualitative Samples

Uncurated paired ImageNet 256×256 samples compare base generators with FD-loss post-trained versions using identical initial noise. The comparisons cover pMF-H/16 and conversion of JiT-H/16 from 200 NFE to 1 NFE.

  • pMF-H/16: pMF-H/16 [30] samples compare the one-step base model with its FD-loss (SIM) post-trained counterpart under identical initial noise.The paired samples are uncurated and span Figures E.1–E.3; SIM denotes SigLIP+Inception+MAE.
  • JiT-H/16: JiT-H/16 [23] samples compare the 200-NFE base model with FD-loss (SIM) post-training at 1 NFE using identical initial noise.The base configuration uses 50 steps × 2 (Heun) × 2 (CFG), and all samples are uncurated.

F Detailed Results · G Text-to-Image Prompts

The appendices detail FD-loss evaluations across representations, population sizes, generator repurposing, and system comparisons, while extending text-to-image qualitative comparisons and listing all prompts. They report both normalized and raw Fréchet metrics and examine one-step models against multi-step baselines.

  • F Detailed Results: Per-representation results report normalized FDr, raw FD, FDr6 across six spaces, and held-out FDr-CLIP evaluations.FDr6 averages FDr over Inception, ConvNeXt, DINOv2, MAE, SigLIP, and CLIP representations.
  • F Detailed Results: The detailed tables cover population-size and representation-model ablations, including FD-Inception and the SIM combination of SigLIP, Inception, and MAE.These tables provide per-representation FDr and raw FD breakdowns for the ablations.
  • F Detailed Results: Additional tables analyze JiT-L repurposing and system-level comparisons using per-representation FDr and raw FD values.The system-level tables identify FD-loss post-trained models in shaded rows, while the JiT-L tables use SIM alongside other representations.
  • F Detailed Results: Expanded metrics cover every FD-loss post-trained model, pairing each base generator with FD-Inception and FD-SIM variants that all use 1 NFE.The expanded table includes the base generator and its post-trained variants, with SIM defined as SigLIP+Inception+MAE.
  • G Text-to-Image Prompts: Qualitative text-to-image comparisons post-train SD3.5 Medium [9] on realistic-photograph and stylized reference distributions, suggesting recognizable prompt content after a 56× NFE reduction.The reference distributions are BLIP3o-Pretrain-Long-3M [3] and BLIP3o-GPT4o-60k [3], and the comparison includes the 56-NFE baseline and 1-NFE FD-loss models.
Loading 2604.28190v1…