Source-linked AI summary
Making Reconstruction FID Predictive of Diffusion Generation FID
Tongda Xu, Mingwei He, Shady Abu-Hussein, Jose Miguel Hernandez-Lobato, Chunhang Zheng, Kai Zhao, Chao Zhou, Ya-Qin Zhang, Yan Wang
TL;DR
Reconstruction FID is poorly correlated with latent diffusion generation FID, motivating a more generation-aligned evaluation metric. The paper proposes iFID, which evaluates decoded nearest-neighbor latent interpolations, and shows that it strongly correlates with gFID while aligning with diffusion’s ridge-set structure.
Problem
VAE reconstruction FID is poorly correlated with latent diffusion generation FID, and better reconstruction can coincide with worse generation quality.
Method
iFID interpolates each latent with its nearest neighbor, decodes the result, and computes FID against the original dataset.
Results
iFID is reported to strongly correlate with diffusion gFID across diverse VAEs, with PCC and SRCC values close to 0.9.
Takeaways & Limitations
iFID provides a proxy for gFID by evaluating decoded latent interpolations aligned with the approximate ridge-set manifold around which diffusion samples concentrate.
Takeaways & Limitations
Although iFID correlates strongly with diffusion gFID, there is no straightforward way to directly minimize it.
Abstract
from arXiv · showhide
It is well known that the reconstruction FID (rFID) of a VAE is poorly correlated with the generation FID (gFID) of a latent diffusion model. We propose interpolated FID (iFID), a simple variant of rFID that exhibits a strong correlation with gFID. Specifically, for each dataset element, we retrieve its nearest neighbor in latent space, interpolate between their latent representations, decode the interpolated latent, and compute the FID between the decoded samples and the original dataset. We provide an intuitive explanation for why iFID correlates well with gFID, and why reconstruction metrics can be negatively correlated with gFID, by connecting iFID to recent results on diffusion generalization and hallucination. Theoretically, we show that iFID evaluates decoded interpolations aligned with the ridge set around which diffusion samples concentrate, thereby measuring a quantity closely related to diffusion sample quality. Empirically, iFID is the first metric shown to strongly correlate with diffusion gFID across diverse VAEs, achieving Pearson and Spearman correlations of approximately $0.85$. The project page is available at https://tongdaxu.github.io/pages/ifid.html.
1 Introduction
The paper introduces iFID to address the reconstruction–generation dilemma, where VAE reconstruction quality does not reliably predict latent diffusion generation quality. It motivates iFID through nearest-neighbor latent interpolation and connects its correlation with gFID to diffusion generalization and ridge-set structure.
- VAEs with excellent rFID can yield poor gFID, while VAEs with worse rFID can achieve better generation performance.
- iFID evaluates decoded interpolations between each data point and its nearest neighbor in latent space, then computes FID against the original dataset.
- The paper explains iFID’s correlation with generation quality by relating latent interpolation to diffusion generalization, hallucination, and the ridge-set manifold.
- iFID is reported as the first metric shown to strongly correlate with diffusion gFID across diverse VAEs, with high Pearson and Spearman correlations.
2 Preliminaries: Latent Diffusion Models
Latent diffusion models use a VAE to represent images in latent space and a diffusion model to generate latent samples. Sampling relies on a learned score estimator and reverse-time diffusion dynamics before decoding generated latents.
- An LDM combines a VAE that projects images into latent space with a diffusion model that generates samples in that space.
- The VAE decoder maps latent representations back into reconstructed or generated images.
- The diffusion model is parameterized by a score estimator that approximates the latent data-distribution score and is trained with denoising score matching.
3 Making Reconstruction FID Predictive of Generation FID
The paper introduces iFID, which evaluates decoded interpolations between each latent and its nearest neighbor instead of direct reconstructions. It explains iFID’s relationship to diffusion quality through interpolation behavior, hallucination, and ridge-set geometry.
- 3 Making Reconstruction FID Predictive of Generation FID: iFID linearly interpolates each latent with its nearest neighbor, decodes the result, and computes FID against the original dataset.This replaces direct evaluation of the original latents used by rFID.
- 3 Making Reconstruction FID Predictive of Generation FID: Diffusion samples can behave as local interpolations or compositions of training examples, motivating iFID as a measure of interpolated-data validity.The paper connects this interpretation to prior results on diffusion generalization and hallucination.
- 3 Making Reconstruction FID Predictive of Generation FID: Isolated latent modes yield off-manifold interpolations and hallucinations, whereas connected or overlapping modes keep interpolations on the data manifold.The toy Gaussian-mixture examples illustrate how latent-space geometry affects both iFID and diffusion samples.
- 3.3 Why Interpolated FID Predicts Sample Quality: Ridge-Set Theory: Diffusion samples concentrate around a ridge set, and in the small-noise regime this set is approximately aligned with the line joining two nearest training latents.This provides the theoretical connection between iFID’s local interpolation directions and diffusion sample concentration.
- 3.3 Why Interpolated FID Predicts Sample Quality: Ridge-Set Theory: The ridge-set eigenvalue is maximized at midpoint interpolation, matching the paper’s observation that midpoint iFID correlates most strongly with gFID.The midpoint is zt = (z(i∗) + z(j∗))/2.
4 Experimental Results
Experiments evaluate iFID across diverse VAEs and corresponding latent diffusion models on ImageNet. They show strong correlations with gFID, realistic decoded interpolations for diffusion-oriented VAEs, and robustness to several design choices.
- 4.1 Experimental Setup: The study evaluates 13 publicly available VAEs spanning diverse latent dimensions, architectures, and training objectives, with corresponding diffusion models trained in each latent space.Diffusion models are trained with SiT-B and SiT-XL, and gFID is reported with and without classifier-free guidance.
- 4.2 Main Results: Image-to-image reconstruction metrics, including PSNR, SSIM, and LPIPS, are strongly negatively correlated with gFID.This supports the reconstruction–generation dilemma: better reconstruction does not necessarily imply better diffusion generation.
- 4.2 Main Results: iFID achieves PCC and SRCC values close to 0.9, substantially outperforming diffusion loss, VF Loss, and GMM Loss as predictors of gFID.The comparison is based on metrics from SiT models trained with 13 pretrained VAEs.
- 4.2 Main Results: Reconstruction-oriented VAEs often pair each latent with a semantically unrelated nearest neighbor, producing invalid decoded interpolations, whereas diffusion-oriented VAEs produce realistic ones.The paper contrasts examples including SD-VAE and FLUX-VAE with VA-VAE and RAE.
- 4.2 Main Results: Spherical interpolation achieves the highest correlation with gFID, while linear and mask interpolation also achieve strong correlations around 0.8.The result indicates that iFID remains effective across multiple interpolation rules.
- 4.2 Main Results: Using fewer reference images or the top K = 10 nearest neighbors has minimal impact on iFID’s correlation with gFID.A 50k training subset yields approximately 0.85 correlation, while top-K selection leaves PCC at 0.89 without CFG and raises CFG PCC from 0.86 to 0.88.
5 Related Work
Prior work identifies a reconstruction–generation dilemma in latent diffusion VAEs and explores alternative training strategies, but effective diffusability metrics and explanations remain underexplored. This paper connects latent interpolation, diffusion generalization, and hallucination to motivate iFID as a proxy for generation quality.
- Standard reconstruction optimization does not necessarily improve diffusion sampling, motivating the reconstruction–generation dilemma.
- Prior approaches improve VAE suitability for diffusion through diffusion priors, signal-processing regularization, or contrastive-learning-based image encoders.
- Diffusion models can generate unseen samples by composing or interpolating training examples in latent spaces.
- Related work also shows that interpolation between nearby training-data modes can produce invalid or hallucinated samples.
- iFID evaluates decoded latent interpolations and is presented as the first metric shown to strongly correlate with diffusion gFID across diverse VAEs.
A Proof of Main Results
The proof shows that, under a local two-point approximation, diffusion samples concentrate around a ridge set determined by nearby latent samples. The ridge's dominant direction follows their connection line, with its largest eigenvalue attained at midpoint interpolation.
- A Proof of Main Results: Theorem 3 establishes that, for small t, the full Jacobian is close to the Jacobian determined by the two nearest-neighbor training samples.The approximation assumes bounded latent norms and a positive distance gap to other samples.
- A Proof of Main Results: The two-point Jacobian is constructed from the nearest samples, their latent difference, covariance terms, and the associated mean shift.The proof separately analyzes pair and tail errors before combining their bounds.
- A Proof of Main Results: Theorem 4 states that diffusion samples concentrate around the ridge set determined by the Jacobian.Together with the midpoint result, this links the proof's local geometry to the interpolation used by iFID.
- A Proof of Main Results: The largest eigenvalue is attained at the midpoint between the two nearest latent samples, where γ = 0.5.This identifies midpoint interpolation as the location of maximal nontrivial ridge curvature in the local analysis.
- A Proof of Main Results: The top eigenvector lies along the connection line between the two nearest latent samples.The remaining eigenvalues are discarded when forming the one-dimensional ridge.
- A Proof of Main Results: The ridge set becomes the line passing through the two nearest latent samples when t is small.The proof uses the fact that the empirical mean passes through the two nearest samples in this regime.
B Additional Experimental Results
The additional experiments evaluate diffusion models built from a diverse collection of pretrained VAEs and standardize training and sampling choices across latent representations. The setup includes multiple latent architectures and controlled sampling procedures.
- VAE configurations: The evaluation includes VAEs spanning 1D and 2D latent representations, including SD-VAE, DE-TOK, DM-VAE, REPAE-VAE, RAE, and other variants.The listed models use different latent shapes and training objectives, including reconstruction, masking, prior matching, and feature alignment.
- VAE configurations: The benchmark also includes FLUX-VAE, QW-VAE, SD3-VAE, EQ-VAE, VA-VAE, SOFT-VQ, and MAE-TOK.These models cover channel-expanded, equivariant, foundation-model-aligned, and token-based VAE designs.
- Training and sampling: All SiT diffusion models are trained on ImageNet for 40 epochs using latent-dependent patch sizes and positional embeddings.A time shift is applied to reduce the effect of differing total latent dimensions.
- Training and sampling: Samples are generated with a 250-step Euler SDE solver, and classifier-free guidance results use the best scale searched over [1.0, 6.0].The guidance scale is evaluated at increments of 0.25.
B.2 Metrics
The metrics section contrasts reconstruction, equivariance, foundation-model alignment, and latent-structure measures with diffusion loss. It emphasizes that reconstruction and diffusion-loss metrics can be weak indicators of generated sample quality, motivating alternatives such as GMM Loss and iFID.
- Reconstruction metrics: PSNR, SSIM, LPIPS, and rFID are common reconstruction metrics, but reconstruction metrics are strongly negatively correlated or weakly correlated with diffusion sample quality.rFID measures divergence between source and target image distributions, yet its correlation with gFID is weak.
- Diffusion loss: Diffusion loss primarily measures memorization capability, whereas sample quality depends on generalization ability.Although diffusion loss bounds latent-distribution divergence when weighted correctly, its correlation with sample quality is empirically weak.
- Alternative metrics: EQ Loss measures whether spatial transformations applied in latent space are preserved after decoding.It averages reconstruction distortion over scaling and 90°-multiple rotations; lower values indicate better equivariance.
- Alternative metrics: VF Loss aligns VAE latent features with frozen vision foundation-model features using a learned linear projection and two feature-similarity terms.The terms compare marginal cosine similarity and pairwise distance-matrix similarity.
- Latent-structure metrics: GMM Loss measures latent structural complexity by fitting a fixed-component Gaussian mixture and evaluating its negative log-likelihood.Lower GMM Loss indicates that the latent distribution is better approximated by the chosen GMM and has a simpler latent structure.
- Latent-structure metrics: Figure 6 reports a strong relationship between GMM Loss and diffusion loss, while diffusion loss need not correlate highly with sample quality across differing latent spaces.This supports the theoretical connection between GMM Loss and diffusion loss without establishing equivalent predictive power for sample quality.
B.3 The Relationship Between Metrics and gFID
Additional visualizations compare source, reconstructed, nearest-neighbor, and interpolated images across VAEs. Reconstruction-optimized VAEs produce semantically different neighbors and unrealistic interpolations, whereas diffusion-optimized VAEs produce more similar neighbors and more realistic interpolations.
- Visual comparisons: Figure 9 visualizes source images, reconstructions, nearest-neighbor images, and interpolated images for different VAEs.The comparison examines how latent neighborhoods and interpolations appear after decoding.
- Visual comparisons: For reconstruction-optimized VAEs, nearest neighbors are semantically different from the source and interpolations are not realistic.This contrasts the latent geometry associated with optimizing reconstruction quality.
- Visual comparisons: For diffusion-optimized VAEs, nearest neighbors are semantically similar to the source and interpolations are more realistic.These visual properties align with the interpolation-based evaluation used by iFID.
C.1 Explanation on Diffusibility of Previous VAEs
The paper attributes differences in diffusion performance across VAEs primarily to latent-space interpolability rather than reconstruction quality. Reconstruction-focused VAEs produce separable latent spaces, while signal-processing regularization and vision-foundation-model features promote smoother or semantically continuous representations.
- Reconstruction-only VAEs exhibit the worst generative performance because reconstruction loss produces a more separable latent space.
- Signal-processing regularization improves diffusion performance by suppressing high-frequency latent components and enhancing latent smoothness.
- The results identify latent-space interpolability as the primary determinant of diffusion-model performance.
- Vision foundation model latents are semantically continuous and interpolable, which explains their improved generation quality.
- The paper recommends avoiding encoder training with reconstruction loss because it encourages separable and isolated latent spaces.
C.4 Impact Statement
The impact statement limits the approach to evaluating diffusion-model latent spaces and notes a possible misuse: improving VAEs could support more effective disinformation. The figure context contrasts realistic versus invalid decoded interpolations across VAE types.
- The approach evaluates the latent space used for diffusion models rather than functioning as a generative model itself.
- Improving VAEs with the proposed metric could potentially enable more effective disinformation.
- Diffusion-oriented VAEs produce realistic interpolated images, whereas reconstruction-oriented VAEs can produce invalid interpolations.
NeurIPS Paper Checklist
The checklist records the paper’s claims, discussion of limitations, theoretical-assumption review, reproducibility materials, experimental reporting considerations, and ethics assessment. It indicates that code is provided in supplementary material and that the research conforms to the stated ethics code.
- The checklist directs reviewers to assess whether abstract and introduction claims match the paper’s theoretical and experimental scope.
- The paper’s limitations discussion is referenced through its Discussion section, with checklist guidance emphasizing assumptions and claim scope.
- The checklist requires theoretical results to state assumptions clearly and provide complete, cross-referenced proofs.
- Code is provided in supplementary material, supporting reproducibility of the experimental work.
- The checklist asks authors to report suitable statistical-significance information and sufficient compute-resource details for experiments.
- The research is stated to conform to the NeurIPS Code of Ethics, while potential negative impacts include misuse for disinformation.