Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples

Motivation

• Likelihood-based evaluation can be unstable, poorly correlated with sample fidelity, or unavailable for models without explicit densities.These limitations affect held-out likelihood evaluation and can obscure mode-dropping, mode-collapse, and overfitting.Source: §1 · pp. 1, 3
• Standard FID can assign a perfect score to a copycat model and remain vulnerable to overfitting even when evaluated against test data.The copycat reproduces training samples, yielding FID 0 under the usual protocol and better-than-SOTA test FID under an alternative protocol.Source: §1 · p. 2
• Evaluating novelty alongside fidelity and diversity targets generalization beyond the training set and can reveal privacy and copyright risks.The proposed trichotomy treats desirable samples as diverse, perceptually similar to the data distribution, and different from training examples.Source: §1 · p. 2

Contribution

• FLD is a sample-based metric designed to evaluate fidelity, diversity, novelty, overfitting, and memorization with scalability comparable to FID and IS.It applies across VAEs, normalizing flows, GANs, and diffusion models, and can be extended to conditional and multimodal settings.Source: §1 · p. 2

Method

• FLD maps samples into a perceptual feature space, fits an isotropic Gaussian mixture whose variances expose memorized samples, and evaluates held-out test-set likelihood.Memorized samples receive vanishingly small variances, worsening the density estimate for held-out data.Source: §1 · p. 3

Background

• Generative-model objectives require generated samples to be novel and high quality while remaining close to the data distribution.Closeness may be expressed through a divergence such as KL or JSD or a distance such as Wasserstein.Source: §2 · p. 3

Likelihood Evaluation

• Negative log-likelihood is theoretically appealing but often unavailable or only bounded, and may not track sample quality or common failure modes.The cited limitations include high-dimensional pitfalls, mode-dropping, mode-collapse, and overfitting.Source: §2 · p. 3

Sample-based Metrics

• Sample-based metrics avoid exact density computation and provide a unified evaluation setting for all deep generative models.They commonly extract features with Inception and compare generated and real distributions using metrics such as IS and FID.Source: §2 · p. 3
• IS and FID are popular because they correlate with perceptual quality and diversity, while KID and precision/recall add unbiased estimation or quality-coverage separation.These metrics are based on feature extraction followed by distributional similarity or distance calculations.Source: §2 · p. 3

Overfitting Evaluation

• Overfitting-detection methods span likelihood-based, discriminator-based, membership-attack, non-parametric, and parametric approaches.Examples include memorization scores, nearest-real-sample tests, data-copying tests, neural-network divergences, and latent recovery.Source: §2 · p. 3

3 Feature Likelihood Divergence

• FLD estimates perceptual density with a sample-based Gaussian mixture, assigning poor likelihood to high-quality failures caused by overfitting.Inputs are first mapped to a perceptually meaningful feature space, then isotropic Gaussians are centered on generated samples and averaged into a density estimate.Source: §3 · p. 4, §3.1 · p. 4

3.1 Overfitting Mixtures of Gaussians

• Memorized training samples drive their associated bandwidths toward zero, producing Dirac-like mixture components that distort the learned density.The bandwidth result follows from the distance between generated and training samples and is illustrated on the Two-Moons dataset with approximate training-set copies.Source: §3.1 · p. 5, Figure 4 · p. 5
• Test-set negative log-likelihood converts density distortion into a generalization signal that FID and IS lose, with higher FLD indicating problems in fidelity, diversity, or novelty.Poor fidelity, insufficient manifold coverage, and training-set overfitting each increase test-set NLL; the score is related to a forward KL divergence up to a dataset-dependent constant.Source: §3.1 · pp. 5–6
• FLD scales linearly with the number of training samples, matching the computational scaling of popular metrics such as FID.Feature extraction and sample generation dominate metric computation, and feature mapping can be performed once before metric evaluation.Source: §3.1 · p. 6

3.2 Detecting Memorization and Overfitting

• Sample-level likelihoods quantify memorization and can rank or filter poor-fidelity outputs after fitting the corresponding mixture model.The distribution of per-sample likelihood quantities identifies unusually likely approximate copies and supports sample-novelty assessment.Source: §3.2 · p. 6
• FLD defines overfitting when generated samples are more likely under the training-set density than the unseen test-set density.Samples close to training data both worsen test-set density estimation and increase likelihood assigned to the training set.Source: §3.2 · p. 6

4 Experiments

• FLD is evaluated on CIFAR10, FFHQ, and ImageNet across GAN and diffusion models to validate its relationship with fidelity, diversity, and novelty.Source: §4 · pp. 6–7, §4.1 · p. 7

4.1.1 Sample Fidelity

• Large image transformations substantially worsen both FLD and FID, whereas nearly imperceptible Posterize and Light Blur changes barely affect FLD but can severely distort FID rankings.Source: §4.1.1 · p. 7, Figure 5 · p. 7

4.1.2 Sample Diversity

• FLD consistently reflects reduced mode coverage and diversity across conditional models and increasingly duplicated generated samples.Source: §4.1.2 · p. 7, Figure 6 · p. 8

4.1.3 Sample Novelty

• FLD is the only evaluated metric that consistently detects overfitting across all transformations, while CT misses the largest transformations and AuthPct detects only a subset.Source: §4.1.3 · p. 8, Figure 7 · p. 8

4.1.4 Interaction effects

• FLD weights fidelity more than diversity and diversity more than novelty, except that copying all samples produces very high FLD.Source: §4.1.4 · p. 8, Figure 8 · p. 9

4.2 Comparison of Evaluation of State-of-the-Art Models

• Across CIFAR10, FFHQ, and ImageNet, FLD generally ranks models similarly to FID while revealing benign CIFAR overfitting through a train–test FLD gap.Source: §4.2 · p. 9, Table 1 · p. 9

4.3 Applications of FLD

• Nearest-neighbor rankings show non-negligible near copies in all evaluated models, while Qj rankings distinguish implausible older-model samples from generally stronger top-half samples of recent models.Source: §4.3 · pp. 9–10, Figure 9 · p. 10, Figure 10 · p. 10

• FLD is a broadly applicable metric that evaluates generation quality, diversity, and generalization while providing explainable insights into overfitting and memorization.Source: §5 · p. 10
• Experiments on synthetic and real-world datasets show that FLD can diagnose memorization and overfitting in photo-realistic generative models.Source: §5 · p. 10
• Future work should extend FLD to text, audio, time series, and conditional generative models.Source: §5 · p. 10

A.1 Number of samples

• FLD remains reliable with fewer reference samples, whereas FID can remain biased at 10k and improve noticeably with larger reference sets.KID shows similar consistency for smaller test sets.Source: §A.1 · p. 15, Figure 11 · p. 15
• Using 10k generated samples is sufficient for robust FLD evaluation, unlike the 50k commonly used for FID.This reduces the burden of evaluating diffusion models whose sampling can take hours for 50k samples.Source: §A.1 · p. 15

A.2 Computational Complexity of FLD

• FLD computation scales linearly with the number of training samples but remains roughly within the same order of magnitude as other metrics and insignificant relative to sample generation.The comparison measures metric computation after samples have been generated and mapped into feature space.Source: §A.2 · p. 15, Figure 12 · p. 15

A.3 Synthetic Experiments

• The synthetic setup uses Two Moons data split into training and test sets before fitting KDE models with varying bandwidths.The first 2,000 of 3,000 points form the training set and the final 1,000 form the test set.Source: §A.3 · pp. 15–16
• As KDE-generated samples approach the data distribution, FLD initially decreases, but it increases when samples become too close to training points.The experiment uses 2,000 training points, 1,000 test points, bandwidths from 10^-4 to 10, and 1,000 generated samples.Source: §A.3 · pp. 15–16
• FLD distinguishes underfit, well-matched, and overfit KDE samples, while overfit samples yield low FID but high FLD.The KDE visualization links low fidelity to high FID/FLD, appropriate matching to low FID/FLD, and near-copying to low FID but high FLD.Source: Figure 13 · p. 16

B Additional Sample Evaluations

• ImageNet samples are ranked according to Oj, with each generated sample paired with its nearest training sample in DINOv2 feature space.The figure presents generated samples on the left and nearest training samples on the right.Source: Figure 14 · p. 17
• FFHQ samples are ranked according to Qj in an additional visual evaluation.The supplied figure caption identifies the ranking quantity but does not state an outcome comparison.Source: Figure 15 · p. 17
• ImageNet samples are also ranked according to Qj in a separate additional evaluation.The supplied figure caption identifies the ranking quantity but does not state an outcome comparison.Source: Figure 16 · p. 18

C Inception Results

• The experiments were updated to use DINOv2 features, with additional results retained using Inception-V3 features.This passage describes the feature-space change but does not report a performance comparison.Source: C Inception Results · p. 19
• The Inception-V3 experiments compare FLD and FID under nearly imperceptible and large transformations of SOTA PFGM++ samples.The figure reports corresponding metric values for various models as references but the supplied passage does not state which metric changes more.Source: Figure 17 · p. 19
• A separate comparison evaluates overfitting metrics when generated data combines transformed generated samples with transformed copies of the training set.The supplied caption defines the comparison but does not report a specific winner.Source: Figure 18 · p. 19
• Table 2 summarizes FLD, FID, Gen Gap FLD, CT, AuthPct, precision, and recall for various models using Inception-V3 features.It also reports that many ImageNet models obtain better NLL than the training set, hypothesized to result from Inception-V3 being trained on ImageNet and producing different activations for the training set.Source: Table 2 · p. 19, C Inception Results · p. 19

D.1 Algorithm for MoG

• The algorithm fits a mixture of Gaussians using generated features for the means and train-to-generated feature distances for likelihood computation.Source: §D.1 · p. 20

D.2 Optimization

• Variance optimization uses batch gradient descent on the negative log-likelihood, with 10k parameters even under diagonal covariance.Source: §D.2 · p. 20
• The optimization used 10k generated samples, batch size 10k, 50 epochs, and learning rate 0.5.Source: §D.2 · p. 20
• A per-sample likelihood floor helps recover low variances when most generated samples are copies, preventing non-copied training samples from receiving exponentially small likelihoods.Source: §D.2 · p. 20
• Pairwise distance computation is O(n^2) and can be numerically unstable in high dimensions, so the implementation caches distances and uses log-variance and logsumexp transformations.Source: §D.2 · p. 20
• Variances usually converged in fewer than 10 steps, although global optimality was not guaranteed; exact or near-exact copies produced very low log variances.Source: §D.2 · p. 20

E.1 Transforms

• The setup applies torchvision transformations to assess both generated-image quality and overfitting.Source: §E.1 · p. 21
• Transformations include posterization, bicubic resizing, color distortion, elastic deformation, blur, center cropping, and JPEG compression with specified settings.Source: §E.1 · p. 21

E.2 Generated samples

• All experiments use samples released by Stein et al. [2023] from state-of-the-art models spanning multiple datasets.Source: §E.2 · p. 21

F Proof of Proposition 1

• Under bounded generated-to-training distances, the proof establishes that each optimized variance satisfies ˆσ_l = O(δ_l), where δ_l is the nearest distance to generated samples.Source: F Proof of Proposition 1 · p. 22
• The proof handles cases according to how often a variance appears in the likelihood sum and separately bounds it when the nearest distance exceeds a threshold.Source: F Proof of Proposition 1 · pp. 22–23

G Feature spaces

• FLD depends strongly on feature embeddings, and better embeddings could improve assessment; the paper evaluates images but suggests extension to other domains with suitable features.Source: G Feature spaces · p. 25

H Broader Impact

• Evaluation metrics cannot by themselves justify copying artistic styles without permission, and responsible assessment must also consider harmful or unethical content.Source: H Broader Impact · p. 25