Source-linked AI summary
Effectively Unbiased FID and Inception Score and where to find them
Min Jin Chong, David Forsyth
TL;DR
Finite-sample FID and IS are biased in ways that depend on the generator, making standard comparisons unreliable. The paper extrapolates these scores toward effectively unbiased infinite-sample estimates and uses Quasi-Monte Carlo methods to improve estimation, while also finding small GAN-training improvements. Its broader QMC treatment and extrapolation accuracy remain subject to the stated scope and tuning limitations.
Problem
FID and IS are biased differently across generators, so finite-sample scores cannot reliably compare models.
Method
The paper extrapolates FIDN and ISN to infinite-sample estimates and uses low-discrepancy Quasi-Monte Carlo integration for lower-variance estimates.
Results
The resulting FID∞ and IS∞ are effectively unbiased estimates and simple drop-in replacements, while QMC methods offer small improvements in GAN training.
Takeaways & Limitations
FID∞ and IS∞ provide a practical replacement for unreliable finite-sample generator comparisons.
Takeaways & Limitations
The paper leaves several high-dimensional integration methods and closed quasi-random evaluation sequences for future work, and notes that further hyperparameter tuning could improve FID100k estimates.
Abstract
from arXiv · showhide
This paper shows that two commonly used evaluation metrics for generative models, the Fréchet Inception Distance (FID) and the Inception Score (IS), are biased -- the expected value of the score computed for a finite sample set is not the true value of the score. Worse, the paper shows that the bias term depends on the particular model being evaluated, so model A may get a better score than model B simply because model A's bias term is smaller. This effect cannot be fixed by evaluating at a fixed number of samples. This means all comparisons using FID or IS as currently computed are unreliable. We then show how to extrapolate the score to obtain an effectively bias-free estimate of scores computed with an infinite number of samples, which we term $\overline{\textrm{FID}}_\infty$ and $\overline{\textrm{IS}}_\infty$. In turn, this effectively bias-free estimate requires good estimates of scores with a finite number of samples. We show that using Quasi-Monte Carlo integration notably improves estimates of FID and IS for finite sample sets. Our extrapolated scores are simple, drop-in replacements for the finite sample scores. Additionally, we show that using low discrepancy sequence in GAN training offers small improvements in the resulting generator.
1. Introduction
The paper argues that FID and IS are unreliable for comparing generators because their finite-sample bias varies by generator. It proposes extrapolated, effectively unbiased replacements and applies Quasi-Monte Carlo methods to improve estimation and GAN training.
- Motivation: FID and IS have generator-dependent bias, so fixing the sample count does not make finite-sample comparisons reliable.The paper notes that evaluation practices also vary between FID50k and FID10k.
- Contributions: Quasi-Monte Carlo integration improves estimates of FID∞ and IS∞ by reducing estimation variance.
- Contributions: Extrapolation of FIDN produces FID∞, an effectively unbiased estimate intended as a drop-in replacement for finite-sample FID.
- Contributions: The same approach produces IS∞, an effectively unbiased estimate of the Inception Score.
- Contributions: Using Quasi-Monte Carlo methods in GAN training offers small improvements in the resulting generator.
2. Background
The paper introduces FID and IS as generative-model evaluation metrics, then frames their estimation through Monte Carlo integration and motivates Quasi-Monte Carlo sampling. FID uses Inception features, while IS evaluates class-prediction structure and diversity.
- FID: FID compares generated and true data using means and covariances of features extracted by an ImageNet-pretrained Inception V3 model.The true and generated feature statistics are denoted (M_t, C_t) and (M_g, C_g), respectively.
- Inception Score: IS rewards confident predictions for individual images and diverse marginal class distributions.Its two stated properties are low conditional-label entropy and high marginal-label entropy.
- Monte Carlo estimation: The generated-feature mean and covariance in FID are random Monte Carlo estimates, whereas the true-data statistics are fixed when the reference set is fixed.Different generated samples therefore produce different estimated FID values.
- Monte Carlo estimation: Monte Carlo integration estimates an expectation from N IID samples, with error represented as a random perturbation around the true integral.The variance term depends strongly on the integrated function and is difficult to estimate directly.
- Quasi-Monte Carlo: Prior work reported that QMC with low-discrepancy sequences can converge up to 5 times faster than Monte Carlo with lower error rates.The sampling advantage is associated with avoiding the gaps and clusters typical of IID points.
- Quasi-Monte Carlo: Quasi-Monte Carlo methods use deterministic low-discrepancy sequences, such as Sobol and Halton, to reduce uneven spacing and potentially lower variance.The paper contrasts QMC convergence near O(N^-1) with MC convergence near O(N^-0.5).
3. Evaluating generative models with FID
Finite-sample FID and IS are biased, and their bias depends on the generator, making fixed-sample comparisons unreliable. Extrapolation and Quasi-Monte Carlo integration provide effectively unbiased estimates with lower variance, while Sobol-based GAN training is also explored.
- 3.1. FIDN is Biased: FIDN has a bias term CF/N + O(1/N^2), and CF depends on the generator being evaluated.Experiments confirm that FIDN is biased and that the bias varies across generators.
- 3.2. FID∞ as an Effectively Unbiased Estimate: Extrapolating FIDN in 1/N yields FID∞, which experiments suggest is effectively unbiased because higher-order terms are small.The extrapolation depends on obtaining sufficiently low-variance FID estimates.
- 3.2. FID∞ as an Effectively Unbiased Estimate: Sobol-based integration reduces bias and variance, producing smaller FIDN and larger ISN than random sampling for finite estimates.Among the evaluated methods, SobolInv generally gives better estimates of FID∞ and IS∞ than SobolBM.
- 3.3. ISN is Biased: The log Inception Score is negatively biased, with a generator-dependent bias; entropy convexity guarantees this negative direction.The first term is an unbiased Monte Carlo integral, while the entropy term introduces the bias.
- 3.3. ISN is Biased: As with FID, ISN bias depends on the generator, can be corrected by extrapolation in 1/N, and decreases with lower-variance integrators.The paper reports these effects for ISN in Figure 8 and Table 1.
4. Experiments
Experiments show that finite-sample FID and IS comparisons are unreliable because bias varies across generators, while QMC sampling and extrapolation produce more accurate, lower-variance effectively unbiased estimates.
- 4.2. QMC Integration: Sobol sequence integrators reliably reduce estimated FID bias and variance relative to normal sampling.The experiments compare Normal, SobolBM, and SobolInv sampling across models and find lower-bias estimates with Sobol sequences.
- 4.1. FIDN Bias: FID varies linearly with 1/N, but generator-specific slopes make rankings severely dependent on the chosen sample size.The slope corresponds to the K term contributing to FID bias, and even identical-architecture DCGANs can reverse rankings as N changes.
- 4.3. FID can be extrapolated: Extrapolating FID100k from smaller sample sizes gives accurate estimates, with SobolInv providing the best combination of accuracy and low variance.Regular intervals over N outperform intervals over 1/N, while at least 5k points are used for reliable FID calculations.
- 4.4. FID∞: FID∞ estimates have similar means and small variances across sampling methods, supporting effectively unbiased estimation from 50k samples.The approach extends the successful FID100k extrapolation using simple linear regression.
- 4.5. FID∞ for VAE: The FID extrapolation behavior also holds for VAEs, including linearity, improved QMC bias and variance, and successful extrapolation.Experiments use a vanilla VAE trained on 64 × 64 CelebA.
- 4.6. Estimating IS∞: IS follows the same 1/N trend as FID, and QMC-based extrapolation accurately estimates IS∞ with good accuracy and low variance.SobolInv with evaluation points at regular intervals over N performs best; accurate IS100k estimates are reported for BigGAN on ImageNet and CIFAR10.
- 4.7. Training with Sobol sequence: Training GANs with Sobol sequences generally lowers FID∞ and substantially reduces run-to-run variance, with negligible computational overhead.At 64 × 64, Sobol-trained GANs generally have lower FID∞; at 128 × 128, scores are comparable with normal sampling.
5. Related works
Prior work addressed related estimation and evaluation issues, but did not identify the generator-dependent bias that makes fixed-sample FID and IS comparisons unreliable.
- Earlier work established that no unbiased estimator for FID exists, while leaving open the possibility of arbitrarily small bias through approximation.The paper argues its FID_∞ and IS_∞ estimates display very small bias because 1/N terms dominate higher-order terms.
- Importance weighting improves finite-sample IS, FID, and KID estimates, but does not identify formal bias in FID_N or IS_N.The reported improvements are attributed to increased effective sample size from dataset augmentation.
- Quasi-Monte Carlo was suggested for VAEs and GANs to reduce gradient-estimator variance, but that work provided neither explanations nor results.
- Correlations between HYPE and FID_N cannot be relied upon because FID_N bias depends on the particular generator.The paper proposes investigating correlations between HYPE and FID_∞ instead.
6. Future work
Future work includes broader quasi-Monte Carlo methods, closed sequences for better evaluation error bounds, and studying correlations between FID_∞ and HYPE.
- The paper leaves sparse grids, higher-order scrambled digital nets, and randomized lattice rules for future investigation.
- Closed quasi-random sequences with N known beforehand could provide better error bounds on the integral.
- The authors plan to investigate whether FID_∞ correlates with HYPE.
7. Best practices
The paper recommends avoiding finite-sample FID_N and IS_N comparisons, instead using Sobol-based estimates, extrapolation, and repeated evaluations to assess reliability.
- Never compare generators using FID_N or IS_N because the comparisons are not reliable.
- Use SobolInv to compute both FID_N and IS_N.
- Extrapolate estimates computed at regular intervals over N to obtain FID_∞ and IS_∞ estimates.
- Repeat the evaluation multiple times to estimate the variance and assess the reliability of FID_∞ and IS_∞.
- Training with Sobol sequences produces better or comparable FID_∞ and lower variance across models, although the reported training results are moderate.The authors state that large-scale experiments are needed to validate the usefulness of Sobol-sequence training.
Appendix
The appendix provides pseudocode and implementation details for generating Sobol points, evaluating FID at multiple sample sizes, and extrapolating to FID_∞.
- IS_∞ evaluation: IS_∞ uses the same pseudocode as FID_∞, replacing the FID calculation with the IS calculation.
- Inputs: The procedure takes a generator, Inception network, image count, extrapolation-point count, and precomputed real-feature mean and covariance as inputs.
- FID_∞ evaluation: The evaluation procedure generates n Sobol samples, maps them through the generator and Inception network, and stores the resulting activations.
- FID_∞ evaluation: FID is evaluated on batch sizes linearly spaced from 5000 to n after shuffling the activations.
- FID_∞ evaluation: Linear regression of FID against 1/batchSize predicts the intercept at zero, returned as FID_∞.
- Sobol sampling: The Sobol sampler can generate standard-normal latent vectors through inverse-CDF or Box–Muller transformations.
- Sobol sampling: The sampler supports cached and reordered Sobol points, including sampling without replacement from cached points.
- Implementation: The implementation can fall back to torch.randn when no sampler is configured and can draw latent batches through the sampler interface.