Source-linked AI summary
A Note on the Inception Score
Shane Barratt, Rishi Sharma
TL;DR
The paper addresses the lack of reliable guidance for evaluating and comparing generative models by rigorously examining the widely used Inception Score. It analyzes the score’s mechanics and usage, finding shortcomings that undermine its usefulness for model comparison and motivating more systematic evaluation.
Problem
Meaningful evaluation of generative models remains difficult, yet the widely used Inception Score is treated as an evaluation metric for image-generating models.
Method
The paper rigorously investigates the Inception Score’s underlying premise, mechanics, and common applications, and proposes alterations to the metric and its usage.
Results
The Inception Score has shortcomings in both its mechanics and application, including sensitivity to network weights that leave classification accuracy nearly unchanged.
Takeaways & Limitations
Researchers should devote significant effort to systematic evaluation and validation because metrics for generative models remain imperfect.
Takeaways & Limitations
Inception Score evaluations on datasets other than ImageNet can produce misleading results, and high scores require an alternative check for training-data overfitting.
Abstract
from arXiv · showhide
Deep generative models are powerful tools that have produced impressive results in recent years. These advances have been for the most part empirically driven, making it essential that we use high quality evaluation metrics. In this paper, we provide new insights into the Inception Score, a recently proposed and widely used evaluation metric for generative models, and demonstrate that it fails to provide useful guidance when comparing models. We discuss both suboptimalities of the metric itself and issues with its application. Finally, we call for researchers to be more systematic and careful when evaluating and comparing generative models, as the advancement of the field depends upon it.
1. Introduction
Deep generative models have advanced across many applications, but evaluating them remains difficult. The paper investigates the widely used Inception Score and identifies shortcomings in both its premise and application.
- Deep generative models now support applications including realistic image generation, style transfer, super-resolution, language, music, medical data, and physical modeling.
- Meaningful evaluation metrics are necessary because generative models are difficult to evaluate rigorously.
- The Inception Score is a popular ad-hoc metric for evaluating image-generating models.
- The paper rigorously examines the Inception Score, identifies five major shortcomings, and proposes changes to its metric and usage.
2. Evaluating (Black-Box) Generative Models
Black-box generative models are evaluated without assuming access to their underlying density or model structure. Existing metrics use density estimation or pretrained networks, but their agreement and reliability remain unsettled.
- Explicit likelihood evaluation is often effective, but models such as GANs may define distributions only implicitly through mappings from noise to samples.
- Black-box evaluation assumes that samples can be drawn from the model while its underlying structure is otherwise unknown.
- Metrics designed around one model class cannot support global comparisons across different classes of generative models.
- Evaluation approaches include nonparametric density estimation and statistics computed from pretrained neural-network outputs or hidden layers.
- Most discussed metrics can be fooled by memorizing training data, and log-likelihood can also be misleading for some mixture distributions.
- A prior study found that common evaluation metrics do not correlate with one another, while another found no clear model superiority across datasets and metrics given sufficient hyperparameter search.
3. The Inception Score for Image Generation
The Inception Score evaluates generated images with an ImageNet-pretrained Inception v3 network. It rewards confident predictions and broad marginal class coverage, and is estimated through sample-based KL-divergence calculations.
- 3.1. Inception v3: Inception v3 classifies ImageNet images into 1000 classes by outputting a probability vector p(y|x).
- 3. The Inception Score for Image Generation: The Inception Score applies an ImageNet-pretrained Inception v3 network to generated images and summarizes its output statistics.
- 3.2. Inception Score: The score is intended to reward low-entropy conditional predictions, corresponding to clear objects and confident single-object classifications.
- 3.2. Inception Score: It also rewards high entropy in the marginal class distribution, corresponding to diverse images across ImageNet classes.
- 3.3. Digging Deeper into the Inception Score: The expected KL-divergence between conditional and marginal class distributions equals mutual information, linking the score to these entropy objectives.
- 3.4. Calculating the Inception Score: The score can be estimated from generated samples by constructing an empirical marginal class distribution and approximating the expected KL-divergence.
- 3.4. Calculating the Inception Score: The original procedure applies the estimator 10 times with N = 5,000 and reports the resulting mean and standard deviation.
4. Issues With the Inception Score
The paper identifies problems both in the Inception Score itself and in how researchers commonly apply it. These issues make scores unstable, dataset-dependent, and potentially misleading for model comparisons.
- 4. Issues With the Inception Score: The authors classify the broader problems as suboptimalities of the metric itself and problems with its popular usage.They also note implementation errors in commonly used scoring code.
- 4.1.1. Sensitivity to Weights: Small changes in Inception-network weights can alter scores despite nearly identical classification accuracy.Different implementations produce mean scores 3.5% higher on ImageNet validation images and 11.5% higher on CIFAR validation images.
- 4.1.2. Score Calculation and Exponentiation: The score depends unnecessarily on the number of dataset splits used during estimation.The usual procedure divides 50,000 generated images into chunks, and changing nsplits can change the final score.
- 4.1.2. Score Calculation and Exponentiation: Computing the marginal class distribution over the full dataset and removing the exponential makes the score batch-invariant and interpretable as mutual information.The resulting average is unchanged by the number of batches, with variance computed over the entire dataset.
- 4.2.1. Usage Beyond ImageNet Dataset: Applying the Inception Score outside ImageNet is misleading because ImageNet and target-dataset classes may not align.For CIFAR-10, the predicted marginal distribution can be far from the actual class distribution, and ImageNet-specific categories can distort conditional entropy.
- 4.2.2. Optimizing the Inception Score: Optimizing models for the score can favor adversarial examples, and memorization can produce high scores without demonstrating generalization.The paper shows a gently altered WGAN output achieving 900.15, while recommending alternative scores to check for overfitting.
5. Conclusion
The paper identifies multiple shortcomings in the Inception Score and argues that generative-model evaluation requires more rigorous and consistent methodologies.
- The paper highlights suboptimalities in the Inception Score and difficulties in designing effective generative-model evaluation metrics.
- Researchers should devote significant effort to evaluating and validating new generative-model techniques and methods.
Proof of Equation 2
The passage describes how the derivation expands the Inception Score into an expression involving conditional probability and mutual information.
- Equation 12 uses the definition of mutual information to complete the derivation.
Proof of Equation 3
The proof bounds the Inception Score using entropy properties and the 1000 ImageNet classes, establishing its possible range.
- 1000 is the maximum possible Inception Score because ImageNet has 1000 classes and the uniform distribution has entropy ln(1000).
- 1 is the minimum possible Inception Score because conditional entropy H(y|x) cannot exceed unconditional entropy H(y).
- Algorithm 1 is presented as the procedure associated with the generator analysis following the bound derivation.
Achieving High Inception Scores
The paper constructs generators that optimize ImageNet class predictions and achieve nearly optimal Inception Scores, despite differing initialization sources.
- The optimization targets each class by repeatedly updating images along the gradient of p(y = j|x).
- A uniform class distribution and deterministic class predictions satisfy the entropy conditions required for an Inception Score of 1000.
- The generator cycles through all 1000 classes, and the procedure is equivalent to applying FGSM-style updates to images.
- The method should achieve a near-perfect score when the product Nϵ is sufficiently large, with replay attacks corresponding to zero updates or step size.
- 986.10 is achieved using random image initialization with ϵ = .001 and N = 100.
- Figure 2 compares samples from random and WGAN initializations after gradient fine-tuning.
- 900.10 is achieved from CIFAR-10-trained WGAN initialization, producing more realistic-looking examples.