PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Failures in Out-of-Distribution Detection with Deep Generative Models

Lily H. Zhang, Mark Goldstein, Rajesh Ranganath

arXiv:2107.06908v2cs.LG

TL;DR

Deep generative models can assign higher probabilities or densities to OOD images than to training-distribution images, while the relevant out-distributions for detection are not fixed. For a constructed pair of distributions, any rejection rule has power equal to its false-positive rate (random guessing); the paper also critiques the typical-set hypothesis’s support-overlap and typical-set assumptions. Without assumptions on relevant out-distributions, no method can guarantee performance beyond random chance, and the paper argues that model estimation error better explains high likelihoods on certain OOD images.

  • Problem

    Deep generative models can assign higher probabilities or densities to OOD images than to training-distribution images, while the relevant out-distributions for detection are not fixed.

    Source: §1 · p. 1

  • Method

    For a constructed pair of distributions, any rejection rule has power equal to its false-positive rate (random guessing); the paper also critiques the typical-set hypothesis’s support-overlap and typical-set assumptions.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    Without assumptions on relevant out-distributions, no method can guarantee performance beyond random chance, and the paper argues that model estimation error better explains high likelihoods on certain OOD images.

    Source: §1 · p. 2, §2.2 · p. 3

  • Takeaways & Limitations

    OOD detection analyses need to specify the relevant out-distributions, and the typical set is an arbitrary choice for defining OOD.

    Source: §7 · p. 9

    If the out-distribution support is small relative to the in-distribution support, a near-perfect model may still assign higher probability or density to samples from a disjoint-support out-distribution.

    Source: §7 · p. 9

Abstract

from arXiv · show

Deep generative models (DGMs) seem a natural fit for detecting out-of-distribution (OOD) inputs, but such models have been shown to assign higher probabilities or densities to OOD images than images from the training distribution. In this work, we explain why this behavior should be attributed to model misestimation. We first prove that no method can guarantee performance beyond random chance without assumptions on which out-distributions are relevant. We then interrogate the typical set hypothesis, the claim that relevant out-distributions can lie in high likelihood regions of the data distribution, and that OOD detection should be defined based on the data distribution's typical set. We highlight the consequences implied by assuming support overlap between in- and out-distributions, as well as the arbitrariness of the typical set for OOD detection. Our results suggest that estimation error is a more plausible explanation than the misalignment between likelihood-based OOD detection and out-distributions of interest, and we illustrate how even minimal estimation error can lead to OOD detection failures, yielding implications for future work in deep generative modeling and OOD detection.

1. Introduction

The introduction frames likelihood-based OOD detection failures as a problem of model estimation, while questioning whether the typical set defines which out-distributions matter. It also argues that detection guarantees depend on specifying relevant out-distributions.

1. Introduction

  • Explicit-likelihood DGMs can assign higher likelihoods to unrelated images and perform worse than random chance on examples such as Fashion-MNIST versus MNIST and CIFAR-10 versus SVHN.The cited work reports this behavior for both dataset pairs.Source: §1 · p. 1
  • Without constraints on relevant out-distributions, no OOD detection method can guarantee performance beyond random chance.The result applies to methods whether or not they use deep generative models.Source: §1 · p. 1
  • The typical set hypothesis treats relevant out-distributions as potentially lying in high-likelihood regions and bases OOD detection on membership in the data distribution’s typical set.The introduction questions this hypothesis, including its support-overlap assumption and the arbitrariness of preferring the typical set.Source: §1 · pp. 1–2
  • The experiments illustrate indistinguishable out-distributions for a given method and a case where a partially trained DGM detects OOD better than the true distribution under support overlap.The latter example is presented as an instance, not a general result.Source: §1 · p. 2
  • The authors attribute high likelihoods on certain OOD images to model estimation error, noting that even models with good generation quality and heldout likelihood can fail.They argue that disjoint supports would require existing models to assign positive probability or density where the true distribution assigns zero.Source: §1 · p. 2

2. Defining OOD Detection

The section formalizes OOD detection as a single-sample test and shows that its guarantees depend on which alternatives are considered. It argues that no test statistic works uniformly across all out-distributions.

2. Defining OOD Detection

  • Without constraints on the out-distributions, OOD detection is impossible to guarantee; the authors conclude that formal analyses must account for the alternatives defining the task.The broad task is framed as identifying whether a test example comes from a different distribution than the training data.Source: §2 · p. 2

2.1. OOD Detection as Goodness-of-fit Testing

  • OOD tests make a decision for one sample at a time using a test statistic; for likelihood-based tests, φ = log pθ may flag points with values below a threshold.This differs from goodness-of-fit tests that decide from a collection of samples.Source: §2.1 · p. 2

2.2. OOD Detection as a Single-Sample Distributional Test is Impossible

  • Even with knowledge of the true in-distribution, there are alternative distributions for which a test’s power equals its false positive rate, equivalent to random guessing.The proof constructs alternatives that preserve the test statistic’s distribution while differing in the underlying samples.Source: §2.2 · p. 3
  • The impossibility result concerns single-sample testing and holds even for test statistics that are asymptotically consistent in power.Source: §2.2 · p. 3
  • For log p, alternatives that induce the same distribution of log probabilities as P cannot be distinguished by that statistic.In the discrete example, Q can redistribute probability among elements that share the same probability under P.Source: §2.2 · p. 3
  • No test statistic is useful for all alternatives, so comparisons between statistics require specifying and standardizing the relevant family of out-distributions.Even likelihood-ratio tests are optimal only when the alternative is correctly specified, and trade off power on other alternatives.Source: §2.2 · p. 3

3. The Implausibility of the Typical Set Hypothesis

The typical set hypothesis explains high-likelihood OOD samples by treating the data distribution’s typical set as the relevant boundary. The paper argues that its support-overlap assumption has troubling consequences and that the typical set is arbitrary for OOD detection.

3.1. The Typical Set Hypothesis

  • The typical set hypothesis treats high-likelihood, low-probability OOD regions as detectable by testing whether samples fall outside the data distribution’s typical set.It assumes that relevant out-distributions can lie within the data distribution’s support.Source: §3.1 · p. 4

3.2. No Method Can Guarantee Perfect Detection When Supports Overlap

  • When in- and out-distributions overlap in support, no method can guarantee perfect detection because some out-distribution samples also lie in the in-distribution support.The paper states that this applies even with exact knowledge of the in-distribution.Source: §3.2 · p. 4

3.3. A Wrong Model Can Perform Better Than a Perfect One When Supports Overlap

  • With overlapping supports, a model misestimated toward the likelihood ratio of a particular out-distribution can achieve higher OOD-detection AUC than the true model.The result assumes integrability; the paper says the same idea can apply to other test statistics and when the out-distribution does not cover the entire sample space.Source: §3.3 · pp. 4–5

3.4. OOD Detection based on the Typical Set is Arbitrary

  • Small size and high probability do not uniquely justify preferring the typical set: other similarly small, high-probability sets can contain different samples.The typical set’s distinguishing property is near-equiprobability of its elements, which the paper says OOD detection does not require.Source: §3.4 · p. 5
  • The paper concludes that estimation error is a more plausible explanation for observed high OOD likelihoods than a need to redefine OOD detection around the typical set.It suggests existing models may assign positive probability or density to natural images that should have zero under the data distribution.Source: §3.4 · p. 6

4. Experiments

The experiments illustrate theoretical limits on OOD detection and show that a partially trained model can outperform the true model for a particular overlapping-support pair. They also construct out-distributions that a likelihood statistic cannot distinguish from the in-distribution.

4.1. Any Test Statistic has Failure Modes When Possible Out-distributions are Unrestricted

  • The typicality statistic based on average log-probability is no better than chance whenever a log-probability test statistic is no better than chance.The paper explains that the average-entropy term is constant, shifting and scaling the log-likelihood scores.Source: §4.1 · p. 6
  • For a bivariate Gaussian in-distribution, the constructed out-distributions Q and R have the same log-density distribution as P, so the likelihood statistic cannot distinguish them.Q flips the sign of the second coordinate in two quadrants, while R maps samples to points preserving their distance from the origin.Source: Figure 1 · p. 6, §4.1 · p. 6

4.2. A Bad Model Can Beat a Perfect One When the Out- and In-Distributions Overlap in Support

  • A 50-epoch GLOW model improves OOD separation for CelebA versus the true CIFAR-10 GLOW model, despite having worse in-distribution BPD.The partially trained model has 3.67 BPD on in-distribution test samples versus 3.45 for the true model; its BPD increase is greater for out-distribution samples, improving separation.Source: §4.2 · p. 6

5. An Alternative Perspective

The section argues that high OOD likelihoods are better explained by model misestimation than by assuming relevant out-distributions lie in high-likelihood regions. It shows that even small errors can harm OOD detection, then discusses test statistics and model changes that may address known biases.

5. An Alternative Perspective

  • If dataset pairs such as CIFAR-10 and SVHN have disjoint supports, existing DGMs assign probability or density where the true distribution assigns none; the disjoint-support assumption is untestable.The authors favor this estimation-error explanation over the claim that such OOD images have high likelihood under the data distribution.Source: §5 · pp. 6–7

5.1. A Good Generator Can Still Exhibit OOD Detection Failures

  • The authors suggest that model misestimation may affect regions with negligible total probability mass, so good generation alone does not ensure good OOD detection.When the out-support is small relative to the in-support, a model can be near-perfect yet assign higher probability or density to samples from an out-distribution with disjoint support.Source: §5 · p. 7, §5.1 · pp. 7–8
  • A generator can produce samples from the true distribution with probability close to 1 yet assign higher probability to out-of-support samples when the out-support is small relative to the in-support.In the discrete example, transferring a small amount of mass can leave in-distribution probabilities close to those of the true model while making OOD probabilities higher.Source: §5.1 · pp. 7–8
  • The same small-mass failure can occur for continuous distributions with bounded support, and the authors suggest real image distributions may differ in support volume.They point to the possibility that textured-image distributions have larger support than smooth-image distributions, while noting that the volume difference is difficult to determine.Source: §5.1 · pp. 7–8
  • The paper distinguishes its account from earlier explanations of high-density, rarely generated points by describing the phenomenon specifically as model misestimation.It notes that even good generators may estimate small-volume, low-probability regions poorly, although those regions matter for OOD detection.Source: §5.1 · p. 7, §5 · p. 7

5.2. Improving OOD Detection with DGMs

  • Test statistics that account for distance from average training likelihood or for rare training likelihood values can correct known biases, including high likelihoods assigned outside the in-distribution support.The authors interpret such statistics as correcting estimation error rather than as evidence that detection should follow the typical set.Source: §5.2 · p. 8
  • Alternative test statistics are not guaranteed to outperform log p across out-distributions, while model architecture and training objective may also affect OOD detection.The authors discuss fully connected layers improving detection in a reported example, and describe MLE's finite-data preference for overdispersed solutions as a challenge to learning good supports.Source: §5.2 · p. 8

6. Related Work

Related work explores likelihood-ratio and typicality-inspired statistics, changes to generative models, density-based arguments, discriminative scores, and support-boundary methods. The cited approaches use different assumptions and target different aspects of OOD detection.

Likelihood Ratio Test Statistics

  • Likelihood-ratio methods use alternative distributions built from perturbed samples, image compressors, or broad image datasets; the optimal alternative depends on which out-distributions matter.The cited approaches include Ren et al. (2019), Serr`a et al. (2020), and Schirrmeister et al. (2020).Source: §6 · pp. 8–9

Test Statistics Motivated by Typicality

  • Typicality-inspired methods compare sample statistics with training statistics or use latent-space transformations, but their empirical success may reflect compensation for model misestimation.Examples include entropy deviation, joint density estimation or one-class SVMs over statistics, latent-space typicality, and whiteness tests.Source: §6 · pp. 8–9

Modifying the DGM; Investigating Density for OOD Detection

  • Some approaches modify generative models to improve OOD performance, sometimes at the cost of worse likelihoods; density can also be arbitrarily re-ranked by transformations of continuous variables.The passage on transformations distinguishes points outside the support, which remain outside it under such transformations.Source: §6 · p. 9

Test Statistics Based on Discriminative Models

  • Discriminative OOD scores can be trained using classifier properties or OOD loss terms, but their learned conditional must extrapolate for unfamiliar inputs.The cited account notes that even the true conditional is undefined where p(x) = 0.Source: §6 · p. 9

Directly Learning a Decision Boundary

  • One-class SVMs, support vector data description, and deep variants estimate a subset of input space and its complement rather than density or probability over that set.These methods directly learn a decision boundary.Source: §6 · p. 9

7. Discussion

The discussion attributes certain DGM OOD failures to estimation error and stresses that OOD detection depends on specifying relevant out-distributions. It also notes that small total misestimation may suffice for failure, while the bias could affect more than a negligible set.

7. Discussion

  • The authors argue that the model, rather than probability-based OOD detection itself, is at fault in the failures they analyze, and emphasize specifying relevant out-distributions.They also characterize the typical set as an arbitrary basis for OOD detection.Source: §7 · p. 9
  • The authors hypothesize that the bias reflects the combination of model architecture and maximum likelihood objective, and suggest correcting it could benefit generative modeling beyond OOD detection.They also identify further comparison of generative modeling and support detection, and study of epistemic uncertainty, as directions for future work.Source: §7 · p. 9
  • If out-support volume is small relative to in-support volume, a near-perfect model may still assign higher probability or density to samples from an out-distribution with disjoint support.The authors say the extent of misestimation could be small, but also note that the bias may affect more than a negligible probability set.Source: §7 · p. 9

A. Proposition 1

For any test based on φ_p, there are distinct alternative distributions that make its power equal its false-positive rate. The construction changes conditional distributions while preserving the distribution of the test statistic.

A. Proposition 1

  • A test based on φ_p can have power equal to its false-positive rate for a distinct alternative, so it performs no better than random guessing.Equal power and false-positive rates yield an ROC curve y = x with AUC 0.5.Source: A. Proposition 1 · pp. 11–12
  • The proof constructs q(x) ≠ p(x) while preserving q(φ_p(x)) = p(φ_p(x)).It keeps the marginal distribution of φ_p(x) fixed and chooses conditional distributions q(x|φ_p(x)) that differ from p(x|φ_p(x)).Source: A. Proposition 1 · p. 11
  • The construction requires p(x|φ_p(x)) to be non-degenerate on a set of test-statistic values with positive measure.There, a set A can have conditional probability strictly between zero and one; q shifts conditional density away from A while retaining the same support.Source: A. Proposition 1 · p. 11
  • Because q and p give φ_p the same distribution, every rejection rule based on φ_p has the same rejection probability under both distributions.Thus the test's power equals its false-positive rate for any specified false-positive rate.Source: A. Proposition 1 · p. 12

B. Rejection Rules Can Be Written in the Form φp(x) < k

Rejection rules using intervals of a test statistic can be recast as threshold rules of the form φ(x) < k. This form covers both one-sided and two-sided intervals.

B. Rejection Rules Can Be Written in the Form φp(x) < k

  • Any rejection rule using intervals of φ(x) can be recast as a threshold rule φ′(x) < k.The result applies to any rejection rule involving intervals.Source: B. Rejection Rules Can Be Written in… · p. 12
  • For one-sided intervals, reversing the sign if needed yields a threshold rule; for bounded intervals, the rule can use distance from the midpoint.A bounded interval centered at m is expressed using |φ(x) − m| < k.Source: B. Rejection Rules Can Be Written in… · p. 12
  • In this form, likelihood-based tests reject above a negative-log-likelihood threshold, while typicality tests reject above an entropy-distance threshold.The examples are the likelihood-based test and typicality test.Source: B. Rejection Rules Can Be Written in… · p. 12

C. Details for Experiment 5.2

The experiment compares a partially trained GLOW model with a pretrained GLOW model used as the data distribution. It evaluates OOD performance on model-generated samples and CelebA.

C. Details for Experiment 5.2

  • The data distribution is a pretrained CIFAR-10 GLOW model, sampled at temperature 1 to generate 40,000 training and 10,000 evaluation samples.The partially trained model is compared with this pretrained model.Source: C. Details for Experiment 5.2 · p. 12
  • The partially trained GLOW model reaches 3.67 average bits per dimension on test samples, versus 3.45 for the true model.The experiment intentionally limits training to 50 epochs, including 10 warmup epochs, to make model misestimation clear; lower bits per dimension is better.Source: C. Details for Experiment 5.2 · p. 12
  • OOD performance is evaluated on test samples from the model distribution and on the CelebA test set.The true model is larger, with 32 affine coupling layers per block versus 8 in the partially trained model.Source: C. Details for Experiment 5.2 · p. 13

D. Existing Model Architectures Can Yield Good OOD Detectors

The authors train PIXELCNN++ with an objective that raises FashionMNIST likelihood while lowering MNIST likelihood. The negative-training objective achieves near-perfect OOD detection while maintaining comparable held-out log likelihood to maximum-likelihood training.

D. Existing Model Architectures Can Yield Good OOD Detectors

  • The negative-training objective maximizes likelihood on FashionMNIST images while minimizing likelihood on MNIST images.The objective is similar to Kirichenko et al. (2020), who show that flows can distinguish problematic OOD dataset pairs when optimized directly to do so.Source: D. Existing Model Architectures Can… · p. 13
  • The study compares five random seeds trained with maximum likelihood against five seeds trained with the negative-training objective.The authors use the architecture and training hyperparameters of Ren et al. (2019).Source: D. Existing Model Architectures Can… · p. 13
  • Negative-training PIXELCNN++ achieves near-perfect OOD detection while maintaining comparable held-out log likelihood to maximum-likelihood-trained models.Table 2 reports mean and standard deviation over five random seeds.Source: Table 2 · p. 13, D. Existing Model Architectures Can… · p. 13

Related papers

Loading 2107.06908v2…