PaperlayerView exact revision on arXiv

Source-linked AI summary

Input complexity and out-of-distribution detection with likelihood-based generative models

Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, Jordi Luque

arXiv:1909.11480v3cs.LGstat.ML

TL;DR

Likelihood-based generative models can produce misleading OOD scores because input complexity strongly influences their likelihoods. The paper uses complexity estimates to construct a parameter-free likelihood-ratio score, which performs comparably to or better than existing approaches across varied settings.

  • Problem

    Generative-model likelihoods can fail to distinguish training data from substantially different OOD inputs, including inputs with different semantics.

  • Method

    The paper estimates input complexity and uses it to derive an efficient, parameter-free OOD score interpretable as a likelihood-ratio for Bayesian model comparison.

  • Results

    The score improves OOD detection over likelihoods alone and is competitive with classifier- and generative-based approaches across the evaluated settings.

  • Takeaways & Limitations

    The proposed score offers an efficient alternative that is easy to use and deploy because it has no hyper-parameters beyond the generative model and compression algorithm.

Abstract

from arXiv · show

Likelihood-based generative models are a promising resource to detect out-of-distribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models' likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.

1 INTRODUCTION

OOD detection is important for reliable machine-learning systems, but generative-model likelihoods can misidentify substantially different inputs. The paper attributes this failure to input complexity and proposes a complexity-adjusted OOD score.

  • OOD detection supports safe and reliable operation in applications involving novel or substantially different inputs.
  • Likelihood-based generative models can fail to distinguish training data from unrelated OOD inputs, including inputs with different semantics.
  • When trained on CIFAR10, models assign SVHN higher likelihood than CIFAR10, while TrafficSign tends to receive lower likelihoods.
  • The paper links puzzling likelihood behavior to a strong bias toward input complexity and introduces a complexity-based likelihood-ratio OOD score.
  • The evaluation spans 12 data sets, two generative-model families, increasing model sizes, and three complexity-estimate variants.

2 COMPLEXITY BIAS IN LIKELIHOOD-BASED GENERATIVE MODELS

Generative-model log-likelihoods increase as images become simpler and are strongly negatively correlated with compression-based complexity estimates. Controlled pooling experiments and broad comparisons indicate that complexity substantially drives likelihood variation.

  • Log-likelihood is defined as ℓM(x) = log2 p(x|M), with negative log-likelihoods reported in bits per dimension.
  • Constant, Omniglot, MNIST, and FashionMNIST obtain higher log-likelihoods than more visually complex data sets under a Glow model trained on CIFAR10.
  • Progressive average-pooling of noise images produces progressively higher log-likelihoods as pooling reduces image complexity.
  • Lossless compression is used to compute an upper bound on otherwise noncomputable Kolmogorov complexity.
  • Pearson correlations between complexity estimates and likelihoods are below −0.75 for FashionMNIST-trained models and below −0.9 for CIFAR10-trained models.

3 TESTING OUT-OF-DISTRIBUTION INPUTS

The proposed OOD score subtracts an input-complexity estimate from negative log-likelihood, compensating for complexity-related variability. It can be used directly for ranking and interpreted as a likelihood-ratio statistic.

  • The OOD score S(x) subtracts complexity L(x) from negative log-likelihood −ℓM(x), with higher S indicating greater OOD likelihood.Both terms are expressed in bits per dimension.
  • S can be interpreted as a likelihood-ratio statistic comparing a trained, semantics-specific model M with a broadly applicable universal model M0.This interpretation follows Bayesian model comparison or the minimum description length principle.
  • The likelihood-ratio formulation follows from comparing model evidence, with uniform model priors reducing the comparison to the reported ratio.The marginal likelihood integrates out model parameters, although its integral is intractable.
  • The universal model M0 corrects abnormal high likelihoods for simple non-training inputs and favors itself over M for complex non-training inputs.Both situations produce large S values, whereas inputs better explained by M produce smaller values.
  • S can be used directly to rank unlabeled instances, including monitoring the top-k potentially problematic inputs.The authors retain S as a score rather than adopting a specific thresholding strategy, enabling AUROC comparisons.

4 RELATED WORKS

Related work explains abnormal likelihoods through background statistics, typicality, or minimum description length, while classification-based methods often assume labeled data. The proposed method differs from some alternatives by avoiding additional training and specialized background models.

  • Ren et al. attribute abnormal likelihoods to background statistics and train a background model on Bernoulli-generated random surrogates.Their approach adds L2 regularization and therefore has two hyper-parameters.
  • Unlike Ren et al.'s method, the proposed method requires neither additional training nor a separate background model for each training-data type.Ren et al.'s approach also requires both models to capture background information equally well.
  • Other work identifies typicality as the cause of generative-model OOD failures, using ensembles or Monte Carlo estimates of empirical entropy.The cited approaches differ in whether typicality is addressed explicitly.
  • Novelty-detection methods combining typicality and minimum description length have focused mainly on bit sequences and rely on strong parametric assumptions.They define atypical sequences using self-description rather than the optimum code for typical sequences.
  • Classification-based OOD methods generally do not apply to unlabeled or self-supervised data, although some extensions leverage generative models.

5 RESULTS

The proposed score S improves OOD detection over likelihoods alone across datasets and generally remains competitive with prior classifier- and generative-based approaches. S is also efficient and parameter-free, while larger models and better compressors tend to improve discrimination.

  • Comparison with likelihoods: AUROCs generally exceed 0.7 with S, often approaching 0.9 or 1, reversing failures on less complex datasets such as MNIST and SVHN.The only noted exception is TinyImageNet with PixelCNN++ and FLIF.
  • Sensitivity to models and data: Increasing model size tends to improve discrimination, while better compressors yield slightly improved AUROCs; performance is easier to obtain on FashionMNIST than CIFAR10.The authors attribute the dataset trend to the ease of learning and approximating the data density, but note that deeper analysis is needed.
  • Comparison with prior approaches: S is competitive with classifier- and generative-based approaches, achieves the best scores when trained on FashionMNIST, and matches or exceeds existing approaches when trained on CIFAR10.Among generative approaches, S is outperformed only twice, both times by WAIC, which uses ensembles of generative models.
  • Efficiency and deployment: Using one generative model and one fast compression library makes S efficient and parameter-free compared with approaches requiring ensembles or hyperparameter tuning.The paper identifies ensembles as a possible direction for improving estimates, while emphasizing the deployment simplicity of the single-model formulation.

6 CONCLUSION

The paper argues that input complexity strongly influences generative-model likelihoods and contributes to puzzling OOD-detection behavior. Complexity estimates compensate negative log-likelihoods to produce an efficient OOD score.

  • Input complexity strongly affects generative-model likelihoods and is proposed as the main culprit behind puzzling OOD-detection results.
  • An input-complexity estimate compensates standard negative log-likelihoods to produce an efficient and reliable OOD score.
  • The score is interpreted as a likelihood-ratio analogous to Bayesian model comparison and has no hyper-parameters beyond a generative model and compression algorithm.
  • The score performs comparably to or better than several state-of-the-art approaches across data sets, models, model sizes, and compression algorithms.

A DATA SETS

The experiments use public data sets together with two synthetic image sets designed to clarify the relationship between input complexity and likelihood. Data are split using predefined partitions when available or randomized partitions otherwise.

  • The experiments use well-known publicly available data sets and two synthetic sets named Noise and Constant.Noise samples uniformly random 3×32×32 tensors quantized to 8 bits, while Constant repeats 3×1×1 values across 32×32 spatial dimensions.
  • Noise and Constant are synthetic image sets developed to facilitate understanding of the likelihood-complexity problem.
  • Predefined train-test splits are preserved, with 10% of training data randomly reserved for validation; otherwise, data are split 80% train, 10% validation, and 10% test.

B MODELS AND TRAINING

The study evaluates two generative models with different architectures: PixelCNN++ as an autoregressive model and Glow as an invertible model. Both are trained with Adam using validation-based learning-rate reduction and model selection.

  • The experiments use PixelCNN++ as an autoregressive model and Glow as an invertible model capable of exact log-likelihood inference.
  • PixelCNN++ uses five residual blocks per stage, 80 filters, and 10 logistic mixture components.
  • Both models use Adam with initial learning rate 10^-4, batch size 50, scheduled reductions after stalled validation loss, and weights selected by best validation loss.
  • Training uses PyTorch 1.2.0 and a single NVIDIA GeForce GTX 1080Ti GPU, taking some hours under that setting.

C COMPRESSORS AND COMPLEXITY ESTIMATE

Input complexity is estimated by losslessly compressing each image and normalizing compressed length by input dimensionality. The study compares PNG, JPEG2000, and FLIF, and also considers selecting the best compressor per input.

  • The study compares three lossless image compressors: PNG, JPEG2000, and FLIF.FLIF is described as promising files up to 53% smaller than JPEG2000; header sizes are excluded from bits-per-dimension measurements.
  • The complexity estimate L(x) is the compressed bit-string length |C(x)| normalized by the input size or dimensionality d.
  • The improved estimate L′ uses the minimum across complexity estimates from different compression schemes, forcing the score to use the best compressor for each input.

D ADDITIONAL RESULTS

Additional experiments evaluate the proposed score S across likelihood measures, complexity estimates, model sizes, datasets, and compressors. The reported results include correlation analyses and AUROC comparisons for Glow and PixelCNN++ models.

  • Table 4 reports average log-likelihood ℓM for each dataset and sorts datasets from highest to lowest log-likelihood.
  • Table 5 reports Pearson’s correlation coefficients between normalized compressed length and model likelihood for different compressors.All reported correlations are statistically significant.
  • Table 6 compares AUROC values for negative log-likelihood −ℓM, complexity L, two-tail test T, and proposed score S.The comparison uses Glow and PixelCNN++ models trained on CIFAR10 with a PNG compressor; qualitatively similar results were obtained for FashionMNIST and other compressors.
  • Table 7 evaluates AUROC for S across increasing Glow model sizes using a PNG compressor.The model sizes are specified by blocks × flow steps, with qualitatively similar results for other compressors.
  • Table 8 compares AUROC for S across different compressors, Glow and PixelCNN++, and FashionMNIST and CIFAR10 training sets.
Loading 1909.11480v3…