PaperlayerView exact revision on arXiv

Source-linked AI summary

Likelihood Ratios for Out-of-Distribution Detection

Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, Balaji Lakshminarayanan

arXiv:1906.02845v2stat.MLcs.LG

TL;DR

OOD detection is needed because neural networks can make confident errors on inputs outside their training distribution, including genomic sequences from new bacteria. The paper introduces a realistic genomics benchmark and a likelihood-ratio method that contrasts generative-model likelihood with a background model. It reports improved OOD detection on genomic and image datasets, including state-of-the-art performance on the genomics benchmark.

  • Problem

    Neural networks can confidently misclassify OOD inputs, while realistic benchmarks for practical OOD challenges such as bacterial genomics are needed.

  • Method

    The paper creates a realistic genomic OOD benchmark and proposes contrasting a deep generative model's likelihood with a background model through a likelihood ratio.

  • Results

    The method significantly improves OOD detection on image and genomic datasets and achieves state-of-the-art performance on the genomics benchmark.

  • Takeaways & Limitations

    Correcting for background components makes likelihood-based OOD detection more effective across genomic and image datasets.

  • Takeaways & Limitations

    The approach assumes background and semantic components can be separated so the in-distribution model and background model capture background information similarly.

Abstract

from arXiv · show

Discriminative neural networks offer little or no performance guarantees when deployed on data not generated by the same process as the training distribution. On such out-of-distribution (OOD) inputs, the prediction may not only be erroneous, but confidently so, limiting the safe deployment of classifiers in real-world applications. One such challenging application is bacteria identification based on genomic sequences, which holds the promise of early detection of diseases, but requires a model that can output low confidence predictions on OOD genomic sequences from new bacteria that were not present in the training data. We introduce a genomics dataset for OOD detection that allows other researchers to benchmark progress on this important problem. We investigate deep generative model based approaches for OOD detection and observe that the likelihood score is heavily affected by population level background statistics. We propose a likelihood ratio method for deep generative models which effectively corrects for these confounding background statistics. We benchmark the OOD detection performance of the proposed method against existing approaches on the genomics dataset and show that our method achieves state-of-the-art performance. We demonstrate the generality of the proposed method by showing that it significantly improves OOD detection when applied to deep generative models of images.

1 Introduction

OOD detection is important for safe, reliable neural-network predictions, especially in medical genomics. The paper introduces a realistic genomic benchmark and a likelihood-ratio approach that corrects background-statistics confounding and achieves strong benchmark performance.

  • OOD inputs can receive confident but incorrect classifications, creating serious risks when predictions inform medical decisions.
  • Existing OOD benchmarks largely use image datasets, motivating more realistic datasets that reflect practical distribution-shift challenges.
  • Bacterial identification from genomic sequences is a medically important application where models must handle bacteria absent from training data.
  • The paper creates a realistic genomics benchmark for OOD detection and evaluates existing methods on it.
  • Likelihood scores from deep generative models can be confounded by population-level background statistics.
  • The proposed likelihood-ratio method significantly outperforms raw likelihood for OOD detection on image datasets and achieves state-of-the-art performance on the genomics benchmark.

2 Background

Generative-model likelihood is an appealing OOD score, but it can fail because likelihood reflects background composition rather than in-distribution-specific patterns. In genomic sequences, GC-content is a concrete confounder that can make OOD inputs appear more likely.

  • Generative OOD methods model p(x) and use the likelihood of new inputs, but deep image models can assign higher likelihood to OOD inputs.
  • In genomic experiments, OOD and in-distribution log-likelihood distributions largely overlap, limiting separation.
  • 0.626 AUROC: Figure 1 reports that log-likelihood hardly separates in-distribution and OOD inputs.
  • GC-content, the percentage of G or C bases, heavily affects log-likelihood and can make high-GC OOD sequences score higher without containing in-distribution-specific patterns.

3 Likelihood Ratio for OOD detection

The method separates population-level background statistics from semantic patterns and uses a likelihood ratio to reduce background effects in OOD detection. A perturbed-input background model provides the contrastive reference for deep generative models.

  • High level idea: The approach models inputs as background components containing population statistics plus semantic components containing in-distribution-specific patterns.Examples include image backgrounds and objects, text stop words and semantic words, and genomic background sequences and motifs.
  • High level idea: The likelihood is decomposed into background and semantic terms, but standard likelihood-based detection does not distinguish between them.This can allow background statistics to dominate the score even when semantic content differs.
  • Likelihood ratio: The proposed likelihood ratio compares an in-distribution model with a background model that captures general background statistics.The models are denoted pθ(·) and pθ0(·), respectively.
  • Likelihood ratio: After taking the ratio, the background likelihood is cancelled under the stated assumptions, leaving a score that captures semantic significance relative to the background model.For autoregressive models, the likelihood is represented through sums of conditional token or pixel probabilities.
  • Background model: The method trains a background model on perturbed inputs so that corruption removes semantic structure while retaining population-level background statistics.Perturbations are applied by randomly substituting sequence characters; the perturbation rate µ is tunable and values in [0.1, 0.2] perform well empirically.

4 Experimental setup

The experiments evaluate likelihood-ratio OOD detection alongside classifier-based and generative baselines on image and genomic-sequence modalities. Evaluation uses held-out balanced in-distribution and OOD inputs with AUROC, AUPRC, and FPR80 metrics.

  • Experimental design: Experiments cover multiple modalities, using autoregressive models to compute log-likelihoods for images and genomic sequences.The same experimental design is applied across the image and genomics datasets.
  • Model training: The background model uses the same architecture as the in-distribution model but is trained on perturbed inputs, optionally with L2 weight regularization.This isolates the training-data perturbation and regularization choices from architectural differences.
  • Baselines: Perturbed-input baselines use the same perturbation procedure as the background-model training, while ODIN is adapted because it cannot directly handle discrete genomic sequences.The adaptation perturbs the last neural-network layer closest to the output.
  • Evaluation: Final evaluation samples equal numbers of in-distribution and OOD test inputs and uses the likelihood-ratio statistic as the detection score.A small score indicates a higher likelihood of being OOD.
  • Evaluation: Performance is measured with AUROC↑, AUPRC↑, and FPR80↓, which are standard metrics for OOD detection.The first two metrics are areas under ranking curves, while FPR80 measures false positive rate at 80% true positive rate.

5 Results

Experiments show that raw likelihood is confounded by background statistics, while likelihood ratios improve OOD detection across image and genomic settings. The paper introduces a temporally structured genomics benchmark and evaluates likelihood-ratio detection against baseline methods.

  • Image OOD detection: The likelihood-ratio method corrects background-statistics confounding and significantly outperforms raw likelihood for image OOD detection.On Fashion-MNIST versus MNIST, AUROC improves from 0.091 to 0.996; on CIFAR-10 versus SVHN, it improves from 0.095 to 0.931.
  • Image OOD detection: 0.85 Pearson correlation shows that the proportion of zero-valued background pixels confounds PixelCNN++ likelihood on Fashion-MNIST versus MNIST.The likelihood-ratio score reverses the misleading ordering, assigning higher scores to Fashion-MNIST than MNIST.
  • Image OOD detection: Likelihood is dominated by background pixels, whereas likelihood ratio focuses on semantic pixels in Fashion-MNIST and MNIST images.The per-pixel visualizations attribute high MNIST likelihood to background pixels and show likelihood-ratio emphasis on semantic content.
  • Genomic OOD benchmark: The genomics benchmark separates in-distribution and OOD bacterial classes by discovery year, using cutoff years 2011 and 2016.The dataset contains 10 in-distribution classes and 60 OOD classes; genomes are fragmented into 250-base-pair sequences, with 100,000 sequences sampled per class.
  • Genomic OOD detection: OOD detection AUROC correlates with genetic distance to in-distribution classes, with Pearson coefficients of 0.570 for likelihood ratio and 0.277 for the ensemble method.The results indicate that OOD classes farther from the in-distribution are generally easier to detect.

6 Discussion and Conclusion

The paper identifies background statistics as a source of confounding in autoregressive likelihoods, proposes likelihood-ratio correction, and introduces a realistic genomic OOD benchmark.

  • Likelihoods from autoregressive models can be confounded by background statistics, explaining failures of PixelCNN-based OOD detection.
  • The proposed likelihood-ratio method contrasts input likelihood with a background model to alleviate this confounding.
  • The method significantly improves OOD detection accuracy on both image and genomic datasets.
  • The released genomic sequence dataset provides a realistic benchmark for an important real-world OOD detection problem.

A Additional details about our proposed likelihood ratio method

The proposed likelihood-ratio method fits an in-distribution model and a perturbed-input background model, then predicts OOD when their likelihood-ratio statistic is small. Its procedure requires two generative-model runs.

  • Algorithm 1 perturbs discrete inputs by independently mutating positions at rate µ and sampling replacement values from the input vocabulary.The vocabulary is {A, C, G, T} for genomic sequences and {0, ..., 255} for images.
  • The complete method is specified in Algorithm 2 and has twice the runtime of a standard generative model.
  • The method fits pθ(x) on in-distribution data and pθ0(x) on perturbed inputs to form a likelihood-ratio statistic for OOD detection.OOD is predicted when the likelihood-ratio statistic is small.
  • The background model is trained on perturbed input data generated by Algorithm 1, with model regularization optionally applied.

B.3 Supplementary tables

The supplementary results compare likelihood-ratio OOD detection with pure likelihood and classifier-based baselines on CIFAR-10 versus SVHN, while visualizing likelihood-ratio behavior on FashionMNIST and CIFAR-10.

  • AUROC improves from 0.095 with pure likelihood to 0.931 with likelihood-ratio detection on CIFAR-10 versus SVHN.
  • Classifier-based ensemble methods achieve AUROCs from 0.937 to 0.946, slightly above the likelihood-ratio method.The likelihood-ratio approach is unsupervised, whereas classifier methods require labels.
  • FashionMNIST visualizations show examples with the highest and lowest log-likelihood-ratios.
  • CIFAR-10 visualizations plot images alongside per-pixel log-likelihoods and log likelihood-ratios, using lighter gray for larger values.The plotted likelihood and likelihood-ratio values use different ranges, with likelihoods mostly negative and likelihood-ratios mostly positive.
  • Table S2 reports AUROC, AUPRC, and FPR80 for likelihood, likelihood-ratio, and baseline OOD detectors on CIFAR-10 versus SVHN.Reported values are means with standard errors from 10 independent runs.

C.1 Dataset design

The dataset uses bacterial discovery dates and taxonomy to construct realistic in-distribution and OOD splits, then evaluates sequence models and OOD detection across genomic classes.

  • Dataset construction: 11,672 bacterial genomes were downloaded from NCBI in September 2018 and organized by taxonomy from species through phylum.Higher taxonomy levels represent broader classifications, while species and genus provide more refined classifications.
  • Dataset construction: Discovery years define the splits: classes found before 2011 form training in-distribution data, while later classes provide OOD validation and test data.The protocol also separates later sequencing releases from old classes for in-distribution validation and test sets.
  • Dataset construction: The benchmark contains 10 in-distribution classes, 60 OOD validation classes, and 60 OOD test classes interlaced under the same taxonomy.Genomes are fragmented into 250-base-pair sequences to mimic current sequencing data.
  • Models and evaluation: The genomic generative model is an LSTM that predicts each DNA base from sequence history using one-hot inputs and a four-letter softmax.It is trained only on in-distribution training data.
  • Models and evaluation: The background model is trained on perturbed in-distribution data, with mutation rate and optional L2 regularization tuned on validation data without final-test OOD exposure.The reported optimal mutation rate is 0.2, yielding AUROC 0.763 on validation data and 0.727 on the test dataset.
  • Models and evaluation: Using mutated in-distribution data as simulated OOD inputs provides a way to tune hyperparameters without using actual OOD inputs.The mutation rate is set to 10% based on reported average bacterial identity at genus and family levels.
  • Models and evaluation: A CNN classifier baseline processes one-hot DNA sequences through convolution, max pooling, and dense layers before class prediction.
  • OOD difficulty: Likelihood-ratio AUROC correlates with minimum genetic distance at 0.570, compared with 0.277 for the classifier ensemble.The analysis uses the minimum genetic distance between each OOD class and the in-distribution classes.
Loading 1906.02845v2…