PaperlayerView exact revision on arXiv

Source-linked AI summary

Denoising diffusion models for out-of-distribution detection

Mark S. Graham, Walter H. L. Pinaya, Petru-Daniel Tudosiu, Parashkev Nachev, Sebastien Ourselin, M. Jorge Cardoso

arXiv:2211.07740v4cs.LGcs.CV

TL;DR

Unsupervised OOD detection matters for safe deployment, but generative-model metrics can misrank OOD samples, while reconstruction methods require difficult bottleneck tuning. This paper uses DDPM reconstructions across noise levels as a multi-dimensional OOD signal. It reports outperforming reconstruction-based methods and state-of-the-art generative approaches on computer-vision and medical datasets, with computational expense as a limitation.

  • Problem

    Generative-based approaches dominate unsupervised OOD detection, while reconstruction methods require careful information-bottleneck tuning to work well.

    Source: §1 · pp. 1–2

  • Method

    The method uses DDPMs to reconstruct inputs noised to a range of levels and uses the resulting reconstruction-error measurements to detect OOD inputs.

    Source: §1 · p. 2

  • Results

    Across computer-vision benchmarks and higher-dimensional medical datasets, the DDPM method outperforms reconstruction-based methods and state-of-the-art generative approaches.

    Source: §5 · p. 8

  • Takeaways & Limitations

    DDPMs provide reconstruction-based OOD detection without choosing a single fixed bottleneck, and the method performs well across the tested computer-vision and medical datasets.

    Source: §5 · p. 8

    The method is more computationally expensive than competing methods, which may make it less suitable where predictions must be real time.

    Source: §4.7 · p. 8

Abstract

from arXiv · show

Out-of-distribution detection is crucial to the safe deployment of machine learning systems. Currently, unsupervised out-of-distribution detection is dominated by generative-based approaches that make use of estimates of the likelihood or other measurements from a generative model. Reconstruction-based methods offer an alternative approach, in which a measure of reconstruction error is used to determine if a sample is out-of-distribution. However, reconstruction-based approaches are less favoured, as they require careful tuning of the model's information bottleneck - such as the size of the latent dimension - to produce good results. In this work, we exploit the view of denoising diffusion probabilistic models (DDPM) as denoising autoencoders where the bottleneck is controlled externally, by means of the amount of noise applied. We propose to use DDPMs to reconstruct an input that has been noised to a range of noise levels, and use the resulting multi-dimensional reconstruction error to classify out-of-distribution inputs. We validate our approach both on standard computer-vision datasets and on higher dimension medical datasets. Our approach outperforms not only reconstruction-based methods, but also state-of-the-art generative-based approaches. Code is available at https://github.com/marksgraham/ddpm-ood.

1. Introduction

Unsupervised OOD detection is useful because it requires only in-distribution training data, but generative-model metrics can misrank in- and out-of-distribution samples. Reconstruction-based methods avoid relying on likelihood, yet their bottlenecks are difficult to tune; this work uses DDPM denoising at multiple noise levels to measure reconstruction errors.

1. Introduction

  • Unsupervised OOD detection requires only in-distribution training data and does not assume a particular OOD-data form or downstream task.OOD detection is presented as important to safe deployment of machine-learning systems.Source: §1 · p. 1
  • Generative-model likelihoods can misrank OOD and in-distribution data; CIFAR10-trained models have assigned higher likelihoods to SVHN samples [3] [9].The introduction notes that proposed fixes still leave these methods vulnerable to at least some OOD data [45].Source: §1 · pp. 1–2
  • Reconstruction-based OOD detection avoids likelihood-based metrics, but a fixed information bottleneck is difficult to tune: too small harms in-distribution reconstruction, while too large can reconstruct OOD inputs.The authors identify this tuning burden as a reason reconstruction methods have been overlooked in favor of generative models.Source: §1 · p. 2
  • DDPM noise level acts as an externally controlled information bottleneck, allowing one trained model to reconstruct inputs at multiple bottleneck levels.Partial noise retains input information and conditions denoising toward reconstruction; full noise retains none.Source: §1 · p. 2
  • The proposed method measures reconstruction errors for inputs noised to a range of levels and uses this multi-dimensional set of metrics to detect OOD images.This applies DDPMs to reconstruction-based OOD detection.Source: §1 · p. 2

2. Related Work

Related work covers unsupervised detection, which uses only in-distribution data, across generative and reconstruction-based approaches. The reviewed approaches address shortcomings in likelihood scoring and bottleneck selection, while the closest DDPM-based work uses a single tuned noise level.

2.1. Generative based

  • Generative approaches commonly threshold likelihoods, but CIFAR10-trained model families have assigned higher likelihoods to SVHN than to CIFAR10 [3] [9].Follow-up work explores WAIC, likelihood ratios, hierarchical VAEs, and typicality-based statistics to address likelihood shortcomings [3] [8] [23] [25] [30].Source: §2.1 · p. 2

2.2. Reconstruction based

  • Reconstruction-based methods flag inputs with low input–reconstruction similarity, relying on a bottleneck to reconstruct in-distribution data better than OOD data.A bottleneck that is too small harms in-distribution reconstruction; one that is too large can let OOD samples be reconstructed with low error, prompting dataset-specific tuning [4].Source: §2.2 · pp. 2–3
  • Mahalanobis feature distances and memory modules have been proposed for reconstruction-based detection, but the reviewed methods still require bottleneck tuning for each dataset pairing [4].Source: §2.2 · p. 3
  • AnoDDPM reconstructs from one dataset-tuned noise level for localized anomaly detection, whereas this work uses a range; a complementary DDPM-corruption approach was developed concurrently [20].Source: §2.2 · p. 3

3. Method

The method trains a DDPM to reverse a Gaussian-noising process, then evaluates reconstructions from multiple noise levels. Noise level supplies an external bottleneck, and normalized MSE and LPIPS measurements across reconstructions form the OOD score.

3.1. The diffusion model

  • Training adds Gaussian noise at timestep t using variance schedule β_t, then trains a single network to reverse the diffusion process from x_t and t.The schedule increases with t so fully noised x_T is close to an isotropic Gaussian and contains no information about x_0.Source: §3 · p. 3

3. Method

  • Noise level serves as an externally chosen information bottleneck, so one model can reconstruct at multiple levels without selecting a dataset-specific bottleneck through its architecture.Source: §3 · p. 3

3.2. Multiple reconstructions

  • With T = 1000 and 100 starting points, PLMS reduces reconstruction cost from 50,500 to 5,050 model evaluations, a 10× speed-up.The comparison uses 100 sampling steps instead of the full 1,000-step chain.Source: §3.2 · p. 3

3.3. Evaluating similarity

  • The method computes MSE and LPIPS for each of N reconstructions, converts each measurement to a validation-set Z-score, and averages the 2N scores into an OOD score.LPIPS measures perceptual similarity using distances between AlexNet deep features and is reported to correlate with human image-similarity evaluations [47].Source: §3.3 · pp. 3–4

4. Experiments and Results

DDPM ranks highest in the reported computer-vision and medical evaluations, and the experiments probe compute trade-offs and scoring variants.

4.1. Experimental details

  • The evaluation covered 15 computer-vision dataset pairings and six MedNIST classes, benchmarking generative and reconstruction-based approaches.Computer-vision tests included flipped versions as OOD data; medical images were evaluated at native and upsampled dimensions.Source: §4.1 · p. 4

4.2. Results for computer vision datasets

  • DDPM had the highest average rank and beat DoSE on 13/15 computer-vision pairings; it also ranked highest on medical comparisons, with 99.3 its lowest pairing score.On SVHN, in-distribution reconstructions remained similar through t=500–600, while OOD reconstructions began looking dissimilar after t=100.Source: §4.2 · p. 5, Table 1 · p. 5, §4.3 · p. 6

4.3. Results on medical datasets

  • In medical comparisons, DDPM’s advantage over other methods was more pronounced than on computer-vision datasets: it ranked highest and performed most consistently, with 99.3 its lowest performance for any dataset pairing.Table 2 reports AUC at image sizes 64 × 64 and 128 × 128; each value considers all other datasets at the same dimension as OOD datasets.Source: §4.3 · p. 6, Table 2 · p. 6

4.4. Bottleneck size

  • Standard reconstruction baselines had no single bottleneck setting that worked well across all pairings, underscoring the need for dataset-specific tuning.A 4-layer MemAE scored 91.5 on FashionMNIST vs MNIST but 89.3 on SVHN vs CIFAR10, the poorest result for that pairing among the models tested.Source: §4.4 · p. 7

4.5. Performance and number of reconstructions

  • Reducing reconstructions from 100 to 25 caused a mean AUC drop of 0.58; with 13 reconstructions, the drop was 1.49 and DDPM still beat DoSE on 13/15 pairings.The 13-reconstruction setting required 637 model evaluations; reducing maxT improved the compute-performance curve, but did not outperform the full maxT=1000, 100-reconstruction model.Source: §4.5 · p. 7, Figure 3 · p. 7

4.6. Variants of the proposed method

  • Z-score averaging consistently outperformed SVM and GMM scoring, while combining MSE and LPIPS was nearly always better than either similarity metric alone.The authors suggest Z-score averaging helps when in-distribution and OOD signals are noisy and overlap substantially.Source: §4.6 · p. 8, Table 3 · p. 8

4.7. Limitations

  • The method’s greater computational cost may limit its use where predictions must be made in real time.The authors contrast possible medical-imaging use, where flagging an input within minutes may be acceptable, with self-driving, where predictions need to be real time.Source: §4.7 · p. 8

5. Conclusion

The paper uses DDPM reconstructions at different noise levels to address the fixed-bottleneck choice in standard reconstruction techniques. Tests on computer-vision and higher-image-size medical datasets show the method outperforming reconstruction-based and state-of-the-art generative approaches.

5. Conclusion

  • The DDPM-based method outperforms reconstruction-based methods and state-of-the-art generative approaches across the reported experiments.The experiments cover computer-vision benchmarks and medical imaging datasets with higher image sizes.Source: §5 · p. 8
  • The method reconstructs inputs noised to different extents, addressing standard reconstruction techniques’ need to choose a single fixed bottleneck.Source: §5 · p. 8
  • The evaluation includes computer-vision benchmarks and medical imaging datasets with higher image sizes.Source: §5 · p. 8

A. Sample reconstructions on computer vision datasets.

The figure presents sample reconstructions from all models at ten starting points spanning the full T-chain.

A. Sample reconstructions on computer vision datasets.

  • The figure shows example reconstructions from all models at ten starting points spanning the full T-chain.Source: Figure 6 · p. 11

B. Sample reconstructions on medical datasets.

The medical-dataset figure shows reconstructions across noise-chain positions for an in-distribution input and several OOD inputs. The accompanying table caption describes AUC results for bottleneck variations.

B. Sample reconstructions on medical datasets.

  • The figure shows reconstructions at ten equally spaced t-values for a MedNIST hand model trained at 6x × 64, with an in-distribution input and five medical OOD datasets.The OOD rows are Abdomen CT, Breast MRI, Chest CT, Chest X-ray, and Head CT.Source: Figure 7 · p. 12
  • Table 4 reports AUC scores as the information bottleneck varies, with the highest value per column bolded separately for each model class.Source: Table 4 · p. 13

C. Effect of bottleneck size on the performance of reconstruction-based methods

Changing reconstruction-model bottlenecks can improve performance on one dataset while reducing it on another. The reported examples illustrate this trade-off for MemAE, supporting the authors’ concern about dataset-specific tuning.

C. Effect of bottleneck size on the performance of reconstruction-based methods

D. Performance of DDPM vs number of reconstructions

The DDPM OOD detector can use fewer reconstructions or a lower maximum reconstruction time step with limited effect on OOD performance.

D. Performance of DDPM vs number of reconstructions

  • Reducing reconstruction count or maximum t substantially cuts model evaluations with limited effect on OOD performance.The section reports both changes as ways to reduce the number of reconstructions performed.Source: D. Performance of DDPM vs number of… · p. 13
  • Table 5 reports AUCs for each dataset pairing as the number of reconstructions used varies.Source: Table 5 · p. 14
  • Table 5's first row uses maxT = 1000 and 100 reconstructions, the parameters reported in Table 1.Source: Table 5 · p. 14

Related papers

Loading 2211.07740v4…