Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study

Christoph Baur; Stefan Denner; Benedikt Wiestler; Shadi Albarqouni; Nassir Navab

Source-linked AI summary

Autoencoders for Unsupervised Anomaly Segmentation in Brain MR Images: A Comparative Study

Christoph Baur, Stefan Denner, Benedikt Wiestler, Shadi Albarqouni, Nassir Navab

arXiv:2004.03271v2Apr 7, 2020eess.IVcs.CVcs.LG

TL;DR

Brain-MRI UAD seeks to model healthy anatomy without extensive pixel-level annotations, but prior evaluations are difficult to compare. This study standardizes architecture, resolution, and data to compare methods, finding restoration-based VAE strongest overall while identifying dataset limitations and benchmark needs.

Problem
Prior UAD studies use different datasets, pathologies, resolutions, and architectures, limiting direct comparison; UAD matters because annotation-heavy supervised methods are costly and may miss unfamiliar or rare lesions.
Method
The study compares reconstruction- and restoration-based methods across multiple brain-MRI datasets using a unified architecture, common resolution, and a shared experimental pipeline.
Results
Restoration applied to a VAE achieves the best performance on DMS, DGB, and DMSLUB, while f-AnoGAN performs best on MSSEG2015.
Takeaways & Limitations
Latent-variable models generally outperform classic autoencoders for anomaly segmentation, with the VAE recommended among them for performance and optimization simplicity.
Takeaways & Limitations
The datasets remain limited, motivating a benchmark spanning many healthy subjects, pathologies, scanners, genders, and ages.

Abstract

from arXiv · show

Deep unsupervised representation learning has recently led to new approaches in the field of Unsupervised Anomaly Detection (UAD) in brain MRI. The main principle behind these works is to learn a model of normal anatomy by learning to compress and recover healthy data. This allows to spot abnormal structures from erroneous recoveries of compressed, potentially anomalous samples. The concept is of great interest to the medical image analysis community as it i) relieves from the need of vast amounts of manually segmented training data---a necessity for and pitfall of current supervised Deep Learning---and ii) theoretically allows to detect arbitrary, even rare pathologies which supervised approaches might fail to find. To date, the experimental design of most works hinders a valid comparison, because i) they are evaluated against different datasets and different pathologies, ii) use different image resolutions and iii) different model architectures with varying complexity. The intent of this work is to establish comparability among recent methods by utilizing a single architecture, a single resolution and the same dataset(s). Besides providing a ranking of the methods, we also try to answer questions like i) how many healthy training subjects are needed to model normality and ii) if the reviewed approaches are also sensitive to domain shift. Further, we identify open challenges and provide suggestions for future community efforts and research directions.

I. INTRODUCTION

Unsupervised anomaly detection models healthy brain anatomy from healthy MRI and identifies pathology through abnormal reconstruction behavior. This study addresses comparability problems by evaluating recent methods under unified experimental conditions.

Motivation: Supervised methods require large, costly annotated datasets and mainly detect lesions resembling their training examples.These limitations are especially important for rare diseases, where training data are difficult to collect.
Motivation: UAD learns normal brain anatomy from healthy data and can potentially detect arbitrary anomalies without pixel-level annotations.The reviewed approaches model healthy anatomy with deep representation learning and treat anomalies as deviations from the learned normative distribution.
Research gap: Comparing prior studies is difficult because they use different datasets, pathologies, resolutions, and model architectures with varying complexity.These differences hinder conclusions about the relative strengths and weaknesses of the methods.
Study design: The study compares reconstruction- and restoration-based UAD methods across four datasets and two pathologies using pixel-wise segmentation measures.A single unified architecture controls model complexity where applicable.
Modeling healthy anatomy: Autoencoders compress healthy anatomy into a lower-dimensional representation and reconstruct it, while variants regularize or otherwise model the latent distribution.The study includes classic AEs, VAEs, adversarial variants, Gaussian-mixture models, and GAN-based approaches.

B. Anomaly Segmentation

The reviewed methods generate anomaly maps by comparing pathological inputs with healthy-oriented model outputs. They differ in whether they use direct reconstruction residuals, sampled consensus reconstructions, gradients, or latent-space restoration.

Reconstruction-based methods: Reconstruction-based methods use pixel-wise residuals between an input and its reconstruction to highlight structures that are poorly reconstructed.The approach assumes unseen anomalous structures produce higher reconstruction errors.
Monte Carlo methods: Monte Carlo methods average residuals from multiple latent-space reconstructions, while deterministic autoencoders can use latent dropout during inference.The number of samples N and dropout rate control the consensus reconstruction procedure.
Gradient-based methods: Gradient-based methods optimize an objective that brings reconstruction and input together while moving the latent representation toward the normal prior.The resulting pixel-wise gradients serve as anomaly saliency maps.
Restoration-based methods: Restoration-based methods optimize the latent representation or image to reconstruct a healthy counterpart of the query image.Anomalies are then detected from residual maps between the input and restored image.

III. EXPERIMENTS

The experiments introduce the datasets and preprocessing, then establish a unified network architecture for the investigated models. They subsequently describe post-processing, evaluation metrics, and analyses of the results.

III. EXPERIMENTS: The experiments begin with dataset descriptions and preprocessing before presenting the unified network architecture underlying the investigated models.This architecture provides the foundation for subsequent model comparisons.
III. EXPERIMENTS: The study also explains post-processing and evaluation metrics before presenting and discussing results from multiple perspectives.

A. Datasets

The study uses multiple MRI datasets spanning healthy subjects, MS lesions, and glioma, with differing scanners enabling evaluation of generalization. All scans undergo a shared spatial and intensity preprocessing pipeline before patient-wise splitting.

Datasets: The primary dataset contains 138 healthy subjects, 48 subjects with MS lesions, and 26 subjects with glioma, scanned on one Philips Achieva 3T system.It includes FLAIR, T2-weighted, and T1-weighted scans reviewed and annotated by expert neuroradiologists.
Datasets: MSLUB contains 30 subjects with MS scanned on a Siemens Magnetom Trio 3T system, with consensus segmentations from three expert raters.
Datasets: MSSEG2015 contains 21 scan sessions from five subjects and is used to test model generalization because its intensity distribution differs from the primary datasets.The data were acquired with a 3.0 Tesla Philips MRI scanner, though the exact device is unknown.
Preprocessing: All scans are registered to SRI24 ATLAS space, skull-stripped, denoised, normalized to [0,1], and randomly split patient-wise into training, validation, and testing sets.Normalization divides each scan by its 98th percentile.

B. Network Architecture and Models

A single empirically selected architecture provides the basis for comparing diverse autoencoder, latent-variable, Bayesian, restoration, and adversarial models. Models are trained on healthy data and evaluated through reconstruction-based or restoration-based anomaly detection.

Unified architecture: The unified architecture was selected through manual iterative search to achieve low reconstruction error on healthy training and validation data.
Autoencoders: The comparison includes non-generative autoencoders, including a spatial AE with bottleneck z ∈ R8×8×128.
Generative models: Generative models use the same unified architecture and bottleneck configurations, while AnoVAEGAN and f-AnoGAN use unified encoder-decoder networks.Their discriminator is a replica of the encoder.
Generative models: AnoVAEGAN and f-AnoGAN are optimized with Wasserstein loss to avoid GAN training instabilities and mode collapse.
Training and detection: Models train on healthy data until convergence, after which VAE and GMVAE additionally perform density-based image restoration using 500 iterations per sample.Training uses automatic early stopping based on held-out validation reconstruction loss.
Bayesian variants: Bayesian AE and Bayesian VAE use Monte Carlo reconstruction, with Bayesian VAE settings including 100 MC samples and dropout rate 0.2.Context VAE also uses a gradient-based approach in the reported configuration.

C. Postprocessing

All models use the same residual post-processing pipeline to suppress boundary artifacts and incorporate lesion-specific intensity prior knowledge.

Residual masking: Residual images are multiplied by a slightly eroded brain mask to remove prominent residuals near brain-mask boundaries and cortical gyri and sulci.
Lesion-specific processing: For MS lesion datasets, only positive residuals are retained because the lesions are known to be fully hyper-intense in FLAIR images.

D. Metrics

The study evaluates voxel-level anomaly segmentation with precision-recall measures and complementary reconstruction diagnostics, then compares model types, bottlenecks, training-set size, and domain shift. Results indicate that latent-space structure and model choice affect performance, while overly accurate spatial reconstruction can preserve anomalies.

Metrics: Voxel-level anomaly segmentation is evaluated with dataset-specific precision-recall curves and their area under the curve, avoiding a fixed operating point.
Metrics: AUROC is also reported, but the authors caution that heavy class imbalance can make ROC-based performance appear overly optimistic.
Reconstruction diagnostics: The analysis supplements segmentation metrics with ℓ1 reconstruction errors on normal and anomalous voxels and χ2 distances between their residual histograms.
Comparative analysis: The experiments compare model types and bottleneck designs across four datasets and two pathologies, using a unified evaluation framework.
Dense vs Spatial Bottleneck: Dense models generally outperform spatial variants, while spatial models at 128x128px reconstruct anomalies too well to support anomaly detection.The spatial AE performs slightly better than its dense counterpart on DGB.

H. Latent Variable Models

Latent-variable comparisons show that model performance depends on both the latent formulation and pathology, while GAN-based approaches improve reconstruction sharpness but can compromise anatomical coherence. Monte-Carlo sampling provides no consistent performance gain.

The VAE performs best among compared latent-variable models on DMS, DMSSEG2015, and DMSLUB.
On Glioblastoma, the GMVAE and Constrained AAE outperform the VAE, while the Constrained AAE is weaker on MS datasets.
GAN-based models produce sharper reconstructions than AEs, especially near brain boundaries.
f-AnoGAN preserves input shape and yields stronger residual separation, outperforming standard AE by over 15% and VAE by over 6% on average.
Monte-Carlo dropout slightly worsens most metrics, while Bayesian VAE offers only dataset-dependent differences from VAE.

K. Reconstruction vs Restoration

Restoration-based UAD generally outperforms reconstruction-based methods, with dense bottleneck models strongest across pathologies. Domain shift reduces anomaly-segmentation performance mainly where lesion contrast is weaker, while more training data and greater complexity have conditional effects.

Reconstruction vs Restoration: Restoration-based UAD exceeds reconstruction-based counterparts by 4–20%, depending on latent model and bottleneck type.
Reconstruction vs Restoration: Dense restoration methods outperform f-AnoGAN in AUPRC and ⌈DICE⌉ except on DMSSEG2015.
Domain Shift: UAD performance drops on DMSLUB and DMSSEG2015, but normal-pixel reconstruction error remains nearly stable across domains.
Domain Shift: Anomalous-pixel reconstruction error is smaller on DMSLUB and DMSSEG2015, indicating weaker contrast between normal tissue and lesions.
Different Pathologies: On Glioblastoma, tumor compartments resembling normal tissue and complex shapes make precise segmentation difficult; restoration methods remain strongest.
Healthy Training Data: The GMVAE with dense bottleneck is generally the top performer as healthy training data increases, although behavior varies by dataset.
Model Complexity: VAE and restoration-based GMVAE models remain stable candidates with original architectures, whereas higher complexity can reconstruct larger lesions better, reducing anomaly contrast.

P. Reconstruction Fidelity and UAD Performance

The study relates anomaly-segmentation performance to reconstruction errors and residual-histogram separability across datasets, finding that distinguishable normal and anomalous residuals matter more than general reconstruction fidelity. It also highlights dataset-specific behavior, method trade-offs, and unresolved evaluation challenges.

Correlation analysis: Residual-histogram separability correlates most strongly with AUPRC and DICE on DMS and DMSLUB.Its correlation with anomalous-pixel reconstruction error is moderate to strong, whereas correlation with normal residuals is weaker.
Correlation analysis: On DGB, segmentation performance improves as reconstruction errors increase for both normal and anomalous intensities.The authors hypothesize that poorer reconstructions of tumors increase their separability from normal tissue.
Correlation analysis: On DMSSEG2015, the expected correlations are largely absent, reflecting poor image contrast that makes UAD unsuitable.Normal and anomalous reconstruction errors are strongly correlated and relate similarly to the other metrics.
Method ranking: VAE (restoration) ranks first on DMS, DGB, and DMSLUB, including under domain shift, but requires multiple seconds per slice and minutes per volume.Purely reconstruction-based methods offer faster feed-forward inference.
Method ranking: f-AnoGAN achieves the best performance on the challenging MSSEG2015 dataset and remains only slightly below the restoration winner elsewhere.Latent-variable models outperform classic AEs despite blurrier reconstructions, because their normal–anomalous reconstruction-error gap is larger.

IV. CONCLUSION

The paper compares autoencoder-based brain-MRI anomaly-segmentation methods under a unified architecture and image resolution. It identifies residual-error differences between normal and abnormal tissue as the best performance indicator and proposes benchmark development as a central next step.

IV. CONCLUSION: The study compares autoencoder-based methods that model healthy anatomy to detect abnormal structures in brain MRI.The comparison covers reconstruction- and restoration-based approaches across multiple datasets and pathologies.
IV. CONCLUSION: A unified architecture and image resolution support a transparent ranking and publicly reusable implementations, preprocessing, and evaluation pipelines.The authors encourage contributions so future studies can compare methods without rebuilding the full experimental pipeline.
IV. CONCLUSION: A benchmark dataset for brain-MRI UAD is the paper’s main proposed future direction, alongside studies of model complexity and image resolution.The proposed benchmark should address variation in healthy subjects, pathologies, scanners, genders, and ages.

APPENDIX A

The appendix figure shows normalized residual histograms for normal and anomalous pixels across the intensity range from 0 to 1.0, excluding zero residuals.

APPENDIX A: Normalized residual histograms compare normal pixels in blue with anomalous pixels in red over residual intensities from 0 to 1.0.Residuals that are completely zero are excluded.