BMAD: Benchmarks for Medical Anomaly Detection

1. Introduction

• Medical anomaly detection aims to alert when abnormalities occur, since rare and heterogeneous anomalies are difficult to enumerate for supervised training.Normal data are comparatively easier to collect, while the open-set nature of medical data makes comprehensive abnormal training samples implausible.Source: §1 · p. 1
• Existing benchmarks focus mainly on industrial or natural images, while medical AD datasets and reorganization practices lack consistency, making fair comparison difficult.Prior work often repurposes datasets built for classification or segmentation, with inconsistencies in source citations and no clear consensus on reorganization.Source: §1 · p. 1
• BMAD provides a standardized medical AD benchmark with six reorganized datasets from five domains, three evaluation metrics, and fifteen SOTA algorithms.The codebase supports comprehensive comparisons and the benchmark evaluates algorithm strengths and weaknesses.Source: §1 · p. 2

2. Related work

• Unsupervised anomaly detection is organized into reconstruction-based methods and feature-embedding or projection-based methods.Reconstruction compares the original and reconstructed data, whereas projection methods analyze abstract representations in embedding space.Source: §2 · p. 2

2.1. Reconstruction-based Methods

• Reconstruction-based methods seek small reconstruction residuals for normal samples and larger errors for anomalies.Methods include autoencoders, GANs, diffusion models, transformers, denoising or masking strategies, synthetic abnormalities, and memory mechanisms.Source: §2.1 · p. 3

2.2. Projection-based Methods

• Projection-based methods map data into embedding space; teacher-student discrepancy, memory banks, one-class models, and normalizing flows provide distinct detection mechanisms.Teacher-student methods use representation discrepancy, memory-bank methods compare queries with stored prototypes, and flow methods model normal-feature distributions.Source: §2.2 · p. 3

3.1. Medical AD Benchmarks

• BMAD comprises six benchmarks across five medical domains, with three supporting pixel-level evaluation and three supporting sample-level assessment only.Dataset selection considered imaging modality and source-domain diversity as well as licenses for reorganization, remixing, and redistribution.Source: §3.1 · p. 3, Table 1 · p. 4
• The six benchmarks reorganize existing datasets for AD, including two retinal OCT datasets for localization and sample-level detection and Camelyon16 patches for histopathology.RESC uses segmentation annotations for localization, while OCT2017 uses disease-free training images and labels three disease classes as abnormal; Camelyon16 contributes 40X patches.Source: §3.1 · p. 4

3.2. Evaluation Metrics

• BMAD uses sample- and pixel-level AUROC, while PRO supplements localization evaluation by weighting anomaly components equally, including small regions.The benchmark excludes DICE comparison from the main results because its optimal threshold varies across algorithms and tasks.Source: §3.2 · pp. 4–5

3.3. Supported AD Algorithms

• BMAD integrates fifteen algorithms: four reconstruction-based methods and eleven feature-embedding methods, including reconstruction, distillation, memory-bank, one-class, and flow approaches.Examples include AnoGAN and DRAEM, RD4AD, PatchCore, and CFlow or CS-Flow.Source: §3.3 · p. 5

4.1. Implementation Details

• The evaluation starts with default hyperparameters, tunes selected settings for models that do not converge, and reports means and standard deviations across five runs.The best checkpoint is selected using validation accuracy, and anomaly maps are min-max normalized for visualization.Source: §4.1 · p. 5

4.2. Results and Discussions

• Feature-based methods perform better overall than reconstruction-based methods, although no single algorithm consistently outperforms others.The authors attribute this pattern partly to generative models reconstructing some anomalous regions well and pixel-level residuals missing high-level contextual abnormalities.Source: §4.2 · p. 5, Table 2 · p. 6
• For Liver CT and Brain MRI, black backgrounds and imbalanced normal and anomalous samples can bias pixel-level AUROC toward correctly classifying normal pixels rather than detecting anomalies.Source: §4.2 · p. 6
• Knowledge-distillation methods, especially RD4AD [19], achieve better localization; PatchCore [46] is more convincing for sample-level detection but produces coarse anomaly maps.DRAEM is particularly sensitive to texture variation, while CFlow has poor localization; the authors hypothesize that DRAEM and UTRAD differ because of their CNN and transformer architectures.Source: §4.2 · p. 6, Figure 2 · p. 7
• PatchCore [46], RD4AD [19], and CS-FLOW [48] rank among the top three models across multiple benchmarks for performance, but CS-Flow is less efficient at generating pixel-level anomaly maps.Figure 3 compares accuracy with inference time per image and uses circle size for GPU memory consumption.Source: §4.2 · p. 6, Figure 3 · p. 8
• Brain MRI and Liver CT are better suited to low-level feature-based anomaly augmentation, whereas Chest X-ray and Histopathology abnormalities often involve observable structural or appearance changes.The authors argue that anomaly synthesis must account for domain-specific shape, texture, color, and morphology, particularly for rare diseases.Source: §4.2 · pp. 6–7
• The results indicate that carefully designed methods using pre-trained representations, including knowledge-distillation and memory-bank approaches, can improve medical anomaly detection.Source: §4.2 · p. 7
• Memory-bank methods face hardware demands for efficient storage and retrieval, and may be weaker at localizing anomalies than feature-reconstruction methods.The authors also report model degradation as network depth increases and identify preprocessing and augmentation as challenging; adversarial training is proposed as a possible approach.Source: §4.2 · pp. 7–8

5. Conclusion and Final Remark

• BMAD brings together six benchmarks from five medical fields and 15 SOTA algorithms spanning major AD design paradigms.The paper presents it as a comprehensive evaluation framework for medical anomaly detection.Source: §5 · p. 8
• The evaluation uses a one-for-one paradigm, while unified one-for-N models remain an unevaluated direction for BMAD.The authors say evaluating unified models would provide insights for designing more generic and versatile solutions.Source: §5 · p. 8
• Data collected mostly in advanced countries may carry geographical and sampling biases.Source: §5 · p. 8
• Following original hyperparameter settings means experiments may not use values optimal for particular datasets.The codebase provides an interface for adjusting supported algorithms’ hyperparameters.Source: §5 · p. 8

Supplementary Material

• The supplement has four sections covering datasets, supported AD algorithms, experimental reproducibility, and evaluation metrics.Source: Supplementary Material · p. 13

6.1.1 BraTS2021 Dataset

• The brain MRI benchmark is reorganized from BraTS2021, a multimodal brain-tumor segmentation dataset with 1,251 training, 219 validation, and 530 testing cases.The original scans are 3D volumes with four modalities and tumor-segmentation annotations.Source: §6.1.1 · p. 13, §6.1 · p. 13

6.1.2 Construction of Brain MRI AD benchmark

• The brain benchmark uses BraTS2021 FLAIR volumes, selecting slices at depths 60–100 and saving them as 240 × 240 PNG images.It uses 7,500 normal training samples, 3,715 test samples containing normal and anomalous slices in a 1:1 ratio, and 83 non-overlapping validation samples.Source: §6.1.2 · p. 13

6.2. Liver CT Anomaly Detection and Localization Benchmark

• The liver benchmark combines anomaly-free BTCV scans for training with LiTS scans for evaluation and testing.The authors exclude lesion-free LiTS regions from training because lesions there may cause morphological changes in non-lesion regions.Source: §6.2 · p. 13, §6.2.3 · p. 14

6.2.1 BTCV Dataset

• BTCV was created for multi-organ segmentation and contains 50 abdominal CT scans from patients with colorectal cancer and retrospective ventral hernia.Source: §6.2.1 · p. 13

6.2.2 LiTS Dataset

• LiTS was created for liver-tumor segmentation and comprises 131 abdominal CT scans with liver and tumor labels.Source: §6.2.2 · p. 13

6.2.3 Construction of Liver CT AD Benchmark

• Liver benchmark preparation converts scans to grayscale with an abdominal window, extracts liver regions using annotations, and normalizes slice intensity with histogram equalization.BTCV supplies 1,542 training slices; LiTS supplies healthy and abnormal samples for testing and validation.Source: §6.2.3 · p. 14

6.3. Retinal OCT Anomaly Detection and Localization Benchmark

• The retinal OCT benchmark contains two datasets: RESC supports anomaly localization, while OCT2017 supports sample-level anomaly detection only.Source: §6.3 · p. 14

6.3.1 RESC dataset

• RESC focuses on retinal edema and provides pixel-level segmentation labels for affected regions.Images are supplied as 512 × 1024 PNGs.Source: §6.3.1 · p. 14

6.3.2 OCT2017 dataset

• OCT2017 was developed for classification and contains retinal images labeled as CNV, DME, or drusen.The images are continuous slices sized 512 × 496.Source: §6.3.2 · p. 15

6.3.3 Preparation of OCT AD benchmarks

• RESC uses normal training samples and adapts its original validation set for evaluation; OCT2017 uses disease-free training samples and divides test data into evaluation and test sets.Source: §6.3.3 · p. 15

6.4.1 RSNA dataset

• RSNA contains 26,684 DICOM lung images labeled Normal, Lung Opacity, or No Lung Opacity/Not Normal.The challenge data may be used for academic research and education, as well as other non-commercial purposes.Source: §6.4.1 · p. 15

6.4.2 Preparation of Chest X-ray AD Benchmark

• The reorganized benchmark treats Lung Opacity and No Lung Opacity/Not Normal as abnormal, with 8,000 normal training images, 1,490 validation images at a 1:1 normal-to-abnormal ratio, and 17,194 test images.Source: §6.4.2 · p. 15

6.5.1 Camelyon16 Dataset

• Camelyon16 contains 400 H&E-stained lymph-node whole-slide images, including 159 with pathologist-annotated tumor metastases.The highest available resolution is level 0 at 40X magnification; the dataset is available under CC0.Source: §6.5.1 · pp. 15–16
• Histopathology images show greater diversity in tissue components than other medical-image anomaly-detection benchmarks.Source: Figure 9 · p. 16

6.5.2 Preparation of histopathology AD Benchmark

• The benchmark evaluates histopathology at patch level because of the images’ large size, extracting 5,088 normal training patches from 160 normal slides.Validation uses 100 normal and 100 abnormal patches from 13 testing slides; testing uses 1,000 of each from 115 testing slides, with patches saved as 256 × 256 PNGs.Source: §6.5.2 · p. 16

• Anomalib was used to benchmark CFA, CFlow, DRAEM, GANomaly, PADIM, PatchCore, RD4AD, and STFPM; code, metric functions, and pretrained checkpoints are also provided.The codebase supports training and inference for the remaining algorithms.Source: §7 · p. 16
• PaDiM and STFPM were evaluated with ResNet-18 and WideResNet-50 backbones, while PatchCore used both backbones with a 0.1 coreset-sampling ratio and 9 neighbors.PaDiM used 100 features with ResNet-18 and 550 with WideResNet-50; STFPM used SGD with learning rate 0.4.Source: §7 · p. 16
• Embedding-based models detect anomalies in feature space, whereas reconstruction-based methods compare original and reconstructed images at pixel level.Source: Figure 10 · p. 17
• RD4AD used WideResNet-50, while DRAEM used a U-Net and CFlow used WideResNet-50 across benchmarks.RD4AD used an Adam learning rate of 0.005; DRAEM and CFlow used 1e-4.Source: §7 · pp. 17–18
• GANomaly was trained with Adam at learning rate 2e-4, using loss weights of 1 for adversarial loss, 50 for image regeneration loss, and 1 for latent-vector encoder loss.f-AnoGAN requires two-stage training and uses Adam with batch size 32 and learning rate 2e-4.Source: §7 · p. 18
• Table 3 reports DICE scores for the supported methods at threshold 0.5, and its caption notes that adjusting thresholds may yield higher performance.Source: Table 3 · p. 18

8.1. AUROC

• AUROC measures the trade-off between true- and false-positive rates across thresholds; pixel-level evaluation thresholds anomaly maps, while image-level evaluation thresholds sample scores.Both procedures record TPR–FPR pairs to form a ROC curve and calculate its area.Source: §8.1 · pp. 18–19

8.2. Per-Region Overlap (PRO)

• PRO evaluates fine-grained detection by averaging, over ground-truth components, the fraction of each component identified as anomalous.The metric uses the number of ground-truth components in the test set for normalization.Source: §8.2 · p. 19

8.3. DICE score

• Dice measures pixel-level overlap but depends on the threshold, so the authors omit Dice comparisons from the main experiments and provide threshold-0.5 scores for reference.The authors note that adjusting thresholds for each result can yield higher performance.Source: §8.3 · p. 19