PaperlayerView exact revision on arXiv

Source-linked AI summary

A Multi-Source Ultrasound Benchmark Revealing the Limits of Contemporary Self-Supervised Anomaly Detection Methods

Marco Riedenauer, Daniel Kienzle, Pratik Mayekar, Rainer Lienhart

arXiv:2610.09677v1cs.CVcs.LG

TL;DR

Ultrasound anomaly detection is difficult to evaluate broadly because prior settings are often anatomy-specific and ultrasound appearance varies across sources. SADUSI is a multi-source, single-image benchmark for visible structural anomalies; current methods struggle, and even the strongest PatchCore variants have limited pixel-level delineation.

  • Problem

    Most existing ultrasound anomaly-detection evaluations are anatomy- or task-specific, and no large-scale public dataset is designed for self-supervised evaluation across multiple ultrasound domains.

    Source: §I · p. 1

  • Method

    SADUSI combines public ultrasound sources across anatomical regions and acquisition settings to train on heterogeneous normal appearance and evaluate visible structural anomalies in single images.

    Source: §I · p. 2, A. Task Definition and Scope · p. 2

  • Results

    PatchCore variants perform strongest, with best AUROC values of 0.82 on TN3K and BUSI and 0.83 on AUL, while corresponding F1max values are 0.40, 0.33, and 0.18.

    Source: §V · p. 4, §VI · p. 5

  • Takeaways & Limitations

    Broad multi-source ultrasound anomaly detection remains challenging, and the findings motivate methods that model broad normality while retaining sensitivity to localized structural abnormalities.

    Source: §VII · p. 6

    SADUSI has incomplete anatomical coverage, uneven source sizes, heterogeneous annotation quality, and limited separability of normal tissue and visible pathology in the analyzed patch-feature space.

    Source: §VI · p. 6

Abstract

from arXiv · show

Self-supervised anomaly detection is a promising paradigm for medical ultrasound, as normal images are often easier to obtain than exhaustive annotations of all possible pathologies. However, most existing evaluations are limited to a single anatomy or task, making it unclear whether models learn a robust notion of normal ultrasound appearance or only a source-specific representation. We introduce the SADUSI benchmark, a multi-source ultrasound dataset designed to train and evaluate anomaly detection methods across a broad range of anatomical regions, views, and acquisition protocols. The goal of SADUSI is to provide a diverse normal ultrasound distribution and a benchmark for visible structural anomalies that can be assessed from single images. We evaluate representative self-supervised anomaly detection methods and find that current approaches struggle in this setting. In particular, reconstruction-based diffusion methods such as AnoDDPM and DeCo-Diff achieve pixel-level AUROC values of 0.56-0.72 and maximum F1 scores of 0.10-0.26, indicating limited separation of pathology from normal image regions. Feature-based PatchCore variants perform better, reaching pixel-level AUROC values of 0.76-0.83, but remain limited with maximum F1 scores of 0.14-0.40. These findings suggest that broad multi-source ultrasound anomaly detection remains an open challenge and that SADUSI can serve as a resource for developing methods that generalize beyond anatomy-specific settings.

I. INTRODUCTION

The paper addresses whether ultrasound anomaly detection can generalize beyond anatomy-specific settings, where normal appearance varies across sources and visible pathology must be distinguished from that variation. It introduces SADUSI and benchmarks representative methods, finding that source-domain variation can dominate pathology-related variation.

I. INTRODUCTION

  • Pathologies are diverse and often rare, pixel-level annotation requires clinical expertise, and ultrasound appearance varies across anatomy, view, device, operator, and protocol.Self-supervised and unsupervised methods learn from visually normal data and identify deviations at inference time.Source: §I · p. 1
  • Existing ultrasound anomaly detection evaluations are mostly anatomy- or task-specific, leaving unclear whether models learn normality that transfers across regions.Broadly trained models must distinguish visible pathological structures from the substantial variability of normal anatomy.Source: §I · p. 1
  • The benchmark focuses on visible structural anomalies in single images, not every pathology or abnormality requiring temporal, functional, or broader clinical context.It evaluates anomalies where suitable masks are available.Source: §I · p. 1
  • SADUSI combines public ultrasound sources across anatomical regions, acquisition protocols, and image formats for single-frame visible structural anomaly detection.The benchmark is intended to support learning heterogeneous normal ultrasound appearance and evaluating visible structural anomalies where suitable masks are available.Source: §I · pp. 1–2
  • The paper contributes a broad multi-source dataset and a benchmark of representative reconstruction, feature-memory, and student-teacher methods.The stated contributions also include showing that source-domain variation can dominate pathology-related variation.Source: §I · p. 2
  • Source-domain variation can dominate pathology-related variation, limiting current anomaly detection methods in this setting.The benchmark finds that reconstruction-based diffusion methods have limited pathology localization, while feature-based methods perform better but remain limited.Source: §I · p. 2

II. RELATED WORK

Prior ultrasound datasets support supervised tasks but are generally released and evaluated in isolation, while anomaly-detection methods have mostly been assessed on industrial data or narrow medical settings. SADUSI addresses this gap with a multi-source benchmark for visible structural anomaly localization.

II. RELATED WORK

  • Existing public ultrasound datasets serve segmentation, classification, measurement, or disease-specific prediction, but are usually evaluated in isolation.They do not directly provide a unified benchmark for heterogeneous normal appearance and visible anomalies across anatomical domains.Source: §II · p. 2
  • Self-supervised learning can exploit unlabeled or weakly labeled medical image collections, while cited medical pretraining work still addresses supervised, disease-specific tasks.The related work distinguishes those tasks from broad anomaly detection trained on normal data.Source: §II · p. 2
  • Anomaly detection methods span one-class, probabilistic, reconstruction-based, and representation-based approaches, with PatchCore, EfficientAD, AnoDDPM, and DeCo-Diff among examples.These methods are commonly evaluated on industrial data or comparatively narrow medical settings.Source: §II · p. 2
  • SADUSI combines diverse ultrasound sources into a multi-source benchmark for visible structural anomaly localization.Source: §II · p. 2

III. DATASET

SADUSI defines its target as single-image detection of visible structural anomalies and represents each sample with an image-level label, an optional pixel mask, and source metadata. Samples without masks may support image-level detection but are excluded from the segmentation benchmark.

A. Task Definition and Scope

  • SADUSI treats an anomaly as a pathological finding visible in a single ultrasound image, excluding abnormalities requiring temporal, functional, or broader clinical context.For example, a cardiac rhythm disorder is considered normal when its corresponding frame shows no visible structural abnormality.Source: A. Task Definition and Scope · p. 2
  • Each dataset entry contains an ultrasound image, a binary structural label, an optional pixel mask, and metadata such as anatomy, view, protocol, and source dataset.The normal/anomalous label reflects available source labels and metadata; anomalous examples are visible structural findings such as nodules, tumors, or deposits.Source: A. Task Definition and Scope · p. 2
  • The dataset separates samples into visually normal and visible-anomaly subsets using their image-level labels.Source: A. Task Definition and Scope · p. 2
  • Datasets without pixel-level annotations can support image-level detection but are not used in the segmentation benchmark.Their pixel mask is undefined, so they cannot provide localization ground truth.Source: A. Task Definition and Scope · p. 2

B. Source Pool and Harmonization

SADUSI combines ultrasound sources across anatomy and acquisition settings to create a heterogeneous image pool for studying normal appearance beyond narrow-domain benchmarks. The pool is converted to 2D images and harmonized to reduce source-specific visual artifacts.

B. Source Pool and Harmonization

  • SADUSI combines source datasets to expose models to variation in anatomy, view, protocol, device, operator appearance, resolution, field of view, and image quality.Its intended use is to support learning and evaluation beyond anatomy-specific benchmarks.Source: B. Source Pool and Harmonization · pp. 2–3
  • Videos, 3D sources, and cine clips are converted by extracting every fifth frame, while still-image datasets are imported directly.The sampling is intended to reduce temporal redundancy while preserving representative variation in probe motion, anatomy, and appearance.Source: B. Source Pool and Harmonization · p. 3
  • Images are manually harmonized by cropping black borders where possible and removing visible interfaces, cursors, overlays, and other non-anatomical markings.Source metadata and labels are preserved and linked to extracted images.Source: B. Source Pool and Harmonization · p. 3

C. Filtered Normal Subset

SADUSI defines a conservative normal training subset using sources whose documentation or labels explicitly support the absence of visible structural anomalies. The benchmark’s current masked evaluation covers selected visible findings from liver, breast, and thyroid ultrasound.

C. Filtered Normal Subset

  • Public ultrasound sources may contain implicit anomalies, so using them as normal training data could teach models to treat evaluated pathologies as normal.The risk arises because source documentation for segmentation, plane-classification, and measurement datasets may not establish that visible pathologies are absent.Source: C. Filtered Normal Subset · p. 3
  • The filtered subset Df includes only normal training sources whose labels, metadata, or descriptions explicitly support the absence of visible structural anomalies.Normality remains operational: the filtering cannot guarantee that every image is clinically healthy because pathologies may be missed in assessment or annotation.Source: C. Filtered Normal Subset · p. 3
  • Table I documents the converted source pool and each source’s contribution to the filtered normal subset.Its caption identifies it as the source-datasets table; no source-level values are given in the supplied passage.Source: C. Filtered Normal Subset · p. 3, Table I · p. 3
  • The filtered normal pool includes fetal, thyroid and neck, liver, abdominal, breast, and lung images; masked anomalous evaluation currently covers liver, breast, and thyroid findings.Examples include focal liver masses, breast pathologies, and thyroid nodules; datasets without masks are excluded from the current pixel-level evaluation.Source: C. Filtered Normal Subset · p. 3

IV. BENCHMARK SETUP

The benchmark tests representative self-supervised methods for pixel-level localization of visible structural anomalies in held-out images. It compares training on the full and conservatively filtered normal pools using the same masked anomalous evaluation set.

IV. BENCHMARK SETUP

  • The benchmark compares diffusion reconstruction methods, feature-embedding PatchCore, and student-teacher EfficientAD for pixel-level anomaly segmentation.PatchCore uses WideResNet-50 and DINOv3 feature-extractor variants; all methods produce pixel-level anomaly maps.Source: §IV · p. 3

A. Training Protocol

  • Methods are trained self-supervised on either the full normal pool Dn or the filtered pool Df, then evaluated on the same masked anomalous set Ga.Using the same evaluation set isolates the effect of training-pool composition.Source: A. Training Protocol · pp. 3–4
  • Runs use seed 1000 without validation; AnoDDPM, DeCo-Diff, and EfficientAD train for 300k, 11,341, and 70k steps, respectively.PatchCore retains 10% of training patch features after coreset subsampling; the augmented DINOv3 variant tests whether mild acquisition-like variation improves localization.Source: A. Training Protocol · p. 4

B. Preprocessing

Preprocessing standardizes image and mask dimensions while preserving image aspect ratio, and keeps method-specific input normalization consistent with original setups. Pixel-level localization is assessed with ranking and threshold-dependent metrics, with Table II reporting results.

B. Preprocessing

  • Images and masks are padded to 256×256 after aspect-ratio-preserving resizing, with nearest-neighbor interpolation applied to masks.This avoids distorting anatomical structures.Source: B. Preprocessing · p. 4
  • Method-specific preprocessing follows original setups; grayscale inputs are triplicated for RGB methods, and PatchCore uses ImageNet-normalized inputs.Diffusion methods use their stated intensity scaling, and no extra augmentation is applied except in the explicit PatchCore-DINOv3 augmentation experiment.Source: B. Preprocessing · p. 4

C. Evaluation Metrics

  • Pixel-level AUROC, AP, AUPRO, maximum F1, precision, and recall compare each predicted anomaly map with its binary ground-truth mask.AUPRO summarizes overlap with connected ground-truth anomaly regions across thresholds.Source: C. Evaluation Metrics · p. 4
  • F1max, precision, and recall use the best threshold-swept operating point, while AUROC, AP, and AUPRO assess ranking without fixing a threshold.Table II reports pixel-level anomaly-localization results, and “(A)” denotes data augmentation.Source: C. Evaluation Metrics · p. 4, Table II · p. 4

V. EXPERIMENTS AND RESULTS

PatchCore variants lead pixel-level localization, but low F1max values show that delineation remains difficult. Anomaly-map contrast varies by method and dataset, and filtering is not uniformly beneficial.

V. EXPERIMENTS AND RESULTS

  • Table II reports pixel-level anomaly-localization results for thyroid nodule, breast, and liver ultrasound subsets with segmentation masks.Source: §V · p. 4
  • AUROC reaches 0.82 on TN3K and BUSI and 0.83 on AUL, while F1max is 0.40, 0.33, and 0.18, respectively, leaving pixel-level delineation difficult.PatchCore variants achieve the strongest localization results; diffusion methods generally have lower AUROC and F1max, and EfficientAD is close to or below random ranking on several evaluations.Source: §V · p. 4

A. Anomaly-Map Contrast Analysis

  • Anomaly-map contrast compares scores inside the annotated abnormality with scores in the remaining image background; positive Δ_i means the annotated pathology scores higher.The image domain is denoted by Ω_i.Source: A. Anomaly-Map Contrast Analysis · pp. 4–5
  • Contrast values are robustly normalized separately for each model variant and anomaly-map type, using the central 95% of each method’s range as the linear scaling interval and clipping values outside it to 0 or 1.The interval is defined by the 0.025 and 0.975 quantiles.Source: A. Anomaly-Map Contrast Analysis · p. 5
  • On BUSI, PatchCore-DINOv3 and PatchCore-WRN show relatively high scaled contrasts; DINOv3 also performs strongly on TN3K, while AUL patterns differ by model and run.AnoDDPM changes little between training pools, DeCo-Diff on the full pool has particularly narrow low-valued distributions, and filtering effects are model- and dataset-dependent; independently scaled distributions do not support absolute raw-contrast comparisons between methods.Source: A. Anomaly-Map Contrast Analysis · p. 5

B. Feature-Space Analysis

Patch-token analyses show that visible pathology is not a consistent out-of-distribution signal across datasets. The small annotated regions and substantial source variation complicate localized pathology detection.

B. Feature-Space Analysis

  • The analysis uses pretrained DINOv3 ViT-B/16 patch tokens, individually L2-normalized, with masks pooled to a 16 × 16 grid and patches labeled anomalous above a 0.5 pathology fraction.Images and masks are resized with aspect-ratio preservation and zero padding; normal patches can come from normal images or non-pathological regions of anomalous images.Source: B. Feature-Space Analysis · p. 5
  • The non-training analysis samples balanced normal–normal and normal–anomalous pairs within and across datasets, then queries anomalous patches against normal patches from other images using rank-1 and k = 10 neighborhoods.Excluding the query image prevents spatially adjacent regions of the same pathology from dominating.Source: B. Feature-Space Analysis · p. 5
  • Cross-dataset normal–normal distances exceed normal–anomalous distances: means are 0.805 versus 0.770 for the filtered pool and 0.802 versus 0.761 for the full pool.The same ordering appears within datasets, with means of 0.738 versus 0.710 for the filtered pool and 0.728 versus 0.717 for the full pool.Source: B. Feature-Space Analysis · p. 5
  • Mean cross-image rank-1 distances to normal patches are 0.265/0.265 for AUL, 0.367/0.365 for BUSI, and 0.414/0.406 for TN3K in filtered/full pools.Normal-neighbor fractions for k = 10 are 0.709/0.722, 0.419/0.461, and 0.415/0.540, respectively; AUL anomalies mostly overlap normal neighborhoods, while BUSI and filtered TN3K more often neighbor anomalous patches.Source: B. Feature-Space Analysis · p. 5
  • Median annotated pathology areas are only 2.4% for AUL, 6.0% for BUSI, and 7.5% for TN3K, while cross-dataset distances indicate substantial source-domain structure.Pathology-related neighborhoods exist for some anatomies, but visible pathology is not a consistent out-of-distribution signal across datasets in patch-token space.Source: B. Feature-Space Analysis · p. 5

VI. DISCUSSION

PatchCore variants perform strongest in broad multi-source ultrasound anomaly detection, but localization remains limited. The discussion attributes the difficulty to overlap between normal variation and pathology and frames SADUSI as a stress test for generalization.

VI. DISCUSSION

  • PatchCore variants reach AUROC values up to 0.82, 0.82, and 0.83 on TN3K, BUSI, and AUL, while F1max remains 0.40, 0.33, and 0.18.Filtering sources with insufficiently documented pathology status improves dataset semantics but does not make the task uniformly easy.Source: §VI · p. 5
  • AnoDDPM and DeCo-Diff show limited localization, possibly because heterogeneous training encourages reconstruction of plausible anomalous structures alongside normal anatomy.Source: §VI · p. 5
  • PatchCore outperforms reconstruction-based methods; DINOv3 has the highest AUROC on TN3K and AUL, while WRN has the strongest AUPRO on TN3K and BUSI.Source: §VI · p. 5
  • Augmentation improves region overlap in several cases but yields no consistent AUROC or F1max gains, and feature-space analysis finds no consistent pathology outliers.Normal structures and pathology can share local texture, speckle statistics, and boundary appearance; feature geometry also reflects source domain.Source: §VI · p. 6
  • SADUSI is a stress test for generalization, with limitations including incomplete anatomical coverage, uneven source sizes, heterogeneous annotation quality, and limited feature-space separability.Source: §VI · p. 6

VII. CONCLUSION

SADUSI provides a multi-source ultrasound dataset for evaluating self-supervised anomaly detection across anatomical regions and acquisition settings. Experiments find that current methods struggle to separate visible structural anomalies from normal ultrasound variation, motivating methods that model broad normality while retaining sensitivity to localized findings.

VII. CONCLUSION

  • Current methods struggle to separate visible structural anomalies from heterogeneous normal ultrasound; feature-memory methods outperform diffusion methods, but PatchCore localization remains limited.Source: §VII · p. 6
  • SADUSI is a multi-source ultrasound dataset for self-supervised anomaly detection across diverse anatomical regions and acquisition settings.Source: §VII · p. 6
  • Patchwise analysis suggests visible pathology is often not cleanly separated from normal tissue and source-domain variation in pretrained representations.Source: §VII · p. 6
Loading 2610.09677v1…