PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing Patient Privacy in Medical Generative Models: Scalable Memorization Detection with DeepSSIM++

Antonio Scardace, Francesco Guarnera, Sebastiano Battiato, Daniele Ravì

arXiv:2609.03615v1cs.CV

TL;DR

Patient privacy is threatened when medical generative models reproduce training images, while existing similarity metrics struggle with anatomical sensitivity, perturbations, and scale. DeepSSIM++ learns a self-supervised structural embedding whose cosine similarity approximates SSIM and supports threshold-based memorization detection. It outperforms evaluated baselines under realistic perturbations and computes similarities several orders of magnitude faster than analytical SSIM, though its thresholds are calibrated to brain MRI data.

  • Problem

    Medical generative models may reproduce training samples, while pixel metrics are sensitive to artifacts and generic embeddings may miss anatomical similarity needed for privacy auditing.

  • Method

    DeepSSIM++ learns a self-supervised embedding space where cosine similarity approximates SSIM, then classifies image pairs by similarity thresholds.

  • Results

    DeepSSIM++ substantially outperforms SSIM and evaluated learning-based baselines under realistic spatial and intensity perturbations while achieving several orders-of-magnitude faster computation than pixel-space SSIM.

  • Takeaways & Limitations

    DeepSSIM++ supports scalable memorization auditing when perfect pixel-level alignment between real and synthetic medical images cannot be assumed.

  • Takeaways & Limitations

    Decision thresholds are calibrated on the brain MRI dataset, and the mapping from continuous similarity scores to memorization categories depends on data distribution.

Abstract

from arXiv · show

While deep generative models offer new opportunities for medical image synthesis and data sharing, their ability to memorize and reproduce training samples raises serious concerns about patient confidentiality. Detecting such memorization at scale remains challenging: traditional pixel-based metrics are sensitive to generation artifacts, whereas generic embedding-based metrics often lack the anatomical sensitivity required for medical data. To address this challenge, we introduce DeepSSIM++, a self-supervised similarity metric for scalable memorization auditing in medical generative models. By leveraging multi-scale feature aggregation and anatomy-preserving augmentations, DeepSSIM++ learns an embedding space where cosine similarity approximates the Structural Similarity Index (SSIM), eliminating the need for exact pixel-level registration. Compared with state-of-the-art baselines, DeepSSIM++ achieves an average Macro F1 improvement of 33 percentage points under ideal alignment and 46 percentage points under realistic spatial and intensity perturbations. Furthermore, it accelerates large-scale similarity computation by several orders of magnitude compared with analytical SSIM. By combining anatomical sensitivity and computational efficiency, DeepSSIM++ provides an open-source tool for scalable memorization auditing in medical generative AI. Code and data are publicly available at: https://github.com/brAIn-science/DeepSSIM.

1. Introduction

Medical generative models can compromise patient privacy when they reproduce training images, but existing memorization metrics struggle to capture anatomical similarity robustly and efficiently. DeepSSIM++ addresses these gaps through architectural and optimization improvements, with reported gains over existing metrics and faster similarity computation.

  • Motivation: Synthetic data does not guarantee privacy when generative models memorize and reproduce individual training examples.This risk is especially serious in healthcare applications involving patient-specific information.
  • Problem: Medical memorization assessment requires identity-preserving anatomical similarity rather than generic visual similarity alone.The paper extends a medical-imaging definition of memorization to include realistic intensity and noise perturbations while preserving subject-specific anatomy.
  • Approach: The proposed framework targets transformation-invariant structural representations because pixel metrics are misalignment-sensitive and generic embeddings may miss subtle anatomical correspondence.The empirical evaluation focuses on 2D structural brain MRI, where anatomical morphology can support patient re-identification.
  • Contributions: DeepSSIM++ enhances the underlying metric with multi-scale feature aggregation and Layer-wise Learning Rate Decay to improve similarity estimation accuracy.The paper also reports extensive ablations and LayerCAM-based explainability analysis.
  • Contributions: DeepSSIM++ consistently outperforms existing memorization metrics and accelerates similarity computation by orders of magnitude compared with analytical SSIM.The authors position these gains as enabling scalable auditing on large synthetic datasets.

2. Related Work

Prior work links memorization to data, model, and optimization factors and uses similarity-based detection strategies, but generic representations and pixel comparisons remain poorly suited to subtle anatomical duplication. DeepSSIM++ is motivated as a structural, transformation-robust, and scalable alternative.

  • Memorization Factors: Memorization arises from interactions among training-data properties, model design, and optimization rather than from a generative architecture alone.Smaller or less diverse datasets, higher model capacity, and longer training can each increase memorization tendencies.
  • Privacy Risks: Memorization can leak training information through direct reproduction or sensitive learned patterns, sometimes becoming visible only through dedicated attacks.The literature describes membership inference, model inversion, and data extraction attacks, alongside privacy-preserving training strategies.
  • Detection Strategies: Existing memorization studies commonly measure image similarity first and then convert scores into decisions using thresholds or ranking criteria.Reported approaches include embedding-based metrics, average memorization scores, unique memorization scores, and semantic deduplication.
  • Embedding-Based Metrics: Generic embedding metrics such as SSCD may overlook fine-grained anatomical structures because they were developed primarily for natural images.Domain-specific learned representations reduce reliance on external features but may still miss strict anatomical duplication.
  • Similarity Metrics: Medical memorization detection must balance structural sensitivity with robustness to spatial misalignment and computational scalability.Pixel metrics preserve structural correspondence but fail under affine transformations, while SSIM becomes prohibitive for exhaustive large-scale comparisons.
  • Motivation: DeepSSIM++ combines structural sensitivity with efficiency, scalability, and transformation robustness for large-scale generative-model evaluation.This design responds to limitations identified in pixel-based and generic embedding-based approaches.

3. Methods

The proposed pipeline learns an embedding space in which cosine similarity approximates SSIM, then thresholds embedding similarities to classify real–synthetic pairs by memorization level.

  • DeepSSIM++ Pipeline: DeepSSIM++ trains a self-supervised feature extractor so embedding cosine similarity approximates SSIM computed on aligned image pairs.The learned representation is intended to preserve structural similarity without requiring exact alignment during comparison.
  • DeepSSIM++ Pipeline: A thresholding strategy classifies image pairs as Unrelated, Near-Duplicates, or Exact Duplicates based on their embedding similarities.The method therefore separates representation learning from the final memorization decision.

3.1. Datasets

The study builds real, synthetic, and paired MRI datasets to train, select, and evaluate DeepSSIM++, using balanced SSIM-based splits and an independently annotated hybrid test set. The dataset design separates model development from final memorization assessment.

  • Dataset Construction: The framework uses three datasets: real MRI scans for training an LDM, synthetic MRI slices generated by that LDM, and real–synthetic pairs for DeepSSIM++.The generative architecture serves as a practical case study because detection operates on generated images rather than internal model weights.
  • Real Dataset: The real MRI collection contains 2,583 scans from IXI and CoRR, including 2,024 T1-weighted and 559 T2-weighted acquisitions.Scans are split 85% for LDM training and 15% for validation after standardized preprocessing and central-slice extraction.
  • Synthetic Dataset: The synthetic dataset contains 65,850 MRI images, generated at 30 samples per real training image to increase the likelihood of memorized instances.The construction follows prior work linking substantially larger synthetic sets with memorization-promoting conditions.
  • Real–Synthetic Pair Dataset: Each real training image is paired with every synthetic image, producing 144,540,750 real–synthetic pairs.The pair dataset is divided into training, validation, and test subsets for feature-extractor training, model selection, and final evaluation.
  • Dataset Splits and Annotation: Training and validation pairs are stratified by reference SSIM to balance similarity levels, while the test set uses hybrid candidate selection to avoid SSIM-based evaluation bias.The test set contains 8,780 pairs independently annotated by three experts, with final labels assigned by majority vote.

3.2. Proposed Self-Supervised Similarity Metric

DeepSSIM++ learns a similarity representation that approximates SSIM while remaining robust to anatomical-preserving spatial and appearance variations. Its feature extractor combines multi-stage ConvNeXt representations with multi-scale pooling and fusion.

  • 3.2. Proposed Self-Supervised Similarity Metric: The feature extractor learns embeddings whose similarity score approximates SSIM targets computed from co-registered, z-score-normalized image pairs.Registration is used during training to create reliable supervision, while inference requires no explicit co-registration.
  • 3.2. Proposed Self-Supervised Similarity Metric: The learned representation assigns high similarity to anatomically corresponding images despite spatial misalignment or minor appearance distortions.The training transformations are designed to preserve anatomical identity while simulating inference-time variation.
  • 3.2.1. Training Augmentations: Anatomy-preserving augmentations expose the model to intensity and spatial variations while maintaining anatomical identity.Intensity transformations include bias-field augmentation, Gaussian noise, and contrast adjustment; geometry transformations include rotations, translations, zooming, and flips.
  • 3.2.2. Feature Extractor Architecture: DeepSSIM++ aggregates representations from four ConvNeXt stages rather than relying only on the deepest representation.Each stage is independently pooled with a learned GeM exponent, then concatenated and projected through an MLP fusion head.
  • 3.2.2. Feature Extractor Architecture: Layer-wise learning-rate decay preserves earlier pre-trained representations while allowing the fusion head to adapt more rapidly.Earlier layers receive progressively smaller learning rates than later layers.

3.3. Proposed Memorization Metric

The proposed memorization metric converts learned structural similarity into three mutually exclusive anatomical correspondence categories. Thresholds separate unrelated pairs from near-duplicates and exact duplicates.

  • 3.3. Proposed Memorization Metric: The thresholding function classifies image pairs as Unrelated, Near-Duplicate, or Exact Duplicate according to learned similarity.The thresholds α and β are pipeline hyperparameters with α < β.
  • 3.3. Proposed Memorization Metric: Exact Duplicates preserve every anatomical structure without structural deviation, while anatomy-independent artifacts such as uniform blur are tolerated.Corresponding structures must retain their position, shape, size, and boundary.
  • 3.3. Proposed Memorization Metric: Near-Duplicates retain recognizable overall anatomical correspondence but contain localized deviations in specific regions.Deviations may involve boundaries, shapes, or tissue composition without eliminating anatomical identifiability.
  • 3.3. Proposed Memorization Metric: Unrelated pairs lack consistent anatomical structure and meaningful correspondence between the real and generated images.The categories describe distinct levels of anatomical correspondence.

3.4. Implementation Details

The implementation uses a pre-trained ConvNeXt-T embedding model in PyTorch and MONAI, with specified optimization, threshold-selection, and case-study settings. These details define the training configuration and deployment of the memorization pipeline.

  • 3.4. Implementation Details: The implementation uses PyTorch and MONAI with a ConvNeXt-T backbone pre-trained on ImageNet and its classification layer replaced by an identity mapping.The replacement enables direct image-embedding extraction while preserving low-level spatial features for neuroimaging adaptation.
  • 3.4. Implementation Details: The network is trained for 65 epochs with AdamW, batch size 64, and a 256-dimensional embedding head.The base learning rate is 1 × 10−3, LLRD uses γ = 0.3, and the projection head uses dropout probability 0.1.
  • 3.4. Implementation Details: The decision thresholds are selected by grid search over [0, 1] with step size 0.01, yielding α = 0.76 and β = 0.92.The search enforces α < β.
  • 3.4. Implementation Details: The latent diffusion model case study follows Pinaya et al.’s protocol adapted for generating 2D axial brain MRI slices.The implementation is evaluated in the 2D axial brain-MRI setting.
  • 3.4. Implementation Details: Experiments run on a workstation with an Intel Core i7-14700 CPU, 32 GB RAM, and one NVIDIA RTX 4070 Ti SUPER GPU.The GPU provides 16 GB of VRAM.

4. Experiments and Results

The experiments sequentially refine DeepSSIM++ through architectural, augmentation, training, optimization, loss, and embedding choices, then evaluate the selected configuration on memorization detection. The ablations identify multi-scale features, partial fine-tuning, LLRD, MSE, and a 256-dimensional embedding as effective design choices.

  • Ablation Study: The evaluation uses validation-set MAE and RMSE against reference SSIM values and reports test-set memorization metrics for registered and randomly perturbed pairs.Table 1 summarizes the sequential ablation protocol, while Table 2 covers precision, recall, macro F1, TPR@5%FPR, and Silhouette score.
  • Ablation Study: Multi-scale feature aggregation reduces MAE by 20.13% and RMSE by 17.41% compared with the single-scale baseline.The architecture aggregates hierarchical spatial representations at different granularities.
  • Ablation Study: Partial fine-tuning provides the best training strategy, reducing MAE by 17.77% and RMSE by 17.41% versus head-only training.Full fine-tuning and training from scratch perform worse than partial fine-tuning.
  • Ablation Study: LLRD reduces MAE by 9.70% and RMSE by 2.71% relative to uniform learning rates.LLRD assigns smaller learning rates to earlier backbone layers and larger rates to the prediction head.
  • Ablation Study: ConvNeXt-T achieves the best overall ablation performance, reducing MAE by 29.04% and RMSE by 22.52% versus the best ResNet50 configuration.It also outperforms the best DenseNet161 configuration while using 28M rather than 88M parameters.
  • Ablation Study: The selected configuration benefits from enriched augmentations, MSE supervision, hyperparameter tuning, and a 256-dimensional embedding.These choices reduce MAE by 4.64% for augmentations, 2.69% for MSE versus MAE, and 17.23% through hyperparameter optimization; 256 dimensions perform best overall.

4.2. Annotation Reliability

The benchmark uses expert annotations and evaluates three-way memorization detection across ideal and perturbed image-pair settings. DeepSSIM++ improves class separation and classification performance, especially under realistic perturbations.

  • Quantitative Comparison against Baselines: All DeepSSIM++ improvements over competing methods reached statistical significance using McNemar’s test with p<0.05.Evaluation reports precision, recall, macro F1, TPR@5%FPR, and Silhouette score across three similarity categories.
  • Quantitative Comparison against Baselines: The evaluation classifies pairs as Unrelated, Near-Duplicate, or Exact Duplicate under ideal registration and anatomy-preserving spatial and intensity perturbations.Perturbations include flips, subtle rotations, and contrast variations.
  • Baselines: Under perfect registration, DeepSSIM++ produces better-separated similarity distributions than baseline methods, consistent with its higher Silhouette score.The distributions are stratified by ground-truth category and paired with method-specific decision thresholds.
  • Performance on Registered Pairs: 84.81% macro F1 under ideal registration exceeded UMS (l2) at 81.55% and original DeepSSIM at 68.33%.DeepSSIM++ also reached a Silhouette score of 0.39 and TPR@5%FPR of 94.29%.
  • Performance on Perturbed Pairs: 81.39% macro F1 under spatial and intensity perturbations exceeded UMS (SSCD) at 41.77%, a relative improvement of 95%.DeepSSIM++ achieved the highest perturbed-pair Silhouette score, 0.36, versus 0.06 for SSIM.

4.4. Qualitative Analysis

LayerCAM explanations show that DeepSSIM++ bases similarity predictions on anatomically relevant brain structures rather than background regions. This pattern holds for both Exact Duplicate and Near-Duplicate examples.

  • Qualitative Explainability Analysis: LayerCAM maps consistently emphasize lateral ventricles, periventricular white matter, and cortical gyri across both similarity categories.The figure includes three Exact Duplicate examples and three Near-Duplicate examples.
  • Qualitative Explainability Analysis: Negligible background activation indicates that the highlighted evidence is concentrated in anatomically meaningful regions.LayerCAM reflects regions driving the predicted similarity score, not image agreement or discrepancy.

4.5. Threshold Sensitivity Analysis

Threshold sensitivity analysis finds gradual performance degradation as Gaussian noise increases, without abrupt collapse. DeepSSIM++ remains above embedding-based baselines at the largest tested perturbations.

  • Threshold Sensitivity Analysis: The perturbed thresholds are formed by adding independent zero-mean Gaussian noise to optimal values α=0.76 and β=0.92.Invalid perturbed thresholds violating α̃<β̃ or the [0,1] range are discarded.
  • Threshold Sensitivity Analysis: Macro F1 declined gradually from 0.85 without threshold noise to 0.71 when σα=σβ=0.05.Noise levels ranged from 0.00 to 0.05 in increments of 0.01.
  • Threshold Sensitivity Analysis: Performance showed no abrupt collapse anywhere across the tested joint perturbation grid.Even under the largest perturbations, performance remained above all embedding-based baselines reported in Table 2.
  • Computational Efficiency: At approximately 2 × 10^8 comparisons, single-process SSIM requires nearly 48 hours, whereas DeepSSIM++ is evaluated in seconds-scale configurations.DeepSSIM++ runtimes distinguish full computation with embedding extraction from similarity computation using pre-computed embeddings.

5. Discussion and Conclusion

DeepSSIM++ combines robust medical-image memorization detection, anatomical interpretability, and large-scale computational efficiency. Its thresholds remain dataset-dependent and may require recalibration when applied to new data distributions.

  • Discussion and Conclusion: DeepSSIM++ remains comparable to SSIM under ideal registration but outperforms SSIM and learning-based baselines under realistic spatial and intensity perturbations.Its computational advantage supports analysis involving hundreds of millions of image pairs.
  • Discussion and Conclusion: LayerCAM indicates that predictions rely on anatomically meaningful structures rather than spurious visual patterns.This supports using DeepSSIM++ as an interpretable auditing tool.
  • Discussion and Conclusion: Decision thresholds are calibrated on brain MRI data, so new datasets may require recalibration because score-to-category mappings depend on data distributions.The paper notes that similarity categories may overlap or form multiple modes, limiting guarantees from adaptive thresholding.
  • Discussion and Conclusion: DeepSSIM++ could screen synthetic datasets for near- or exact-duplicates before public release to reduce unintended training-data disclosure risk.The paper also proposes integrating the metric into generation pipelines.

A. Results Using Original Threshold Values

Using thresholds reported by prior publications, baseline performance degrades substantially, while DeepSSIM++ retains an advantage on threshold-independent metrics.

  • A. Results Using Original Threshold Values: Macro F1 falls below 3.3% for UMS and from 68.33% to 41.32% for original DeepSSIM on registered pairs, while perturbation performance drops from 67.16% to 40.48%.The original methods use overly aggressive Exact Duplicate classification under their published thresholds.
  • A. Results Using Original Threshold Values: DeepSSIM++ remains more discriminative than UMS on threshold-independent metrics, with Silhouette score 0.39 versus 0.23 and TPR@5%FPR 94.29% versus 90.65%.These results indicate that its advantage is not attributable to threshold calibration alone.
Loading 2609.03615v1…