PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated screening of privacy risks in medical AI

Konstantinos Vrettos, Eleftherios Tzanis, Haridimos Kondylakis, Evangelia E. Vassalou, Florence X. Doo, Michail E. Klontzas

arXiv:2610.30229v1cs.LGcs.AIcs.CR

TL;DR

Membership-inference risk can remain high for individual patients even when cohort-average attack performance appears near chance. Across clinical datasets and model configurations, the paper finds that high-risk patient rankings largely persist and develops a lightweight screen to identify them before production-model training.

  • Problem

    Cohort-average membership-inference performance can conceal substantial individual risk, while it remains an open question whether inexpensive screening can consistently identify the same vulnerable patients across clinical training configurations.

    Source: Main · pp. 2–3

  • Method

    The study audits patient-level risk across clinical datasets and builds a lightweight screen using four small proxy models to estimate atypicality before production-model training.

    Source: Cohort average risks conceal… · p. 4, Main · p. 4, Study design · p. 25

  • Results

    Across patient-identified cohorts, 49–95% of patients exceeded the risk threshold, while training choices changed the number at risk but largely preserved the highest-risk patients.

    Source: Discussion · pp. 22–23

  • Takeaways & Limitations

    Privacy screening can be performed before production-model training, and the authors propose it as a routine step in preparing medical datasets.

    Source: Discussion · pp. 22, 25

    All datasets were public research collections, so performance in independent hospital pipelines remains to be established.

    Source: Discussion · p. 24

Abstract

from arXiv · show

Medical artificial intelligence (AI) models are often shared without their training data, assuming this protects the patients who contributed them. However, a model can reveal whether a patient's records were used in training, a privacy breach known as a membership inference attack. This risk is usually reported as a cohort average, which can appear negligible while individual patients are almost certainly exposed. Identifying potentially vulnerable records before developing a production model could make privacy assessment more practical. We measured each patient's vulnerability to membership inference attack, across 7,789 models trained on six clinical datasets in four modalities, including two adapted pretrained medical models. Training choices changed how many patients had high attack scores, but the patients who ranked highest by risk largely stayed the same across the configurations we tested. Stronger weight decay reduced the exposed cohort up to threefold, yet largely the same patients remained most at risk across training recipes, architectures and initializations, including foundation models. Therefore, we developed a lightweight screening tool that uses four small models to identify patients with high risk of exposure before production-model training. We also assessed if flagged patients could be protected without removing their data. Privacy screening should become incorporated into data preparation for clinical AI.

Main

Audits across clinical datasets show substantial patient-level privacy risk concealed by cohort averages, while training choices change risk counts more than patient rankings. The findings motivate screening before production-model training and testing mitigation without removing records.

Cohort average risks conceal patient-level exposure variation

  • Across five patient-identified datasets, 49–95% of patients exceeded the risk threshold, even though pooled population attack AUCs ranged from 0.56 to 0.93.On ISIC-2020, pooled AUC was 0.56 while approximately half of patients exceeded the threshold; the 0.7 cutoff is operational, not a validated privacy-harm boundary.Source: Cohort average risks conceal… · pp. 4–5, Figure 1 · p. 6

Training changes how many patients exceed the risk threshold

  • Weight decay reduced at-risk fractions 1.38–2.95-fold on four datasets, while mixup and quarter-sized training sets raised PneumoniaMNIST exposure by 38% and 33%.Reseeding changed the at-risk count by only 1.07-fold; on NIH-CXR14, weight decay reduced exposed patients from 4,767 to 1,615.Source: Training changes how many patients… · p. 7, Figure 2 · p. 10
  • Higher diagnostic performance did not ensure low attack risk: 45% of patients remained above threshold on improved ISIC-2020, and 90% on MRI-CORE.On PTB-XL, partially fine-tuning ECGFounder reduced the above-threshold fraction relative to full fine-tuning while lowering training energy; energy and diagnostic trade-offs varied across configurations.Source: Training changes how many patients… · pp. 8–9

The data determine who is exposed

  • Across recipes, architectures, initializations, and foundation-model adaptations, training changed how many patients exceeded threshold but largely preserved the highest-risk patient rankings.Normalized ranking stability under weight-decay changes was 0.75–0.98 across five datasets whose arms passed the canary control.Source: The data determine who is exposed · pp. 13–14

Atypicality identifies exposed patients

  • Atypicality separated the top-risk 5% from other patients across all six datasets, while patients aged 80 or older were over-represented above threshold on all four datasets analyzed.Other demographic differences varied by dataset; no evaluated racial or ethnic contrast showed statistically significant over-representation.Source: Atypicality identifies exposed patients · pp. 14–15

Screening patients before production-model training

  • DataConfessor recovered 72.7–100% of reference-audit top-5% patients across five patient-identified datasets, 3.6–5 times random flagging, in 1.4–7.4 GPU-minutes.Transfer to modified recipes recovered 65–100% of the top 5%, with weaker transfer on PTB-XL.Source: Screening patients before… · pp. 16, 19
  • Across four datasets, the best statistical predictor varied from 40–99% recall, whereas the screen achieved at least 73% and had the highest point estimate in each comparison.Label rarity approached the screen on NIH-CXR14 and ISIC-2020 but recovered only 50.0% of the CheXpert tail and 39.6% of the PTB-XL tail.Source: Screening patients before… · pp. 16–17

Testing mitigation while retaining patient records

  • Self-distillation reduced flagged patients’ mean risk scores while retaining their records, but did not establish protection against membership inference at low false-positive rates.Fixed softened labels did not lower risk scores, while class-wise histogram matching lowered them but reduced AUROC to 0.51.Source: Testing mitigation while retaining… · pp. 19–20, Figure 5 · p. 21

Discussion

Across datasets and model families, training choices changed how many patients had high membership-inference scores, while largely preserving which patients ranked highest. The authors connect this stability to data-driven atypicality, discuss mitigation options, and note limits on the findings.

Discussion

  • Across more than 7,700 models, training choices changed how many patients had high attack scores while largely preserving the identity of those who ranked highest.The study covered six datasets and four modalities, from small networks to fine-tuned foundation models.Source: Discussion · p. 22
  • Between 49% and 95% of patients in cohorts with identifiers exceeded the risk threshold, even where population attack AUC was 0.56.The paper notes that cohort averages can conceal substantial patient-level risk.Source: Discussion · p. 22
  • Atypicality can be estimated before production training, unlike early-training predictors that require information from the model being trained.The authors report that a lightweight proxy screen anticipates much of the patient ranking across training choices, including foundation-model adaptations.Source: Discussion · p. 23
  • The authors propose that atypicality reflects how poorly a record can be predicted from other patients, while training affects how strongly this difference appears in the model.They caution that the association between atypicality and audit score does not establish a causal mechanism.Source: Discussion · p. 23
  • Stronger weight decay and partial foundation-model fine-tuning reduced attack risk, while mixup and smaller training sets increased the fraction above threshold.Self-distillation reduced mean attack scores with negligible impact on diagnostic performance; screening should complement, not replace, formal privacy guarantees.Source: Discussion · p. 24
  • Because all datasets were public research collections, performance in independent hospital pipelines remains to be established.The 0.7 threshold is operational rather than a validated boundary for privacy harm, and the screen does not establish that unflagged patients are safe.Source: Discussion · p. 24
  • The authors propose routine privacy screening during medical-dataset preparation, before production-model training.They describe the screening as inexpensive, open-source, and not dependent on the final production model.Source: Discussion · p. 25

Methods

The study compared patient-level membership-inference risk across shadow-model training arms, using random patient halves and seed replicates. It used six clinical datasets, with dataset-specific pool handling and preprocessing.

Study design

  • Each shadow-model arm trained models on random halves of a patient pool, then scored every record; seed replicates retrained the same recipe with a different seed.For NIH-CXR14, 128 reference models compared each patient's records across about 64 member and 64 non-member models; patient risk was the maximum across records.Source: Study design · p. 25

Datasets and pools

  • The study used CheXpert, NIH-CXR14, ISIC-2020, PTB-XL, SCOPE-MRI, and PneumoniaMNIST across radiography, dermoscopy, ECG, and MRI.Each record was one radiograph, dermoscopic image, electrocardiogram, or MRI series.Source: Datasets and pools · p. 26
  • Canary records with randomized labels served as positive controls and were excluded from subsequent analyses; PneumoniaMNIST examples were analyzed individually because patient identifiers were unavailable.SCOPE-MRI localizers and duplicate reconstructions were excluded, and its remaining series were standardized by cropping, resizing, and resampling.Source: Datasets and pools · p. 26

Model training

Reference training recipes varied by dataset, with model architectures, initialization, and training duration chosen for each task. Foundation-model experiments compared fine-tuning strategies and parameter counts across ECGFounder and MRI-CORE.

Model training

  • Reference recipes used weight decay 10⁻⁴ and dataset-specific architectures, including ImageNet-initialized ResNet-18 models and randomly initialized models for PTB-XL and PneumoniaMNIST.ISIC-2020 also used EfficientNet-B3, while SCOPE-MRI compared ResNet-18 and ViT-S/16 slice-based backbones.Source: Model training · pp. 26–27
  • Training modifications were screened on PneumoniaMNIST before weight decay 10⁻² was applied across datasets; PTB-XL instead used group normalization when its weight-decay arm failed the canary control.Shadow-model counts varied by configuration, and utility was compared with a published model when available using a 0.05 AUROC tolerance.Source: Model training · p. 27

Foundation models

  • ECGFounder was fine-tuned on PTB-XL with full, partial, and head-only training, comparing 30.7 million, 3.42 million, and 5,125 trainable parameters, respectively.The study also compared a seed replicate and a full-fine-tuning arm with weight decay 1.0.Source: Foundation models · pp. 27–28
  • MRI-CORE was adapted to SCOPE-MRI using LoRA, linear probing, or full fine-tuning; the linear probe failed the canary control and was not analyzed further.LoRA used rank 8 with 0.44 million trainable parameters; the weight-decay comparison used full fine-tuning because strong decay could switch off the zero-initialized LoRA adapter.Source: Foundation models · p. 28

Membership inference and per-patient risk

The authors define membership-inference risk by comparing model confidence on records from patients included in training with confidence on records from patients not included. Patient risk is the maximum attack AUC across a patient's records, with 0.7 used as the exposure threshold.

Membership inference and per-patient risk

  • Membership-inference attacks use model outputs to infer training-set membership, based on models typically being more confident on records they trained on.For each record, attack AUC is the probability that a random member model is more confident than a random non-member model.Source: Membership inference and per-patient… · p. 28
  • The calibrated excess over null is the fraction of records with AUC above 0.7 minus the fraction expected under the exact null distribution.Without leakage, member and non-member confidences would be interchangeable.Source: Membership inference and per-patient… · p. 29
  • Patient risk is the maximum attack AUC across that patient's records, and scores above 0.7 count as exposed.At this threshold, a model trained on the patient is more confident in 70% of member–non-member pairs; threshold changes affect exposed counts but not rank-based stability or screen recall.Source: Membership inference and per-patient… · p. 29

Validity checks

The analysis excluded arms that failed patient-level membership assignment or minimum model-count checks, and matched arms by model count. Analyses used final checkpoints except for within-tail ordering on SCOPE-MRI.

Validity checks

  • Every included arm required each patient’s records to be wholly in or out of training and at least four member and four non-member models per record.Arms failing either check were excluded from every count.Source: Validity checks · p. 29
  • Canaries passed only when their median AUC was at least 0.65, exceeded ordinary-record AUCs, and shuffled labels reproduced the analytic null.The ordinary-record comparison used a one-sided Mann–Whitney U test with P ≤ 10⁻⁶.Source: Validity checks · p. 29

Matched model counts and checkpoints

  • Arms were truncated to their first 32–128 models for matched-count comparisons, and counts and normalized stability used each recipe’s final checkpoint.SCOPE-MRI used epoch 5 for Cohen’s d and screen recall because many patients had reached maximum AUC by epoch 30.Source: Matched model counts and checkpoints · p. 30

Normalized stability

Normalized stability compares the rank agreement between training arms with the agreement between a reference arm and its seed replicate. Noisy ECGFounder arms were assessed with a noise-corrected correlation instead.

Normalized stability

  • Normalized stability divides the Spearman correlation of per-patient risk between reference and comparison arms by the reference arm’s seed-replicate correlation.The statistic is capped at 1.0 and interpreted only when reference replicate correlation exceeds 0.5.Source: Normalized stability · p. 30
  • The study defined invariant rankings as normalized stability S ≥ 0.8, comparing arms that share the reference seed against independent replicates.Source: Normalized stability · p. 30
  • For ECGFounder partial-fine-tuning and linear-probe arms, noisy rankings prompted reporting a noise-corrected Spearman correlation, with r estimated from a seed replicate.The uncorrected stability measure understates their agreement with the reference.Source: Normalized stability · p. 30

Atypicality and candidate predictors

Atypicality measures held-out model confidence in a patient’s correct labels and was compared with four other predictors of exposure. DataConfessor uses proxy models and atypicality rankings to flag patients before production-model training.

Atypicality and candidate predictors

  • Atypicality is the maximum across a patient’s records of the negated mean non-member-model confidence in each record’s correct label.The study compared it with memorization gap, lowest non-member confidence, patient record count, and held-out disagreement.Source: Atypicality and candidate predictors · p. 30
  • Predictors were evaluated by Cohen’s d between the 5% most exposed patients and the rest, with Benjamini–Hochberg correction across predictors and datasets.Effects were also computed within record-count strata and from a seed-replicate arm to address record-count effects and shared estimation noise.Source: Atypicality and candidate predictors · p. 31

Screening tool

  • DataConfessor trains four one-epoch proxy models on dataset halves and flags the 20% of patients with highest atypicality.If independent proxy-pair rankings correlate below 0.7, the tool increases proxy training epochs; SCOPE-MRI used five epochs.Source: Screening tool · p. 31

Tool’s performance vs statistical baselines

Statistical baselines were scored using the same flagging and recall procedure as the screening tool, while record count served as a baseline floor. The study also describes three strategies for mitigating vulnerability without removing records.

Tool’s performance vs statistical baselines

  • Baselines flagged the top 20% of patients and measured recall in the same top-5% exposure tail as the tool.The study compared 27–29 predictors per dataset across metadata and signal statistics, frozen-model outlier scores, pixel statistics, and unsupervised image models.Source: Tool’s performance vs statistical… · p. 31
  • Record count predicted the exposure tail through order statistics alone and performed worse than atypicality, serving as the baseline floor.Source: Tool’s performance vs statistical… · p. 32

Mitigation of vulnerability

  • Mitigation strategies were self-distillation, softened labels fixed at 0.75 on the true class, and class-wise histogram matching.The primary endpoint was the mean risk change for flagged patients versus the undefended arm; a reduction required a bootstrap interval entirely below zero.Source: Mitigation of vulnerability · p. 32

Demographic analysis

The analysis compared demographic representation among patients above the exposure threshold and in the highest-risk tail. It also describes GPU energy measurement and the statistical procedures used to estimate uncertainty and test differences.

Demographic analysis

  • The analysis compared group representation among exposed patients (risk > 0.7) and the top-5% risk tail, using specified reference groups and omitting groups with fewer than 30 patients.The analysis covered 23 reference arms across four datasets; on SCOPE-MRI, where 95% exceeded the exposure threshold, only the top-5% tail was analyzed.Source: Demographic analysis · pp. 32–33

Energy measurement

  • GPU energy was measured using the board’s built-in energy counter and reported with the CEDARS AI Research Label and the 2026 Greek-grid carbon intensity.The passage cites sources 34, 35 for the label and 36, 37 for the carbon intensity.Source: Energy measurement · p. 33

Statistics

  • Uncertainty was estimated with 2,000 patient-bootstrap resamples, while held-out AUROC differences used Welch’s t-test and multiple comparisons used Benjamini–Hochberg correction.Arms trained on the same pool used the same patient resample; tests for individual figures were specified in their legends.Source: Statistics · p. 33

Computational resources

The study reports its training hardware and compute totals, and notes that local open-source models were used for language polishing and text formatting.

Computational resources

  • The study trained 7,789 models using a 16-GB Quadro RTX 5000 and, for SCOPE-MRI, a 32-GB RTX 5090, consuming 1,420 GPU-hours and 233–236 kWh.The RTX 5000 accounted for 7,152 models and 1,123 GPU-hours; the RTX 5090 accounted for 637 models and 297 GPU-hours.Source: Computational resources · pp. 33–34

Use of large language models

  • Local open-source llama3.1 and qwen3.5 models were used for language polishing and text formatting, with the authors reviewing and editing the content.The authors state that they take full responsibility for the final publication.Source: Use of large language models · p. 34

Funding

The work was funded by the European Union under Horizon grant agreement No. 101287679, with a disclaimer that its views need not reflect those of the EU or REA.

Funding

  • The study was funded by the European Union under Horizon grant agreement No. 101287679; the disclaimer says the views expressed are those of the authors and not necessarily those of the EU or REA.The disclaimer also states that the EU and REA cannot be held responsible for those views.Source: Funding · p. 34

Related papers

Loading 2610.30229v1…