PaperlayerView exact revision on arXiv

Source-linked AI summary

Detection and Validation of Hidden Subgroup Disparities in Clinical AI

Alceu Bissoto, Trung-Dung Hoang, Tim Flühmann, Nina Weng, Susu Sun, Christian F. Baumgartner, Lisa M. Koch

arXiv:2610.32362v1cs.LG

TL;DR

Clinical models may perform differently across patient groups, while annotated metadata can miss hidden performance stratifications. This paper evaluates algorithmic subgroup discovery and finds larger disparities than metadata analysis, supporting its use alongside traditional subgroup analysis.

  • Problem

    Metadata-defined subgroup audits may miss performance-relevant hidden stratifications, while rich annotations are difficult to obtain and often unavailable.

    Source: Introduction · pp. 1–2

  • Method

    The study evaluates discovered subgroups’ performance gaps and coherence, profiles them using existing metadata, and examines their use in bias mitigation.

    Source: Introduction · p. 3, Discussion · p. 10

  • Results

    Discovered subgroups exposed larger, statistically significant disparities than metadata alone across prevalence-dependent and prevalence-independent metrics.

    Source: Introduction · p. 3, Discussion · p. 9

  • Takeaways & Limitations

    The authors conclude subgroup discovery should complement traditional subgroup analysis as a safeguard in real-world ML validation and deployment.

    Source: Discussion · p. 10

    Metadata profiling is limited to annotated factors, and observed associations do not establish causal mechanisms for performance gaps.

    Source: Discussion · p. 10

Abstract

from arXiv · show

Machine learning (ML) models used in clinical workflows often perform systematically differently across patient groups, raising concerns about equity and patient safety. Traditional subgroup analyses can expose such disparities only when rich metadata (e.g., patient demographics) is available and captures the sources of performance variation, conditions rarely met in practice. Algorithmic subgroup discovery methods offer a solution by identifying coherent groups from learned feature representations, but because true underlying stratifications are unknown, these methods are hard to evaluate. As a result, subgroup discovery has not been systematically applied or assessed for performance monitoring. Our main objective is to evaluate algorithmic subgroup discovery for performance monitoring, assessing the discovered subgroups' performance gaps and semantic coherence. Investigating cardiomegaly classification in a dataset of 223,228 chest X-rays, we showed that the discovered subgroups exhibited larger performance gaps (accuracies of 0.61-0.90) than traditional metadata-based analysis (0.80-0.94) and were frequently interpretable using patient, acquisition, and model metadata. As a proof of concept, we also showed that annotations from learned subgroups can be used to mitigate the biases they expose. We conclude that subgroup discovery is a practical and effective tool to enhance the performance validation and monitoring of ML systems in healthcare.

Introduction

Clinical models can perform differently across patient groups, while metadata-defined audits may miss hidden stratifications. This study evaluates algorithmic subgroup discovery for performance disparities, semantic coherence, and potential mitigation.

Introduction

  • Systematic performance differences across patient subgroups threaten safe deployment, particularly for underserved populations.Performance can vary with patient attributes and image characteristics, and hidden failures may also conflict with expectations for safety and effectiveness.Source: Introduction · p. 1
  • Rich metadata can support subgroup audits, but collecting it is difficult and reporting remains sparse.A scoping review found that fewer than 4% of approved devices reported race or ethnicity, fewer than 1% reported socioeconomic factors, and fewer than 19% reported age subgroups.Source: Introduction · p. 2
  • Metadata-defined groups can miss hidden performance stratifications, even in extensively annotated datasets, because model-relevant factors may not align with human-interpretable categories.For cardiomegaly, projection view matters, while sex-based differences may reflect correlated support-device factors; hidden stratifications remained prevalent in CheXpertPlus.Source: Introduction · p. 2
  • Algorithmic subgroup discovery clusters learned feature representations to find model-relevant groups, but validating groups without ground-truth labels remains a central challenge.Prior evaluations had used toy datasets with predefined biases, leaving effectiveness in real-world medical contexts unproven.Source: Introduction · p. 2
  • The study asks whether discovered groups expose performance gaps across metrics, show semantic coherence through interpretable characteristics, and can support mitigation.The evaluation uses a richly annotated chest X-ray dataset for cardiomegaly classification, contrasts metadata-defined and discovered groups, and profiles subgroup metadata.Source: Introduction · pp. 2–3
  • Discovered subgroups showed significantly larger performance disparities than conventional demographic metadata, revealing gaps missed by traditional fairness auditing.The study reports comprehensive evidence across synthetic and real-world settings.Source: Introduction · p. 3
  • Discovered subgroups were cohesive, and many had statistically significant associations with annotated metadata.The proposed subgroup profiles use metadata attributes to explain similarities among samples within a subgroup.Source: Introduction · p. 3
  • The authors report that discovered subgroups can be used to mitigate the biases they expose.The introduction presents this as a demonstrated capability and asks whether subgroup annotations can support mitigation.Source: Introduction · pp. 2–3

Results

In cardiomegaly classification, discovered subgroups showed larger performance disparities than metadata-defined groups and often aligned with acquisition or model features. Synthetic-data checks recovered hidden performance gaps, and subgroup-based annotations improved performance on pacemaker cases.

Subgroup discovery exposes higher performance gaps than traditional subgroup analysis

  • Subgroup A5 combined sensitivity 0.984 with specificity 0.017 and had the largest accuracy gap, 0.260.The discovered groups ranged from 223 to 4,354 images, with cardiomegaly prevalence from 8% to 61%.Source: Subgroup discovery exposes higher… · p. 4, Results · p. 4
  • Discovered subgroups had larger performance disparities than metadata-defined groups, with accuracy 0.61–0.92 versus 0.80–0.94.Sensitivity differed significantly in 7 of 15 discovered subgroups, while accuracy differed in 11 of 15; metadata-defined groups showed significant sensitivity differences in 6 of 17 and accuracy differences in 8 of 17.Source: Subgroup discovery exposes higher… · p. 4, Figure 2 · p. 5

Discovered subgroups are diverse and often align with clinical or acquisition-related concepts

  • Acquisition and model features strongly characterized discovered groups, while patient attributes were less consistently associated; pacemakers marked poor-performing A10.Figure 3 profiles subgroup columns against metadata rows, with associations measured using Cohen’s h for categorical and Cohen’s d for continuous attributes.Source: Discovered subgroups are diverse and… · pp. 5–6, Figure 3 · p. 6
  • Subgroup profiles were limited to available metadata, and similar profiles could still accompany accuracy differences of 6 percentage points or sensitivity differences of 24 points.The authors propose using exposed differences to prompt domain-expert investigation of missing subgroup attributes.Source: Discovered subgroups are diverse and… · p. 6

Use case: subgroup annotations for improving model robustness

  • Using Subgroup A10 annotations with groupDRO improved balanced accuracy on pacemaker cases by 3.2 percentage points, from 70.0 to 73.2.The 200 pacemaker annotations were reserved for final test evaluation; training and tuning used subgroup-based annotations.Source: Use case: subgroup annotations for… · p. 7

Subgroup discovery passes sanity checks on synthetic data

  • On synthetic data, subgroup discovery isolated artifact combinations and captured hidden-subgroup performance gaps that analysis using only one known artifact missed.The synthetic experiment introduced vertical lines and a hospital tag as artifacts correlated with the positive disease label.Source: Figure 4 · p. 8, Subgroup discovery passes sanity… · p. 8

Feature extractors trained on natural images are sufficient for exposing meaningful performance gaps

  • BiomedCLIP and generalist CLIP exposed similar performance disparities but yielded different metadata profiles, with BiomedCLIP linking lung diseases to 4 of 15 groups.BiomedCLIP also identified an older-patient subgroup, while the generalist CLIP profiles did not link groups to lung diseases.Source: Feature extractors trained on natural… · pp. 8–9, Table 3 · p. 9

Discussion

The study finds that metadata alone can miss meaningful subgroup disparities, while algorithmically discovered groups show larger, statistically significant gaps and can be profiled using existing metadata. The authors describe uses for flagged groups and note that metadata-based profiling cannot reveal factors that were not annotated.

Discussion

  • Discovered subgroups showed larger, statistically significant disparities than metadata-defined groups across accuracy, sensitivity, specificity, and AUC.The analysis used a real-world chest X-ray dataset with extensive metadata.Source: Discussion · p. 9
  • Subgroup profiles use existing metadata to characterize discovered groups, while statistical analysis and sanity checks assess content fidelity and profiling makes subgroup structure clinically legible.The pipeline also assesses actionability to inform whether and how to act.Source: Discussion · pp. 9–10
  • Disparities can prompt deferral to specialist review and stratified monitoring; examples include SG A0 with sensitivity 0.10 and specificity 0.95, and SG B14 with sensitivity 0.17 and specificity 0.97.The authors state that these patients can be flagged for manual assessment rather than relying on AI predictions.Source: Discussion · p. 10
  • Responses depend on subgroup characteristics: the authors discuss recalibration for clinically relevant factors, additional imaging and robustness testing for imaging distortions, and targeted domain adaptation for confounding factors.They demonstrated targeted domain adaptation using subgroup labels in a proof-of-concept experiment.Source: Discussion · p. 10
  • Even without an identified unifying concept, subgroups with elevated error rates can support deferral and serve as challenging holdout groups for ongoing validation.The authors give SG A5 and SG A6 as examples.Source: Discussion · p. 10
  • Some discovered groups could not be linked to annotated clinical, demographic, or acquisition factors, and metadata profiling is bounded by what was annotated.The authors state that unannotated factors may drive subgroup structure and that metadata associations do not establish causal mechanisms for performance gaps.Source: Discussion · p. 10

Methods

The method seeks subgroups with substantial performance gaps while retaining cohesion, then evaluates discovered groups using statistical comparisons, metadata profiles, and a synthetic-artifact sanity check.

Methods

  • The proposed approach uses subgroup discovery to investigate hidden stratifications that metadata-defined groups may not expose because their attributes may not reflect the main factors affecting model performance.The approach seeks large performance gaps while preserving subgroup cohesion so performance can be attributed to a shared characteristic.Source: Methods · pp. 10–11
  • The workflow fits a simplified DOMINO model on validation data and infers subgroup assignments for test samples after training the cardiomegaly classifier.The passage specifies train/validation/test splits but not their proportions.Source: Methods · p. 11
  • Evaluation measures performance gaps with multiple metrics and statistical tests, characterizes discovered groups using existing metadata, and checks recovery of ground-truth performance using injected synthetic artifacts.The artifact scenario supplies known subgroup structure for the sanity check.Source: Methods · p. 11

Dataset description

The study uses CheXpert-Plus, with extensive metadata, and applies DOMINO to derive subgroups from image representations and target-model predictions.

Dataset description

  • CheXpert-Plus provides 20 metadata attributes, including demographics, comorbidities, and artifacts, and the dataset was split 80/10/10 into 178,684 training, 22,263 validation, and 22,281 test images.Source: Dataset description · p. 11

Preliminaries: subgroup discovery algorithm

  • DOMINO extracts image representations with an external pretrained model, reduces their dimension with PCA, combines them with target-model softmax predictions, and clusters samples with a generalized Gaussian Mixture Model.Source: Preliminaries: subgroup discovery… · p. 11
  • The parameter γ balances predicted-label and embedding influence on slicing; removing classification labels from the GMM allows discovery on unlabeled post-deployment test sets, with fitting on validation data and inference on test data.Source: Preliminaries: subgroup discovery… · p. 11

Statistical analysis of subgroup performance

Subgroup performance is measured with prevalence-dependent and prevalence-independent metrics, and disparities are tested against each subgroup’s complement with correction for multiple comparisons.

Statistical analysis of subgroup performance

  • Accuracy, sensitivity, specificity, and AUROC are estimated within each predefined or discovered subgroup, with accuracy, sensitivity, and specificity evaluated at a fixed threshold of 0.5.Wilson 95% intervals are used for the first three metrics; AUROC uses a 95% bootstrap interval from 1,000 within-subgroup resamples, skipping single-class resamples.Source: Statistical analysis of subgroup… · p. 11
  • Each subgroup is compared with all remaining examples using proportion tests for accuracy, sensitivity, and specificity and DeLong’s method for AUROC; Bonferroni-corrected p-values below 0.05 define significance.Source: Statistical analysis of subgroup… · p. 11

Statistical analysis of metadata profiles

Metadata profiles characterize how attributes differ within each discovered subgroup compared with the rest of the dataset. The analysis uses effect sizes, significance testing, and filtering to produce a heatmap of retained associations.

Statistical analysis of metadata profiles

  • Cohen’s h measured categorical proportion differences and Cohen’s d measured standardized mean differences for continuous attributes within versus outside each subgroup.Positive values indicate greater proportions or higher means within the subgroup; negative values indicate lower proportions or means.Source: Statistical analysis of metadata… · p. 12
  • Significance was tested with chi-square tests for categorical attributes and independent-samples t-tests for continuous attributes, with Benjamini–Hochberg correction across subgroup–attribute pairs.Results required p < 0.05 and an absolute effect size of at least 0.3 for visualization.Source: Statistical analysis of metadata… · p. 12
  • The heatmap displays retained standardized effect sizes, with columns for discovered subgroups, rows for metadata attributes, and colors indicating association direction and strength.Only attributes meeting both the significance and effect-size thresholds were visualized.Source: Statistical analysis of metadata… · p. 12

Synthetic scenario with generated artifacts

The synthetic evaluation introduced two artifacts correlated with disease labels to create known subgroup structure. An unbiased test set supported comparisons across training bias levels.

Synthetic scenario with generated artifacts

  • Two clinically inspired artifacts—a bottom-left hospital tag and vertical hyperintense lines—were added to simulate performance disparities.The artifacts followed prior work and were applied to CheXpertPlus.Source: Synthetic scenario with generated… · p. 12
  • Each artifact was independently correlated with the positive label, with insertion probability p for positive samples and 1−p for negative samples, creating four ground-truth subgroups.Biased training and validation sets were used to train the classifier and select hyperparameters; validation data also fit DOMINO.Source: Synthetic scenario with generated… · p. 12
  • Testing used an unbiased set with p = 0.5 to enable fair comparisons across training bias levels.Source: Synthetic scenario with generated… · p. 12

Experimental setup

The study used a fixed classification setup and describes how subgroup discovery could support clinical AI post-market surveillance. It presents subgroup alerts as a prompt for assessment and, if needed, targeted model correction.

Experimental setup

  • Experiments trained ResNet-50 models for cardiomegaly versus all, used 15 discovered subgroups, and selected models by validation balanced accuracy.Training used SGD and learning-rate search over {10−5, 10−4, 10−3} with weight decay 10−4; CLIP was the external model, with BiomedCLIP added for real-world experiments.Source: Experimental setup · p. 12

Perspective: regulatory implications

  • Under the MDR, manufacturers must maintain post-market surveillance that assesses device performance and safety and identifies situations that may increase patient risk.Subgroup discovery is described as complementing predefined sociodemographic evaluations by revealing additional clusters with meaningful performance gaps.Source: Perspective: regulatory implications · p. 12
  • Subgroup discovery can partition validation outputs using a separate model’s feature representation without retraining or access to model internals, then flag new patients belonging to lower-accuracy groups.Metadata profiles can help clinicians assess associated characteristics when deciding whether to use or override a model output.Source: Perspective: regulatory implications · p. 13
  • Alerts would prompt assessment of whether cases reflect isolated events or a broader model limitation, and subgroup-derived labels can support targeted retraining when corrective measures are needed.Source: Perspective: regulatory implications · p. 13

Data availability

The study identifies public sources for the CheXpertPlus dataset and the pacemaker-labeled images used in method validation.

Data availability

Code availability

The code used to train and evaluate the models is publicly available on GitHub.

Code availability

  • Training and evaluation code is available at https://github.com/alceubissoto/hidden-sg-disparities.Source: Code availability · p. 13

Related papers

Loading 2610.32362v1…