Source-linked AI summary
FRAME: separating sampling variation from representational cause in medical imaging fairness
Mahshad Lotfinia, Daniel Truhn, Andreas Maier, Soroosh Tayebi Arasteh
TL;DR
Medical imaging fairness research often interprets subgroup performance gaps as model bias because representations encode demographics, but the contribution of sampling variation and mechanisms remains uncertain. FRAME estimates the gap expected under exact fairness at observed subgroup sizes and tests whether representation interventions alter the remainder. Across encoders and published studies, the reference explains substantial portions of reported differences, while demographic decodability does not change the remainder and image-text pretraining improves worst-group performance.
Problem
Medical imaging fairness studies commonly use subgroup performance differences as evidence of bias, although the roles of sampling variation and demographic encoding in those differences have not been quantified or tested by intervention.
Method
FRAME simulates a fair-model reference at observed subgroup counts, subtracts it from the reported difference, and tests candidate representation-space causes of the remainder.
Results
The reference accounts for a median 41% of reported race differences, 22% of age differences, 25% of rate differences, and 70% of AUROC differences; demographic decodability leaves the remainder unchanged, while image-text pretraining raises worst-group AUROC by about 0.05.
Takeaways & Limitations
Applying FRAME before selecting an intervention can distinguish differences compatible with sampling variation at current cohort sizes from findings requiring a mechanistic explanation.
Takeaways & Limitations
The mechanism behind the remainder is not identified because the interventions act on cached features rather than during pretraining, and the pooled race contrast is partly a site contrast.
Abstract
from arXiv · showhide
Subgroup performance differences are the standard evidence for fairness bias in medical imaging, and the usual response removes the demographic information that a model encodes. Here we introduce Fair-model Reference And Mechanism Evaluation (FRAME), a two-step framework for auditing such a claim. The first step derives a fair-model reference, the distribution of the difference under exact fairness at the observed subgroup sizes. In the second step, we test the remainder with two operators in representation space. One operator cannot change a within-group ranking by construction. Across 702,206 images and 36 encoders, the reference accounts for a median 41% of the reported race difference and 22% of the age difference. Injecting demographic decodability leaves the remainder unchanged, while entangling the group with the disease direction raises the race difference from 0.077 to 0.118. No intervention we tested changes the remainder more than a change of random seed does. Those interventions reduce a difference at the operating point and leave the within-group ranking difference at a median of 0.000. Applied to 89 differences in 9 published studies across 6 medical imaging modalities, the reference accounts for a median 25% of a rate difference and 70% of a difference in the area under the receiver operating characteristic curve. Image-text pretraining instead raises worst-group performance by about 0.05. Applying FRAME before choosing an intervention could distinguish differences that need a mechanistic explanation from differences compatible with sampling variation at the current cohort sizes.
Introduction
Medical imaging fairness work commonly treats subgroup performance differences as evidence of unfairness and responds by removing encoded demographic information. FRAME audits this reasoning by separating sampling variation at observed subgroup sizes from mechanisms that could affect the remaining difference.
- Motivation: Medical imaging models show unequal performance across race, sex, insurance, and skin-tone groups, motivating model-level fairness interventions.Reported examples include underdiagnosis and AUROC gaps across diagnostic tasks.
- Motivation: The standard response assumes that demographic encoding causes subgroup differences and that removing this encoding will equalize performance.Mitigations range from subgroup thresholds and constrained losses to worst-group objectives, adversarial classifiers, and concept erasure.
- Open problem: Existing evidence leaves the causal mechanism untested, while task difficulty, confounding, correction scope, and small-sample statistics can also produce unequal performance.The maximum-minus-minimum statistic rises with sampling variation, and within-group AUROC changes only when score ordering changes within a subgroup.
- FRAME: FRAME first simulates the difference expected under exact fairness at observed subgroup counts, then tests whether representation-space interventions change the remainder.One operator cannot change within-group rankings by construction; the framework also evaluates encoders across objectives, backbones, pretraining composition, and seeds.
- Key findings: The fair-model reference accounts for a median 41% of reported race differences and 22% of age differences across the study’s evaluations.In published studies, it accounts for a median 25% of rate differences and 70% of AUROC differences.
Results
FRAME shows that subgroup differences can partly arise from sampling variation, while representation-space interventions distinguish demographic decodability from disease-direction entanglement. Across modalities and interventions, operating-point changes can reduce rate disparities without changing within-group rankings, whereas erasure methods trade disparity reduction against disease AUROC.
- Step one of FRAME: the reference against the reported difference: 41% of the reported race difference and 22% of the age difference are accounted for by the fair-model reference.The observed race difference has median 0.082 against a reference of 0.034; age has 0.066 against 0.014.
- Step one of FRAME: the reference against the reported difference: 79 of 130 race combinations and 116 of 130 age combinations significantly exceed their fair-model references.Across the full study, the corresponding counts are 129 of 243 for race and 226 of 252 for age.
- Step two of FRAME: candidate causes of the remainder: Demographic decodability does not predict the achievable difference, with linear-decoding correlation 0.036 and nonsignificant out-of-sample predictors.The cost of erasure and geometric overlap likewise show nonsignificant correlations of −0.142 and −0.027.
- Step two of FRAME: candidate causes of the remainder: Injecting decodability raises linear race decodability from 0.866 to 0.942 while the race difference stays at 0.077, whereas entanglement raises it from 0.077 to 0.118.The decodability injection leaves the race difference unchanged; entanglement with the disease direction increases it.
- The remainder under every intervention: Retraining changes the race difference by a median 0.012, compared with a median 0.005 reduction from nine mitigation methods.No trend across unfreezing levels is significant for any attribute or encoder.
- Operating point and ranking disparities under mitigation: Per-group operating-point shifts reduce median sensitivity and FPR differences by 0.075 and 0.069 while leaving the race AUROC difference at 0.000.Strictly increasing score maps cannot change within-group rankings, although calibration differences increase by 0.016.
- The remainder under every intervention: LEACE lowers the race AUROC difference by 0.012 while lowering disease AUROC by 0.030; iterative nullspace projection lowers them by 0.010 and 0.049.Other methods also trade disparity reductions against disease AUROC reductions.
- Acquisition view and site as competing explanations: Acquisition view and data source show median differences of 0.100 and 0.095, compared with 0.081 for race and 0.074 for age.All four differences exceed their own fair-model references.
Discussion
FRAME separates subgroup differences compatible with sampling variation from remainders that may require a mechanistic explanation. Across medical imaging studies, the reference often explains substantial reported differences, while tested interventions rarely change the remainder and image-text pretraining improves worst-group performance.
- Discussion: A median 41% of race differences and 22% of age differences matched the fair-model reference in 125,992 chest radiographs.For race, 51 of 130 encoder–finding combinations did not exceed the reference.
- Discussion: Across 89 published differences, 40 of 53 thresholded-rate differences exceeded their reference, compared with 1 of 36 AUROC differences.The reference retained a median 75% of reported rate differences but represented a median 70% of AUROC differences.
- Discussion: Demographic decodability did not change the remaining subgroup difference, whereas entangling group with disease direction raised the difference from 0.077 to 0.118.Decodability varied substantially across encoders without predicting the achievable difference.
- Discussion: Across nine mitigation methods, controlled pretraining, and four unfreezing levels, subgroup AUROC differences changed less than under a pretraining-seed change.Strictly increasing threshold and calibration maps leave within-group rank statistics unchanged; LEACE reduced race AUROC difference by a median 0.012 while lowering disease AUROC by 0.030.
- Discussion: Image-text pretraining and deeper finetuning each raised worst-group AUROC by about 0.05, exceeding the gain from tested fairness interventions.Image-text pretraining changed the subgroup difference by at most a median 0.015.
- Discussion: The study cannot identify how entanglement arises in trained encoders because interventions operated on cached features rather than during pretraining.The pooled race contrast is also partly a site contrast because most records in the Other category come from sites without race fields.
Methods
FRAME estimates how much of a reported subgroup difference is compatible with sampling variation, then tests candidate mechanisms by intervening on cached representations.
- Data and evaluation units: 702,206 images span chest radiography, dermatology, and retinal funduscopy, with sensitive attributes harmonized across sites.Subgroups with fewer than 20 evaluable images are dropped, and each attribute must have at least two evaluable subgroups.
- Difference statistic: The reported subgroup difference is the range between the maximum and minimum subgroup performance values.The framework evaluates AUROC, sensitivity, false-positive rate, and expected calibration error.
- Fair-model reference: The fair-model reference simulates that range under exact fairness using the observed subgroup counts and the model’s overall performance.For AUROC, sampling distributions use subgroup positive and negative counts; the construction extends to measures with an available sampling distribution.
- Fair-model reference: The reference is estimated from 2,000 simulations, and the remainder is the observed difference minus the reference.The reported interval uses the 2.5th and 97.5th percentiles of the simulated distribution.
- Representation interventions: Two operators inject candidate causes into cached encoder features before refitting the prediction head.The decodability operator adds a group-dependent shift, while the entangling operator modifies features using a disease subspace.
- Representation interventions: The decodability operator preserves within-group rankings for any fixed linear head, whereas the entangling operator has no such invariance.Heads are refitted at every intervention strength, so only refitting can reorder scores within a group under the decodability operator.
Experimental design
The experiments test FRAME’s claims using matched controls: each difference is compared with its own fair-model reference, interventions are evaluated causally, and disease performance is held constant.
- Controls: Every subgroup difference is compared against its own reference at the observed subgroup counts rather than against zero.This prevents the expected positive range under finite sampling from being treated as disparity by default.
- Controls: Candidate causes are tested by intervention as well as observation because cross-model association cannot establish that changing a candidate changes the difference.The design directly evaluates whether the remainder responds to representation-space manipulations.
- Controls: Interventions are compared at matched disease performance so reducing a subgroup difference by reducing overall performance is not counted as success.The comparison uses the best achievable difference subject to a disease-performance tolerance.
- Experimental scope: The full design runs on chest radiography, while dermatology and funduscopy provide scope checks under the identical protocol.A final experiment applies step one to 89 published subgroup differences from nine studies across six imaging modalities.
Representation geometry and the synthetic model
The study measures sensitive-attribute information in frozen representations, manipulates its geometry relative to disease directions, and evaluates mitigation methods at matched disease performance.
- Representation measurements: Four measures quantify how much frozen representations encode sensitive attributes, including linear and nonlinear decodability and geometric overlap.Decodability is evaluated on held-out test features, with chance at 0.5 for linear attribute prediction.
- Synthetic model: The synthetic model varies collinearity between group and disease directions while holding both group effects fixed.It uses 160-dimensional features with signal in 32 dimensions and recomputes discrimination and geometry measures at each collinearity level.
- Synthetic model: The synthetic group-direction term is rank-neutral, whereas the group-specific variance term is not and projects onto disease according to their alignment.The alignment parameter ρ ranges from 0 to 1 across five seeds and 4,000 samples per setting.
- Encoder panel: The frozen encoder panel includes ten chest-radiography encoders, with native global embeddings extracted once and cached.The broader study uses released models and controlled pretraining recipes spanning image-image, disease-label, and image-text objectives.
- Mitigation evaluation: Nine mitigation methods cover data, objective, score, and feature-erasure interventions, and are evaluated for race and age across modalities and head types.The reported comparison selects the smallest subgroup difference among methods whose disease AUROC remains within 0.010 of the unmitigated model.
- Mitigation evaluation: Ridge models predict the matched-performance difference from erasure cost, geometric overlap, or linear and nonlinear decodability using five-fold cross-fitting.The reference distribution uses 1,000 target shuffles, and negative out-of-sample R2 indicates performance below the held-out mean.
Finetuning and the acquisition controls
Additional experiments examine finetuning, acquisition-view variation, and published evidence, while using clustered resampling and multiplicity control for uncertainty and comparisons.
- Finetuning: Four unfreezing levels compare a linear head, last transformer block, LoRA adapters, and full-backbone finetuning.Early stopping is used because the full level otherwise memorized the training split and underperformed the LoRA level on test AUROC.
- Acquisition controls: Acquisition-view analyses recompute race and age differences with view-specific references and require at least 500 evaluable images per view.View and site are also analyzed as attributes in their own right.
- Published-study audit: The published-study audit includes 89 claims from nine studies across six modalities, selected using four criteria for recoverable subgroup differences and performance values.The authors state that the study set is not a systematic search.
- Published-study audit: Published references use reported overall performance when available, otherwise recovering it from denominator-weighted subgroup values for rate statistics.Rate references use exact binomial draws, while AUROC references use the corresponding subgroup sampling procedure.
- Statistical analysis: Patient-level cluster bootstrap, fixed simulation counts, and FDR correction govern uncertainty, paired comparisons, and exceedance testing.The fair-model reference uses 2,000 simulated fair models, while performance uncertainty uses 1,000 patient-level bootstrap resamples.
Data availability
The study uses existing, publicly released data sources and does not redistribute images or derived records. The published-difference audit uses only information printed in the audited articles.
- Data availability: Existing, publicly released sources provide all study data, with one chest radiograph reproduced from NIH ChestX-ray14.No images or derived records are redistributed.
- Data availability: The audited-article analysis uses no data beyond what the published articles report.
Code availability
The analysis code is publicly available and documents the data-building, pretraining, analysis, and figure-generation workflow. Evaluated encoders were local, open-weight models run as frozen feature extractors without cloud services or network-served models.
- Code availability: The analysis repository provides data-build, controlled-pretraining, analysis, and figure code, including fixed bootstrap, permutation, and simulation seeds.It does not redistribute model weights or underlying datasets.
- Code availability: All evaluated encoders were local, open-weight frozen feature extractors run entirely on-site without cloud services or third-party APIs.No closed model or network-served model was used.
- Code availability: The encoder inventory includes chest-radiograph, general-purpose, dermatology, and fundus models, plus a text encoder for image-text pretraining.
- Code availability: Analyses used a pinned Python and machine-learning environment and ran encoder training, finetuning, and feature extraction on NVIDIA L40S GPUs.Other stages ran on CPU.
Competing interests
The disclosure reports employment, editorial-board roles, lecture honoraria, and shareholdings among the authors.
- Competing interests: Reported interests include company employment, editorial positions, lecture honoraria from several medical-technology companies, and shares in two companies.
Supplementary Note 1: Additional checks on the chest radiograph protocol
Additional checks examine sensitive attributes, calibration, model-head capacity, encoder replication, entanglement, and a separate optimal-transport mitigation. These analyses identify substantial scope and interpretation boundaries for the protocol’s conclusions.
- Sex and insurance: For sex, the observed difference has a median of 0.009 against a fair-model reference of 0.004 across 104 combinations.The observed difference exceeds the reference in 35 combinations.
- Sex and insurance: Insurance differences have a median of 0.062 across 78 combinations, but no fair-model reference or significance test was computed.The authors do not interpret these values as evidence of disparity because the Other category is strongly confounded with site.
- Calibration: Per-group Platt calibration leaves the race or age AUROC difference unchanged in 252 of 260 combinations or changes it by less than 1 × 10−6 in three more.Five larger changes occur in race comparisons with very small positive subgroups, where fitted logistic maps can become constant.
- Disease-head capacity: Changing from a linear to a two-layer disease head changes median differences by −0.002 for race, +0.004 for age, +0.002 for insurance, and +0.001 for sex.The largest single change is 0.088.
- Encoder replication: Injecting demographic decodability raises decodability while leaving the race difference unchanged on both transformer and convolutional encoders.This reproduces the invariance across a second embedding width and backbone type.
- Entanglement: At k = 256, entanglement changes race differences little across findings, while the overall disease AUROC is unchanged on every finding for one encoder.Because disease AUROC is unchanged on DenseNet-121, the entanglement effect is not tested there and the main-text causal claim depends on RAD-DINO alone.
- Optimal-transport mitigation: Optimal-transport matching yields a median race difference of 0.078 versus 0.079 unmitigated across 13 findings.Only pleural effusion shows a significant reduction, falling by 0.006.
Supplementary Note 2: Data and reporting caveats
The pooled race and insurance attributes include an “Other” category that combines recorded categories with patients from sites lacking the field. The fair-model reference uses image counts, although patient-level dependence makes those counts optimistic for sampling variability.
- Attribute harmonization: Race is recorded at only two of six chest radiograph sites, so records from the other four sites are assigned to “Other.”The same two-site limitation applies to insurance.
- Attribute harmonization: 68,521 radiographs are in the race “Other” category, including 52,038 (76%) from sites without a race field.
- Sampling model: The reference draws subgroup AUROCs at image counts, while radiographs from one patient are not independent.Bootstrap intervals instead resample patient clusters.
- Sampling model: Positive within-patient correlation means the image-count reference understates the spread produced by a perfectly fair model.The resulting correction is conservative: it attributes less of the reported difference to sampling, leaving the remainder as an upper bound.
- Sampling model: The direction of this bound is an assumption because within-patient correlation was not estimated.
Supplementary Note 3: Dermatology and funduscopy
Across dermatology and funduscopy, the fair-model reference often matches or exceeds observed subgroup differences, especially outside dermatology age comparisons. With these cohort sizes, mitigation tests generally cannot separate the observed differences from sampling variation, and cross-modality geometry prediction does not generalize.
- Dermatology: Dermatology age differences exceed the fair-model reference in 9 of 9 combinations, with observed median 0.068 versus reference 0.023.
- Funduscopy: Fundus sex has an observed median difference of 0.008, below its fair-model reference of 0.014.
- Dermatology and funduscopy: No other attribute in either modality exceeds its reference in more than 2 of 9 combinations.
- Mitigation tests: None of 126 mitigation tests across both modalities, attributes, and metrics is significant.At these cohort sizes, the analysis cannot distinguish most observed differences from sampling variation under exact fairness.
- Cross-modality prediction: Out-of-sample R2 is −3.40 for dermatology and −1.92 for funduscopy, neither significant (pFDR = 0.787 for both modalities).Within chest radiography, the leave-one-encoder-out predictor has out-of-sample R2 of 0.024 (pFDR = 0.021).
Supplementary Note 4: The steps of FRAME
FRAME first estimates the subgroup difference expected under exact fairness at the observed counts, then tests whether candidate representation mechanisms change the remainder. The procedure can audit published results from summary counts alone in step one, while step two requires cached features.
- Scope of application: Step 1 needs overall AUROC and subgroup positive and negative counts, whereas step 2 requires cached features and therefore applies only when the representation is available.
- Reference implementation: The closed-form reference draw is an approximation, but it agreed with slower binormal simulations on 12 randomly chosen units per run.
- Inputs: FRAME takes scores, labels, subgroup assignments, cached features, a train index, simulation draws, a significance level, and operator strength grids.
- Step 1: fair-model reference: Step 1 simulates subgroup AUROCs with the same true AUROC for every subgroup and forms the fair-model reference from their maximum-minus-minimum differences.
- Step 1: fair-model reference: The remainder is the reported difference minus the reference; if the exceedance test is nonsignificant, FRAME classifies it as consistent with exact fairness.
- Step 2: mechanisms: Step 2 injects demographic decodability and group–disease entanglement into cached features, then tests how each changes the remainder.The decodability direction is orthogonalized against the disease direction, and the operators are applied over graded strengths.
- Published-study audit: The published-study audit applies step one using each study’s reported values and subgroup counts, without using the authors’ models.