PaperlayerView exact revision on arXiv

Source-linked AI summary

Attributing Cohen's d: Training Data Attribution for Disease-Related Effects in Normative Age Biomarkers

Jakob Snel, Marc-Andre Schulz

arXiv:2609.07729v2cs.LGq-bio.QM

TL;DR

Normative age models use age gaps to assess disease-related effects, but reference cohorts formed by diagnosis-based exclusions may include subclinical or undiagnosed cases. This paper attributes held-out case-control effect size directly to training samples. Across four disease conditions and two modalities, attribution-guided removal increases held-out effect size in every seed, unlike random removal.

  • Problem

    Reference cohorts are composed using diagnosis-based a priori exclusions, leaving subclinical or undiagnosed cases absent from exclusion lists unaddressed.

    Source: §1 · p. 2

  • Method

    Cohen’s d influence functions attribute the held-out disease-related effect size directly to training samples, with separate selection and evaluation halves.

    Source: §2 · p. 3

  • Results

    Across four conditions, two modalities, and all tested removal levels, paired changes were positive in every seed-level cell, while random removal was centred on zero even at 50%.

    Source: §4.1 · p. 5

  • Takeaways & Limitations

    Flagged subjects carry subclinical cardiometabolic burden missed by diagnosis-based exclusion, with elevated HbA1c separation strongest in type 2 diabetes.

    Source: §1 · p. 2, §4.3 · p. 5

    The study tests only ridge regression and has not established whether its rankings transfer to nonlinear age models.

    Source: §5 · p. 5

Abstract

from arXiv · show

Normative age models are trained to predict chronological age in a nominally healthy cohort. Applied to patients, they deviate, and the gap between predicted and chronological age is read as disease risk. Here, we attribute the disease-related effect size of the age gap directly to individual training samples, rather than using a prediction-level loss as the attribution target. For Cohen's $d$, the resulting closed-form influence functional, validated against leave-one-out retraining, ranks training samples by their effect on held-out case-control separation. Across four diseases and two biomarker modalities in UK Biobank, removing the 10% most influential training samples raises held-out disease-related effect size in every seed. It more than doubles the metabolomic-age effect for type-2 diabetes and raises the brain-age effect for multiple sclerosis by roughly a third. Random removal leaves effect size flat even at 50% removal, confirming the gain comes from which samples are removed, not how many. Flagged subjects carry subclinical cardiometabolic burden that diagnosis-based exclusion misses, on markers the model never sees. For type-2 diabetes, where the method gains most, the marker recovered is HbA1c, the standard measure of blood sugar control. We release pyinfluence, our influence-function package, for reproducibility and reuse.

1 Introduction

Normative age models use deviations from chronological age as a disease-risk signal, but diagnosis-based reference-cohort exclusions can miss subclinical contamination. This paper targets attribution to held-out disease-related effect size and reports that attribution-guided removal raises that effect across four diseases and two modalities.

1 Introduction

  • A normative age biomarker predicts chronological age in a nominally healthy cohort, and its patient age gap is read as disease risk because disease was absent from training.Source: §1 · p. 1
  • Reference-cohort contamination may inflate variance in disease-carrying features, leading an age-error-minimising model to downweight those features.The passage notes that models with worse age-prediction accuracy can show larger disease-related effects.Source: §1 · p. 1
  • Current reference-cohort practice uses a priori exclusion lists based on recorded diagnoses, with no post-hoc approach to catch subclinical or undiagnosed cases absent from those lists.Source: §1 · p. 2
  • Conventional attribution targets training points responsible for individual predictions, whereas normative age biomarkers are evaluated by Cohen’s d between held-out cases and matched controls.Source: §1 · p. 2
  • Cohen’s d influence functions attribute the disease-related effect size itself, rather than an individual age prediction, to training samples that reduce the downstream effect size.The authors report that the flagged samples carry subclinical cardiometabolic burden missed by diagnosis-based exclusions.Source: §1 · p. 2
  • Across four diseases and two UK Biobank modalities, attribution-based removal raises held-out disease-related effect size in every seed, while same-size random removal does not.The multiple-sclerosis example widens case-control mean age-gap separation to 5.2 years versus 3.4 years after random removal of the same share.Source: §1 · p. 2, Figure 1 · p. 2

2 Attributing a disease-related effect size

The method propagates each training sample’s influence through the age-gap pipeline to the held-out case-control effect size. For ridge regression, its rankings closely match exact leave-one-out retraining, with evaluation separated from attribution-score selection.

2 Attributing a disease-related effect size

  • The biomarker uses ridge regression to predict age from imaging or NMR features, applies a linear age-bias correction, and measures Cohen’s d of age gaps between cases and matched controls.The age gap is the bias-corrected predicted age minus chronological age.Source: §2 · p. 3
  • For each training sample, the estimand is the change in held-out effect size after removing it, and ridge parameter influence is propagated through raw held-out predictions into the functional F.Scores are computed in closed form, and only their ranking is used.Source: §2 · p. 3
  • At scoring, the age-bias correction coefficients are held fixed, although they are refit for actual removals, and samples are removed in descending estimated effect-size change.Source: §2 · p. 3
  • Matched cases and controls are split by pair into a selection half for attribution and an evaluation half for reporting, because selection-half effects inflate mechanically with removal.Source: §2 · p. 3
  • 98.9% to 99.8% of removed samples overlapped between influence and leave-one-out rankings; retraining under either ranking changed evaluation-half Cohen’s d by a mean of 0.001 and maximum of 0.009.Source: §2 · p. 3

3 Experimental setup

The experiments use UK Biobank data across two biomarker modalities and four disease conditions. Removal rankings are tested against matched random-removal controls across seeds, with paired changes used to compare effect sizes.

3 Experimental setup

  • UK Biobank brain age from T1 imaging is evaluated for multiple sclerosis and cerebrovascular disease, while NMR metabolomic age is evaluated for type 2 diabetes and chronic kidney disease.The conditions have 84, 404, 1,000, and 1,000 evaluation cases, respectively.Source: §3 · p. 3
  • Training pools are subsampled to 40,000, and people diagnosed after the visit are excluded from training and control pools before attribution.Any disease-relevant burden found afterward is therefore subclinical or undiagnosed rather than already caught by diagnosis-based exclusion.Source: §3 · p. 4
  • Each seed tests 5%, 10%, 25%, and 50% removal against random removal of the same number of samples, with model refitting and re-scoring at each level.Rankings are computed once per seed, and the study repeats the procedure across 10 seeds per condition.Source: §3 · p. 4
  • The reported paired change is attribution-arm Cohen’s d minus random-arm Cohen’s d within each seed, averaged across seeds to cancel shared cohort-draw variance.Both Cohen’s d and mean age gap are tested as attribution targets.Source: §3 · p. 4

4 Results

Influence-guided removal improves held-out disease-related effect sizes across conditions, while random removal leaves them near baseline. The preferred attribution target depends on selection-half size, and flagged subjects show cardiometabolic differences in three of four conditions.

4.1 Removing what attribution flags works and removing at random does not

  • At every removal level, paired Δd is positive across all tested seed-level cells; random removal stays near baseline even at 50%, so gains reflect ranking.The comparison is attribution-based removal versus random removal, with the shared 0% value serving as baseline.Source: Figure 2 · p. 4, Table 1 · p. 4, §4.1 · p. 5
  • At 10% removal, metabolomic-condition gains (+0.85 and +0.56) exceed brain-condition gains (+0.20 and +0.05).Source: §4.1 · p. 5

4.2 Choice of attribution target

  • At 1,000 cases per group, Cohen’s d wins by 0.007 to 0.033; at 404 and 84 cases, mean age gap wins by 0.004 to 0.006 and 0.043 to 0.079, respectively.The difference is a bias-variance tradeoff: the pooled-SD term is informative with 1,000 cases but noisy at smaller selection halves, where the stable, misspecified mean-age-gap surrogate is preferred.Source: §4.2 · p. 5

4.3 Who gets flagged

  • In three of four conditions, the top 10% flagged subjects have higher BMI, waist-to-hip ratio, CRP, and HbA1c, plus worse self-reported health than matched unflagged subjects.The type 2 diabetes HbA1c separation is d ≈ 0.16, more than three times that in any other run; these markers are flat or reversed for chronic kidney disease.Source: §4.3 · p. 5
  • Unlike the other three conditions, chronic kidney disease shows an ethnicity gap (d = −0.077), larger than any cardiometabolic marker, raising a fairness question.Source: §4.3 · p. 5

5 Limitations

The reported evidence is bounded by the model class, cohort design, and interpretation of flagged phenotypes. Some comparisons and causal explanations remain unresolved.

5 Limitations

  • Nonlinear age models were not tested, and the study’s batch-wide ranking may differ from an iterative approach if contamination has cascade structure.The method’s closed-form influence calculation is for ridge regression.Source: §5 · p. 5
  • For CKD, cardiometabolic bars were near zero while ethnicity had the largest separation, d = −0.077, exceeding CRP at +0.039.In type 2 diabetes, the largest reported differences were HbA1c +0.162, BMI +0.119, CRP +0.104, and health rating +0.097.Source: Figure 3 · p. 6
  • The study does not compare against conventional loss-based attribution because its target is group-level effect size rather than single-subject predictive accuracy.This leaves performance relative to those attribution baselines unreported.Source: §5 · p. 6
  • Results come from one biobank and condition-specific reference cohorts; gains under a shared cohort excluding all studied conditions remain unknown.Two metabolomic conditions also used a capped control pool; paired Δd results do not depend on baselines, but absolute effect sizes do.Source: §5 · p. 6
  • Flagged-group phenotype differences do not explain why samples were ranked, and the CKD ethnicity separation could reflect clinical signal or fairness concerns.Establishing a causal link is outside the analysis; deriving interpretable inclusion criteria from flagged profiles is future work.Source: §5 · p. 6

6 Recommendations

The recommendations call for matched random controls, paired changes, and phenotype reporting alongside removal results.

6 Recommendations

  • Pair each attribution arm with a count-matched random arm under identical seeds to avoid confounding attribution with training-set size.This controls removal count while comparing the attribution-based and random arms.Source: §6 · p. 6
  • Report the paired change between attribution and random arms rather than absolute values or ratios of means, which can amplify cohort-draw variance.The recommendation concerns paired comparisons of removal outcomes.Source: §6 · p. 6
  • Report the flagged-sample phenotype table alongside the removal curves.The paper presents this as the third reporting rule.Source: §6 · p. 6

7 Conclusion

The paper concludes that attributing held-out disease-related effect size identifies training samples missed by diagnosis-based exclusions, and that removing them improves held-out effects where random removal does not. It also describes the age-gap biomarker’s construction and scope, and discloses the use of language models as coding and writing tools.

7 Conclusion

  • Retargeting attribution from training loss to disease-related effect size identifies samples missed by diagnosis-based exclusions.Source: §7 · p. 6
  • Across four diseases and two modalities, removing flagged samples consistently increases held-out effect size, while random removal does not.The approach may extend beyond Cohen’s d to held-out evaluation statistics when the relevant target differs from the one conventionally optimized.Source: §7 · p. 7

A AI disclosure

  • The authors used LLM assistants for experimentation and plotting code and for language editing, draft condensation, and citation checks; they say authors verified and take responsibility for the content.The disclosure attributes scientific ideas, experimental design, claims, and references to the authors.Source: A AI disclosure · p. 10

B The age-gap biomarker

  • An age-gap biomarker predicts age from biological measurements using a model trained on people presumed healthy; a positive gap means biology is read as older than the person’s years.The gap is predicted age minus true age, with a training-fitted bias correction formalized later.Source: B The age-gap biomarker · p. 10
  • Its disease-related effect size is the case-control separation in age gaps, normalized by pooled standard deviation as Cohen’s d and evaluated on held-out people.Wider gaps have been linked to greater symptom severity and faster progression across conditions.Source: B The age-gap biomarker · p. 10
  • Subclinical disease in a presumed-healthy training pool can distort the learned normal ageing trajectory, motivating direct attribution to downstream effect size rather than training loss.A model with less tightly fit training data can still produce a gap with a larger disease-related effect size.Source: B The age-gap biomarker · p. 10
  • The evaluated modalities are brain age from T1 imaging phenotypes and metabolomic age from NMR blood metabolomics, fit and evaluated identically.Source: B The age-gap biomarker · p. 10

C Experimental setup

The section records pipeline-specific design choices for reproducing the results and omits re-derivations of standard components.

C Experimental setup

  • The section documents compute and pipeline-specific design choices needed to reproduce the reported results.Source: C Experimental setup · p. 11
  • Standard components, including ridge regression, leave-one-out retraining, and matching, follow textbook form and are not re-derived.Source: C Experimental setup · p. 11

C.1 Cohorts and training pool

The cohorts combine brain-age and metabolomic-age data, with disease status defined relative to each assay visit. Training pools are condition-specific, capped by per-seed subsampling, and filtered for missingness and subject coverage.

C.1 Cohorts and training pool

  • Brain age uses 1,440 T1 imaging features from 45,755 subjects, while metabolic age uses 251 NMR features from 98,349 subjects.The brain-age data are from visit 2; the NMR data are from baseline.Source: §C.1 · p. 11
  • Incident diagnoses after the assay visit are excluded, prevalent cases are reserved for influence computation and evaluation, and subjects without a diagnosis record serve as controls.Source: §C.1 · p. 11
  • Each condition’s training pool excludes prevalent and incident subjects for that condition and is capped at 40,000 by a seed-specific subsample.The subsequent 5% feature-missingness and 90% subject-coverage filters remove no subjects in either cohort; Table 2 presents cohort sizes by condition.Source: §C.1 · p. 11, Table 2 · p. 11

C.2 Matching and evaluation split

The evaluation design matches cases to controls on age and sex, then separates attribution scoring from evaluation. The model and attribution comparisons use a frozen ridge-regression penalty and both influence functions and leave-one-out retraining.

C.2 Matching and evaluation split

  • Cases are matched 1:1 to controls on age and sex, reducing mean absolute standardised mean differences from 0.38–0.54 to 0.000 across condition-seed matchings.Matched pairs are split in half: one half computes attribution scores, and the other evaluates effect sizes.Source: §C.2 · p. 11

C.3 Model and attribution

  • The age model is ridge regression on modality features plus sex, with mean imputation, standardisation, and a training-fitted linear age-bias correction.The bias-corrected gap is g = (ŷ − b)/a − y; α is chosen by cross-validation once per seed and frozen for refits.Source: §C.3 · p. 11
  • Each training subject is scored using Cohen’s d or the unnormalised mean case-control age gap, by closed-form influence functions or exact leave-one-out retraining.The leave-one-out comparison uses the standard closed-form ridge update with the intercept unpenalised.Source: §C.3 · p. 11

C.4 Removal experiment and effect sizes

The section describes score-ranked training-sample removal, paired evaluation against random removal, and supporting analyses across conditions, seeds, and influence-score distributions. It also reports the corresponding mean-age-gap comparisons and implementation details.

C.4 Removal experiment and effect sizes

  • Attribution scores determine removal order at 5%, 10%, 25%, and 50% cuts, after which the model is refit at frozen α and evaluation-half Cohen’s d is recomputed.The reported paired change is attribution-arm Cohen’s d minus random-arm Cohen’s d within each seed, averaged with across-seed standard deviation; pairing cancels shared cohort-draw variance.Source: §C.4 · p. 11

C.5 Phenotypic characterisation

  • The top 10% are selected by percentile rank averaged across ten seeds, then matched to remaining training subjects by age and sex for phenotypic comparisons.The bootstrap resamples matched pairs, with 1,000 replicates.Source: §C.5 · p. 12

C.6 Seeds

  • One seed controls each experiment’s stochastic steps, including subsampling, matching, training draws, splits, random removal, and bootstrapping.Source: §C.6 · p. 12

C.7 Computational details

  • The 40 condition-seed tasks used roughly 1.6 compute hours, or about 12.5 core-hours, on CPU-only AMD EPYC nodes.The software stack includes Python 3.12 and scikit-learn 1.8; cohortmatch and pyinfluence are released under the MIT License.Source: §C.7 · p. 12

D Removal effect on the mean age gap across conditions

  • Attribution-based removal leaves case-control mean age-gap separations of 5.2, 1.7, 25.7, and 14.2 y, versus 3.4, 1.5, 8.5, and 5.1 y after random removal.The paired gains are +1.8, +0.2, +17.2, and +9.1 y, respectively; controls remain within 1.2 y of zero, and brain-age and metabolomic-age scales are not directly comparable in years.Source: Figure 4 · p. 13

E Influence distribution

  • Influence-score distributions have a sharply peaked, heavy-tailed shape in every condition and seed, with the dashed line marking the 10% removal cutoff.Source: Figure 5 · p. 14

F Mean age gap attribution

  • Influence-guided removal raises evaluation-half case-control effect size at every removal level in all four conditions, with paired differences reported across seeds.The grey zero line represents random removal by construction; Table 3’s marginal standard deviations are dominated by cohort draw, so the paired change in Figure 6 is the quantity to read for significance.Source: Figure 6 · p. 15, Table 3 · p. 15
  • Table 4 compares mean-age-gap attribution with Cohen’s d attribution in evaluation-half Cohen’s d; positive differences favour mean-age-gap attribution.The comparison is paired within seed, so shared cohort-draw variance cancels; its standard deviations should not be compared directly with marginal standard deviations in other tables.Source: Table 4 · p. 15
Loading 2609.07729v2…