Source-linked AI summary
AutoAdapt: Reliable Few-Shot Adaptation under Clinical Distribution Shifts
Song Wang, Jie Peng, Davis Hobley, Zachary Plotkin, Tianlong Chen
TL;DR
Clinical adaptation is difficult because effective strategies vary across tasks and small labeled cohorts make comparisons unreliable. AutoAdapt combines complete recipes using evidence from adaptation patients, and the reported experiments show improvements across clinical settings, including with very few positive patients.
Problem
Adaptation strategies vary across target tasks, while small labeled cohorts can make candidate comparisons unstable and leave apparent advantages dependent on a few patients.
Source: §1 · p. 1Method
AutoAdapt represents checkpoint, update, data, and input choices as complete recipes, then uses patient predictions and other evidence to form a reliability-aware weighted combination.
Source: §1 · p. 2Results
AutoAdapt improved over fixed and existing adaptation strategies across ICU transfer, emergency care, and diagnostic ECG datasets, including settings with one to six positive adaptation patients.
Source: §1 · p. 2Takeaways & Limitations
The experiments support reliable recipe selection as a practical path for reusing clinical models when target labels are scarce and no single adaptation rule is consistently appropriate.
Source: §7 · p. 10Patient-specific fitting and numerical combination-weight estimation use the adaptation cohort, and evaluation is limited to a patient-disjoint cohort.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Large pretrained clinical models provide a practical way to reuse learned prior knowledge across hospitals by adapting models to them. In practice, a target hospital may only have a small labeled patient cohort, a setting commonly referred to as few-shot adaptation. This requires making multiple decisions, such as which pretrained model to adapt, how much of the model to update, and which patients to use. Nevertheless, this process faces two primary challenges. First, the best adaptation strategy varies across clinical tasks. Second, evaluating and comparing candidate strategies becomes unreliable due to the small patient cohort. In this work, we introduce AutoAdapt with two core designs to deal with these challenges. The Adapter defines an extensible space of adaptation recipes, and the Automator forms a weighted recipe combination from evidence within the adaptation patients. We propose a reliability rule to ensure that only the most effective strategy on most available patients will be selected. These selected strategies then form a combination for effective few-shot adaptation. We conduct extensive experiments across critical care, emergency care, and diagnostic datasets, and the results show that AutoAdapt consistently achieves state-of-the-art performance using only a few patients for adaptation.
1 Introduction
Clinical adaptation must contend with task-dependent strategy effectiveness and unstable comparisons on small labeled cohorts. AutoAdapt combines complete adaptation recipes with reliability-aware selection, and reports improvements across clinical datasets, including settings with very few positive adaptation patients.
1 Introduction
- Adaptation choices vary by target task, while comparisons among candidates can be unstable when only a few labeled patients are available.An apparent strategy advantage may be driven by only a small number of patients.Source: §1 · pp. 1–2
- The Adapter jointly specifies checkpoint, update rule, target-data policy, and input route as complete recipes rather than optimizing these choices independently.The recipe space is extensible.Source: §1 · p. 2
- The Automator uses held-out patient predictions, task information, and label-free shift measures to form a task-specific weighted recipe combination.It removes unreliable choices when evidence is insufficient.Source: §1 · p. 2
- AutoAdapt improved over fixed and existing adaptation strategies across ICU transfer, emergency care, and diagnostic ECG datasets, including settings with one to six positive adaptation patients.The evaluation spans multiple clinical settings rather than a single target task.Source: §1 · p. 2
2 Related Work
Prior work supplies pretrained clinical representations and tools for individual parts of transfer, but does not resolve which complete adaptation strategy is reliable for a particular target.
2 Related Work
- Physiological pretraining and general time-series models make source knowledge available, but do not determine which checkpoint to trust for a target outcome.The cited approaches include cross-view or temporal agreement and broadly transferable representations.Source: §2 · p. 2
- Frozen transfer, partial fine-tuning, LoRA, and full fine-tuning are candidate responses to distribution shift, not rules for choosing among them from a small target cohort.The appropriate response depends on assumptions about how much of the input-label relation remains stable.Source: §2 · p. 2
- Existing approaches retrieve patients, transport features, score representations, search updates, or rank models, whereas AutoAdapt compares complete recipes and treats comparison reliability as part of adaptation.AutoAdapt can retain a predefined default when patient-level evidence does not support another recipe.Source: §2 · p. 2
3 Problem Setting
The problem setting considers prediction at a new site using source checkpoints whose input distributions and clinical targets may differ from the target. A small labeled adaptation cohort supports fitting, while evaluation is patient-disjoint.
3 Problem Setting
- A source checkpoint may differ from the target in both the distribution of inputs and the relationship between inputs and clinical outcomes.The setting separates input/population shift from source-target outcome compatibility.Source: §3 · p. 3
- For each target task, patient waveform windows and optional structured context are used to predict whether an event occurs within a specified future horizon.The task specifies an index time, observation length, gap, and prediction horizon.Source: §3 · p. 3
- Patient-specific fitting and numerical combination-weight estimation use the small labeled adaptation cohort, while evaluation uses a patient-disjoint cohort.Evaluation labels do not fit candidate predictors or numerical combination weights.Source: §3 · p. 3
4 Method
AutoAdapt separates recipe construction from recipe selection: the Adapter defines valid, complete adaptation recipes, and the Automator weights them using task and patient-level evidence. A reliability rule blends the weighted recipes with a predefined default when comparison evidence is uncertain.
4.1 Adapter
- The Adapter combines checkpoint, update, target-data, and input-route choices into valid complete recipes, preserving interactions among these choices.Incompatible choices are removed before adaptation; for example, a specialist may work when frozen but overfit after full updating.Source: §4.1 · p. 3
- For each recipe, the Adapter fits a patient-risk predictor using patient-averaged supervised loss and recipe-specific window selection, class weighting, and aggregation.This prevents longer recordings from contributing more independent evidence than other patients.Source: §4.1 · p. 4
4.2 Automator
- The Automator scores complete recipes using a task-and-shift prior and internal patient-held-out predictions, rather than ranking recipe dimensions independently.The prior combines clinical task similarity with label-free representation compatibility; held-out predictions come from folds within the adaptation cohort, without using evaluation patients.Source: §4.2 · pp. 4–5
- A shared scoring map converts recipe and task evidence into regularized combination weights, with structural coefficients fixed across tasks.Recipe features include held-out utility, stability, prior compatibility, prediction disagreement, and update size; task features include positive and negative patient counts and prevalence.Source: §4.2 · p. 5
- A reliability rule maps positive-patient support, bootstrap improvement probability, and leave-one-positive stability to the blend with a predefined default recipe.When the comparison is not identifiable, the default is preserved; otherwise, nonzero-weight recipes are refit on the adaptation cohort and locked before evaluation.Source: §4.2 · p. 5
5 Experimental Setup
The evaluation spans hospital transfer, changes in care setting, and diagnostic ECG acquisition, comparing AutoAdapt with few-shot, existing adaptation, and automatic-composition baselines. The protocol uses dataset-specific input routes and adaptation data, with patient-level evaluation choices described for the binary experiments.
5.1 Datasets And Baselines
- The benchmarks separate hospital transfer from changes in care setting and acquisition, covering ICU, emergency-care, and diagnostic ECG datasets with dataset-specific reporting.Baselines include standard few-shot approaches, existing adaptation methods, and automatic composition rules.Source: §5.1 · pp. 5–6
5.2 Adaptation Protocol
- The protocol uses dataset-specific routes and adaptation cohorts, with disjoint patient splits when identifiers are available and patient-level AUROC for binary experiments.ALOTT gives every method the same 64 patients per task, while the primary MIMIC-IV comparison fixes the route to waveform-only; ECG datasets have no structured EHR route.Source: §5.2 · p. 6
6 Experimental Results
AutoAdapt outperformed the strongest displayed baseline on five of six ALOTT tasks and led the displayed baselines on every MIMIC-IV task. On ALOTT, the full 20-task roster also showed higher mean AUROC than the previous same-cohort reference.
6.1 Main Results on ALOTT ICU Transfer
- AutoAdapt reached .701 mean AUROC versus LoraHub’s .656 across six ALOTT tasks, leading five of six; across 20 tasks, it scored .686 versus .652 for the previous same-cohort reference.OTTEHR was stronger on oxygenation crisis among the six reported tasks.Source: §6.1 · p. 6, Table 1 · p. 7
6.2 Extreme Few-Shot Selection On MIMIC-IV
- With only one to six positive patients across five MIMIC-IV partitions, AutoAdapt led the displayed baselines on all six tasks.The reported finding concerns independent positive-patient support, not the number of correlated waveform windows.Source: §6.2 · pp. 6–7, Table 2 · p. 7
6.3 Cross-Dataset Transfer On MC-MED And Diagnostic ECG
Across clinical and diagnostic shifts, AutoAdapt routes source features according to target compatibility rather than assuming they always transfer. The analyses support complete-recipe selection and reliability-aware routing over independent component ranking or naive few-shot selection.
6.3 Cross-Dataset Transfer On MC-MED And Diagnostic ECG
- AutoAdapt retains the local EHR route on MC-MED unless evidence supports waveform fusion, but uses source waveform features more on ECG-compatible CPSC and Ningbo.Figure 4 compares macro AUROC across four supported MC-MED tasks and five shared diagnostic ECG tasks, using the same target folds within each dataset.Source: §6.3 · p. 7, Figure 4 · p. 8
6.4 Checkpoint And Prediction-Head Analysis
- Checkpoint–head pairings interact: mean AUROC ranges from .505 to .626, and preferred heads change with the representation.The deterioration checkpoint with a cosine prototype is strongest on average, but independent component ranking does not explain the results.Source: §6.4 · p. 8, Figure 3 · p. 8
6.5 Reliability-Aware Selection Analysis
- Naive selection can underperform a fixed recipe; patient-level reliability checks reject unsupported changes, while weighted combinations preserve complementary candidates.Task and shift priors narrow plausible choices, and target prediction geometry helps distinguish complementary candidates from a single noisy winner.Source: §6.5 · p. 8, Figure 3 · p. 8
6.6 Adapter and Automator Ablation Study
- Expanding the Adapter alone explains only the first cumulative gain; the larger improvement appears when the Automator routes alternatives using reliability and shift evidence.Separating checkpoint, update-method, and data-policy evidence avoids dilution, while target geometry distinguishes redundant from complementary predictions.Source: §6.6 · pp. 8–9, Figure 3 · p. 8
6.7 Clinical-Task And Waveform-Shift Analysis
Task relation and waveform compatibility provide distinct but incomplete signals for checkpoint routing. The analyses show gains from combining them, complementary checkpoint knowledge, and stronger selection support as more positive adaptation patients are available.
6.7 Clinical-Task And Waveform-Shift Analysis
- Combining task, waveform, and adaptation evidence raises mean AUROC from .562 to .646 and reduces harmful transfer by 67% relative to adaptation-only routing.Clinical task distance correlates at −.43 with specialist transfer gain, while waveform distance correlates at −.24; each association is informative but incomplete.Source: §6.7 · p. 9
6.8 Checkpoint-Pool Composition Analysis
- Adding the deterioration specialist raises mean routing AUROC from .562 to .641, and the circulatory specialist raises it to .646; the respiratory specialist adds no gain.The respiratory specialist is never selected, supporting a benefit from complementary physiological knowledge rather than checkpoint count alone.Source: §6.8 · p. 10
6.9 Adaptation-Patient Budget And Stability Analysis
- At 32 positive patients, AutoAdapt reaches .747 mean AUROC; with one positive, the support rule always retains the pretrained default.From four positives onward it exceeds both fixed baselines, while different panels continue to select different recipes.Source: §6.9 · p. 10
7 Conclusion
AutoAdapt treats few-shot clinical adaptation as a decision problem involving task-varying recipe choices and limited positive outcomes. Its Adapter and patient-reliability-aware Automator support task-specific predictors, with a predefined default limiting unsupported transfer under larger shifts.
7 Conclusion
- AutoAdapt pairs an extensible Adapter with a patient-reliability-aware Automator to make task-specific predictors from heterogeneous source models under scarce target labels.The conclusion frames reliable recipe selection as a practical path for reusing clinical models when no single adaptation rule is consistently appropriate.Source: §7 · p. 10
A Detailed Task And Data Contracts
The task contracts define labels from clinical events after the waveform cutoff and specify observation, blank, and prediction windows. The primary MIMIC-IV input is waveform data standardized to a fixed channel vocabulary.
A Detailed Task And Data Contracts
- Task labels follow clinical events after the waveform cutoff, with observation, blank, and prediction periods specified in hours.Windows are masked when the future state cannot be observed.Source: A Detailed Task And Data Contracts · p. 13
- Outcomes include new organ support with death for acute deterioration, new pressor initiation for circulatory deterioration, and future in-hospital death for dynamic mortality.SOFA requires a future component increase of at least two, delirium a new positive CAM record, and AKI incident urine-output KDIGO stage 2 or 3.Source: A Detailed Task And Data Contracts · p. 13
- MIMIC-IV input is waveform-only, resampled to 125 Hz and mapped to 62 channels, with missing channels zero-filled and marked by a presence mask.Respiration and aVR aliases were corrected; labels and patient partitions remained unchanged.Source: A Detailed Task And Data Contracts · p. 13
B Reproducibility Details
This section specifies the evaluation cohorts and model configurations, then documents patient-level aggregation, selection criteria, and artifacts used to reproduce the analyses.
B Reproducibility Details
- MIMIC-IV uses five fixed executions with about 20% of patients for adaptation; CPSC uses five native folds, and Ningbo uses three participant-disjoint folds.CPSC patient disjointness cannot be established because identifiers are unavailable; Ningbo uses one ECG per participant.Source: B Reproducibility Details · p. 13
- TF-dual processes five minutes of 62-channel waveform at 125 Hz, using eight Transformer blocks and a 512-dimensional pooled embedding.The checkpoint has 37.7M parameters, with 26.81M in the active classification path.Source: B Reproducibility Details · p. 13
- Pretraining uses 3,500 steps with batch size 512; partial tuning exposes 6,306,305 parameters, LoRA 197,121, and full tuning 26.81M.Partial tuning updates the final two blocks, final LayerNorm, and target head; LoRA updates attention projections and the target head.Source: B Reproducibility Details · p. 13
- Four TF-dual source states are tracked in a manifest, and machine-readable artifacts generate reported values and figures; task-distance concepts are fixed before analysis.The general pretrained encoder has an empty outcome-specific concept set but remains eligible through distribution and adaptation evidence.Source: B Reproducibility Details · p. 13, Table 9 · p. 16
- Patient windows are combined into one risk before metrics, using recipe-fixed rules such as mean, maximum, upper quantile, or bounded temporal summary.Recipe selection uses 5,000 patient bootstrap draws, and main shift analyses use 2,000.Source: B Reproducibility Details · pp. 13, 16
- Binary-task recipe selection uses AUROC with δQ = .01 and a paired safety margin of .02 AUROC; fixed thresholds also govern support and complexity decisions.A source head is reused only for an aligned label.Source: Table 8 · p. 16, B Reproducibility Details · p. 16