PaperlayerView exact revision on arXiv

Source-linked AI summary

MedAdaptAgent: An Automatic Harness for Adapting Medical Imaging Models to New Sites

Hanxue Gu, Zhonghao Liu, Xingrui Gu, Kang Wang, Yang Yang

arXiv:2610.34884v1cs.CV

TL;DR

Pretrained medical-imaging models can perform poorly at new hospitals because protocols and preprocessing differ, while adaptation choices depend on the model and available resources. MedAdaptAgent plans evidence-guided adjustments and verifies replacements against the pretrained model. Across evaluated transfers, adaptation improves reported segmentation and classification performance, with fewer harmful classification replacements under pretrained-model comparison.

  • Problem

    Medical-imaging models can perform poorly at new hospitals because imaging protocols and preprocessing differ, and effective adaptation depends on architecture, domain shift, labels, and compute.

    Source: §1 · pp. 1–2

  • Method

    MedAdaptAgent turns a supplied model, target data, and free-text request into a compatible adaptation workflow that revises experiments using model-site evidence and gates replacement against the pretrained model.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    Across 10 segmentation transfers, mean Dice rises from 0.483 to 0.774 with 20 labels and no harmful replacements, while across 13 classification transfers mean AUROC rises from 0.810 to 0.837.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Across evaluated transfers, adaptive planning improves on a fixed experiment sequence, pretrained-model comparison reduces harmful updates among identical candidates, and supervised training produces the largest segmentation gains.

    Source: §7 · p. 10

    The four-planner comparison covers one CT-to-MRI transfer, segmentation and planning comparisons use single runs, and clinical translation requires independent patient-level local validation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reusing pretrained medical-imaging models offers an alternative to training from scratch. A model trained for the same task may perform poorly at a new hospital because imaging protocols and preprocessing differ. Effective adaptation depends on the model architecture, domain shift, and available labels and compute. Adaptation can improve scores on a small labeled set yet reduce performance on unseen data. We present MedAdaptAgent, an automatic harness that turns a supplied model, target data, and a free-text request into a new-site adaptation workflow. Its LLM-driven Adaptation Planner uses model-site evidence to choose compatible experiments and revise subsequent adjustments. The harness requires evidence of improvement over the pretrained model, then delivers the selected model as a complete, runnable package. Evaluation spans 26 segmentation and classification transfers across architectures and compute settings, with segmentation annotation budgets ranging from zero-shot transfer to few-shot adaptation. With 20 local annotations, mean segmentation Dice rises from 0.483 to 0.774 across 10 transfers, a 60.1% relative gain, with no harmful replacements. Mean classification AUROC rises from 0.810 to 0.837 across 13 transfers, a 3.4% relative gain. Comparing adapted models with the pretrained model reduces harmful classification replacements by 30.8% versus selecting the highest-scoring model on the available labels, using identical candidates. Seven public-model deployments demonstrate adaptation and reproducible package delivery across contrast and modality shifts. A CT-to-MRI study extends the evaluation to four LLM planners, including two open-weight models. We release the code on GitHub.

1 Introduction

Pretrained medical-imaging models can avoid training from scratch, but hospital and preprocessing shifts can make direct deployment perform poorly. MedAdaptAgent plans compatible adaptation experiments from model-site evidence, verifies replacements against the pretrained model, and packages a supported model or reports a safe stop.

1 Introduction

  • Pretrained models can reduce the need to collect a large annotated cohort, but differences in scanners, protocols, populations, modality, annotation conventions, resolution, or preprocessing can undermine direct deployment.Those shifts can change model inputs, expected outputs, or both, and direct deployment can lead to poor performance or complete failure.Source: §1 · p. 1
  • Adaptation choices depend on model capabilities, domain shift, labels, and compute; a method that helps one shift may harm another, and small labeled-set gains may not generalize to unseen data.These choices also require diagnosing mismatches, configuring experiments, interpreting results, and packaging the selected model, creating a barrier for teams without dedicated ML engineering support.Source: §1 · pp. 1–2
  • Unlike automation centered on model development or candidate selection, new-site deployment starts from a trained model and must address local data, resource constraints, comparison with the pretrained model, and runnable delivery.The harness contribution combines those deployment tasks with evidence-guided adaptation and controlled release.Source: §1 · pp. 2–3
  • MedAdaptAgent turns a free-text request, supplied model, and target data into a model-specific adaptation workflow that revises experiments as evidence accumulates.The planner selects among compatible adjustments, while deterministic controls enforce architecture, budget, candidate identity, and provenance; the Evidence Gate authorizes replacement.Source: §1 · pp. 2–3
  • With 20 labels, mean Dice rises from 0.483 to 0.774 across 10 segmentation transfers, with no harmful replacements; mean AUROC rises from 0.810 to 0.837 across 13 classification transfers.Comparing adapted models with the pretrained model also reduces harmful classification replacements by 30.8% versus selecting the highest-scoring model on the labeled set, using identical candidates.Source: §1 · p. 3

2 Related Work

Related work covers test-time and supervised adaptation, automated candidate selection, and agents for model development. MedAdaptAgent differs by adapting an existing model for a new site under local constraints and comparing proposed replacements with the pretrained model.

2 Related Work

  • Test-time methods adjust pretrained models using unlabeled new-site data, while linear probing, surgical fine-tuning, and weight interpolation provide supervised alternatives.Method behavior depends on batch size, initial performance, normalization layers, and distribution shift; rankings vary across modalities, and adaptation can degrade individual transfers.Source: §2 · p. 3
  • Existing selection approaches include label-free validators and small-label selection, while AutoMiSeg uses Bayesian optimization and a proxy validator to configure segmentation adaptors.MedAdaptAgent instead generates candidates from a supplied model under local resource constraints and compares each proposed replacement with the pretrained model.Source: §2 · p. 3
  • General and medical ML agents automate development tasks such as code modification, data processing, debugging, and training; MedAdaptAgent addresses post-training deployment by selecting registered adjustments and recording decision measurements.The recorded measurements support traceability of model updates.Source: §2 · p. 3
  • The planner's adjustment families are exposed only after model-capability and resource filtering, and normalization/TTA methods have mixed GPU requirements.That family includes both forward-only BN reset and gradient-based methods.Source: Table 1 · p. 4

3 Method: Automatic New-Site Model Adaptation

MedAdaptAgent takes a pretrained model, target-site images, resource budgets, and a deployment objective, then uses model and data evidence to plan compatible adjustments. A deterministic Evidence Gate governs replacement, and the selected model is exported with its inference assets and supporting measurements.

3 Method: Automatic New-Site Model Adaptation

  • The harness accepts a pretrained model, target images, label and compute budgets, and a deployment objective; it returns a complete package or a report that the objective remains unmet.The default objective is improvement over the pretrained model, and a request may specify a minimum performance requirement.Source: §3 · p. 4
  • The Onboarding Analyzer converts the request into a validated deployment contract and inspects model architecture, task, outputs, normalization layers, and head.It profiles supported data formats and selects task interfaces; three-dimensional tasks require registered preprocessing metadata, with missing metadata producing an error.Source: §3 · p. 4
  • Initial predictions and image statistics, model capabilities, and remaining resources inform the feasible action space.Compatibility rules restrict methods to suitable layers, heads, source references, labels, compute, and registered three-dimensional preprocessing.Source: §3 · p. 4
  • At each iteration, the LLM receives deployment state and tool descriptions and returns one structured diagnostic, adjustment, label request, or terminal action.Execution updates measurements, candidate records, and resource usage; malformed, incompatible, or over-budget actions are rejected, and model changes pass through registered tools.Source: §3 · p. 4
  • The action library includes input matching, normalization adaptation, output-layer training on frozen features, LoRA, and partial fine-tuning.Each successful experiment records a candidate model and its measurements for subsequent planning.Source: §3 · p. 4
  • Without labels, the Evidence Gate checks prediction confidence and plausibility; with labels, it compares pretrained and candidate models on the same selection set.Structural and score checks apply, with leave-one-out or held-out evaluation, paired resampling, or a minimum-improvement rule; search and final selection reuse the selection set.Source: §3 · pp. 4–5
  • The deterministic packager exports the selected model with preprocessing, inference code, environment metadata, and a report linking the model to its supporting measurements.Source: §3 · p. 5

4 Experimental Setup

The evaluation covers segmentation and classification transfers across tasks, architectures, annotation budgets, and compute settings. It measures test performance and harmful replacements under defined data splits and compares adaptation and selection baselines.

4 Experimental Setup

  • The main benchmark includes 26 model-site transfers: 13 two-dimensional segmentation and 13 classification pairs, with separate suites for segmentation budgets and planning comparisons.The evaluation varies annotation and compute budgets, experiment planning, model-selection rules, and public-model deployment.Source: §4 · p. 5
  • Segmentation covers six anatomies and multiple architecture families, while classification spans diabetic retinopathy, dermatology, histopathology, chest radiography, and breast histology.The primary budget study uses U-Net across 10 transfers; controlled and stress tests vary architectures and shifts.Source: §4 · p. 5
  • Segmentation budgets are K=0, 1, 5, and 20; zero-label runs use compatible test-time methods, while larger budgets unlock probing and fine-tuning.Test labels remain withheld through selection, and a compute ablation disables supervised training at K=5 and K=20; classification uses 12 labels except 18 for nine-class CRC-VAL.Source: §4 · p. 5
  • In two-dimensional segmentation, K counts labeled images or slices drawn from a shared adaptation pool, and slices from one patient may occur in both partitions.Classification counts images or patches, with patient grouping undocumented in available split records; the external three-dimensional study counts scans and separates adaptation and test scan identifiers.Source: §4 · p. 5
  • The planner study evaluates four LLMs, while broader benchmarks use GPT-5.2; target-test labels are excluded from planner prompts.The fixed-sequence baseline shares the adjustment library and Evidence Gate, but compute use depends on the experiments each policy chooses.Source: §4 · pp. 5–6
  • Outcomes include target-test Dice or AUROC and harmful replacements, defined as updates that reduce test performance relative to the pretrained model.Classification selection compares identical candidate models and labels over four annotation draws; external deployment uses separate withheld test scans, and exported predictions are checked after reload.Source: §4 · p. 6

5 Results

Across the evaluated transfers, adaptation outcomes depend on annotation and compute budgets, model architecture, and the selection strategy. Adaptive planning and comparison with the pretrained model improve results in their respective comparisons, while deployment tests show runnable packages across external models and shifts.

5.1 Adaptation changes with evidence, compute, and model

  • With no target labels, the harness accepts two beneficial updates and stops on eight transfers, raising mean decision value from 0.483 to 0.502.By contrast, each fixed TTA method degrades a majority of the ten transfers.Source: §5.1 · p. 6
  • With one annotation, the harness reaches mean Dice 0.516, while fixed partial fine-tuning degrades seven transfers; using the label for selection reduces harm but misses a training-based Fundus improvement.Training on one Fundus image raises the pretrained model from zero Dice to 0.560 Dice, but the harness’s one-label policy evaluates models adapted without labels.Source: §5.1 · p. 6
  • The harness improves eight transfers while retaining two pretrained models, achieving a mean Dice of 0.774 compared with 0.697 for fixed partial fine-tuning.Source: §5.1 · p. 7
  • With 20 labels, disabling supervised adaptation limits mean Dice to 0.527, while enabling it raises mean Dice to 0.774; architecture also changes which adjustments help.On prostate, the pretrained U-Net is retained, while SegFormer and nnU-Net benefit when supervised adaptation is available; STU-Net benefits from SAR with one label and further from partial fine-tuning. BN-based tools leave LayerNorm-, InstanceNorm-, and GroupNorm-only models unchanged.Source: §5.1 · pp. 7–8
  • Across classification transfers, the harness improves mean AUROC and reduces harmful updates relative to fixed fine-tuning, while head fine-tuning achieves higher mean AUROC.The head-fine-tuning comparison illustrates a trade-off between average performance and frequency of degradation.Source: §5.1 · p. 8

5.2 Adaptive search improves on a fixed sequence

  • The planner achieves mean Dice 0.629 versus 0.596 for a scripted ladder using the same adjustment library and acceptance rules.Both policies improve nine transfers and preserve pretrained-model performance on the other two; the planner configures experiments using model, local-data, and prior-experiment measurements.Source: §5.2 · p. 8

5.3 Comparison with the pretrained model reduces harmful updates

  • Comparing each proposed classification update with the pretrained model reduces harmful replacements from thirteen to nine across 52 decisions and improves mean test AUROC versus selecting the highest labeled-set score.The candidates and labels are identical, and the worst-case loss also decreases; the comparison reduces variation across annotation draws, with standard deviation falling from 0.015 to 0.006.Source: §5.3 · p. 9

5.4 Deployment from external public models

  • Seven public-model configurations produced verified packages, with five adapted predictors and two unchanged pretrained models; a separate CHAOS experiment improved Dice from 0.117 to 0.256.Public-model packages reproduced all ten test masks after reload; the CHAOS experiment used five labeled images and reproduced all twenty checked masks after two independent reloads. The CT-to-MRI comparison also produced adapted models with four LLM planners, including open-weight planners.Source: §5.1 · p. 8, §5.4 · p. 9, §5.2 · p. 9

6 Discussion

The discussion connects model inspection, adaptive search, and pretrained-model comparison to adaptation outcomes, while emphasizing evaluation and deployment boundaries.

6 Discussion

  • Architecture comparisons, budget studies, and matched-candidate analyses support distinct roles for model inspection, adaptive search, and pretrained-model comparison.Each selected model is linked to its experiment record, acceptance decision, and prediction checks after reload.Source: §6 · p. 10
  • Annotations are used both to train adapted models and to evaluate their performance, so planning must account for supported adjustments and compute.Reserving one label for selection reduces tuning failures across the suite but leaves the Fundus model unchanged; training becomes essential to the strongest segmentation gains at larger budgets.Source: §6 · p. 10
  • A selection set that understates gain can reject a beneficial update, while one that overstates gain can accept a harmful update.Harm frequency and severity should be evaluated alongside mean performance; an absolute requirement separately rules out deployment when every tested model falls below the requested minimum.Source: §6 · p. 10
  • The four-planner comparison covers one CT-to-MRI transfer, and the broader benchmarks use GPT-5.2; segmentation and planning comparisons use single runs.Acceptance thresholds are heuristic, search and final selection reuse the labeled evaluation set, and clinical translation requires independent patient-level local validation against intended performance requirements.Source: §6 · p. 10

7 Conclusion

MedAdaptAgent packages adaptation workflows for pretrained medical-imaging models and records evidence supporting the selected model.

7 Conclusion

  • MedAdaptAgent turns a pretrained medical-imaging model and local adaptation request into a runnable package that records experiments and measurements supporting its selected model.Across the evaluated transfers, adaptive planning improves on a fixed sequence, pretrained-model comparison reduces harmful updates among identical candidates, and supervised training produces the largest segmentation gains.Source: §7 · p. 10

A Classification results

The harness improves mean classification AUROC over pretrained models and reduces degraded transfers relative to fixed fine-tuning, while performance and harmful-update frequency vary by model and annotation draw.

A Classification results

  • The harness improves mean AUROC over pretrained models and reduces degraded transfers relative to fixed fine-tuning.Head fine-tuning has higher mean AUROC and a smaller worst-case loss, but harms six transfers versus three for the harness.Source: A Classification results · p. 13
  • For CAMELYON17, the harness retains ResNet18, applies histogram matching to Swin-Tiny, and tunes the ViT-small head; ViT-small AUROC rises from 0.703 to 0.911.The Swin-Tiny preprocessing adjustment improves an already strong pretrained model.Source: A Classification results · pp. 13–14
  • Across four annotation draws, mean AUROC is relatively stable but the selected action changes on 12 of 13 tasks.In one NIH-to-CheXpert draw, the selected model’s test AUROC is 0.617, below the pretrained model’s 0.788.Source: A Classification results · p. 14

B Supporting ablations and controls

Supporting controls examine zero-label selection, architecture dependence of BN-based adaptation, and recoverability in cross-center prostate MRI classification.

Supporting ablations and controls

  • The zero-label selector ablation compares entropy-based selection with the harness on 10 primary U-Net transfers, withholding test labels from every selection rule.Decision value averages test Dice and uses the pretrained-model score when no replacement is made.Source: Table 6 · p. 14
  • The BN-adaptation control reports full-target Dice or AUROC changes by architecture, with exact zeros marking actions that leave the model unchanged.Source: Table 7 · p. 14
  • The prostate csPCa stress test evaluates fine-tuned models on 338 cases outside the 12-label adaptation set; other target rows use all 350 PCNN cases.Each reported delta uses a matched evaluation set.Source: Table 8 · p. 14

C Implementation details

The implementation sets distinct search and fine-tuning budgets, while the Evidence Gate applies evidence and structural checks before replacement. Deployment experiments report adaptation outcomes and package reproducibility.

C Implementation details

  • Classification fine-tuning uses AdamW at learning rate 10^-3 for 80 steps, updating either the output head or all parameters; segmentation partial fine-tuning updates the final 40% of parameterized leaf modules plus the head.Search allows 44 controller steps and 16 candidate trials; public 3-D deployments allow 24 steps and a 75-minute compute budget.Source: C Implementation details · pp. 14–15

D Evidence Gate details

  • At zero labels, segmentation selection uses prediction-derived signals, with structural checks and thresholds governing eligibility; classification checks class collapse and requires a confidence-based gain of 0.03.The coefficient, margin, and percentile settings are manually specified heuristics that require independent calibration to bound harmful-update risk.Source: D Evidence Gate details · p. 15
  • For labeled candidates, paired verification uses bootstrap resamples and requires the lower percentile to exceed zero; when paired evidence is unavailable, replacement requires a 0.05 score improvement.The Gate also applies a mean-gain floor, with an exception for classification probes.Source: D Evidence Gate details · p. 15

E CHAOS adaptation and deployment

  • With five labeled CHAOS selection images, partial fine-tuning improved Dice from 0.117 to 0.256 on the remaining 401 images, and two reloads reproduced all 20 checked masks.Source: E CHAOS adaptation and deployment · p. 16

F Public-model deployment outcomes

  • All seven public-model configurations completed, with five selecting adapted models and two retaining pretrained models; each package reproduced all ten test masks after reload.Delivered packages include the selected model, preprocessing, and inference code.Source: F Public-model deployment outcomes · p. 16

F.1 Comparison of LLM planners

Four planners were compared on a controlled public-model CT-to-MRI transfer. All delivered adapted models with exact reload agreement, while planner choices and scores differed.

F.1 Comparison of LLM planners

  • The comparison held the checkpoint, preprocessing, annotation split, action library, adaptation seed, and four-candidate budget constant across planners.Twenty annotations provided 13 training and seven selection scans; ten separate scans measured test Dice, and each planner was evaluated once.Source: §F.1 · p. 16
  • All four planners delivered adapted models with exact prediction agreement after reload; both open-weight planners recovered segmentation from the pretrained model’s zero target Dice.GPT-5.2, Luna, and Qwen-4B selected LoRA, while Qwen-8B selected partial fine-tuning.Source: §F.1 · p. 16, Table 9 · p. 16

G Per-transfer outcomes

  • Per-transfer tables report segmentation Dice at K=20 and classification AUROC for the annotation draw summarized in Table 5.The segmentation table distinguishes fundus FOV and ROI preprocessing; the classification table uses 12 labels, or 18 for nine-class CRC-VAL.Source: G Per-transfer outcomes · p. 16, Table 10 · p. 17, Table 11 · p. 17

Related papers

Loading 2610.34884v1…