Source-linked AI summary
Can Coding Agents Build Robust Baselines? A Skill-Based Approach for Automating the Medical Imaging Model-Development Pipeline
Eugenia Moris, José Ignacio Orlando
TL;DR
Medical imaging baseline development remains expertise-intensive, while prior automation often addresses isolated stages rather than the complete process. This paper presents a literature-guided, skill-based workflow combining automated implementation with hypothesis-driven experimentation, and reports improved validation performance across four tasks with competitive leaderboard results and MIDOG25 domain generalization. The authors also identify limits from single-seed experiments, uncontrolled comparisons, and a predefined orchestration strategy.
Problem
Developing competitive medical imaging baselines requires iterative, expertise-intensive decisions, while existing automation generally optimizes isolated stages instead of the complete development workflow.
Method
The workflow combines literature-guided planning, automated model implementation, and evidence-driven experimentation that iteratively tests hypotheses and updates a persistent experiment record.
Results
Validation performance improved across all four tasks, with final models ranking 6th of 15 teams in both PUMA tracks and 31st of 125 submissions on MILK10k.
Takeaways & Limitations
The same workflow produced competitive baselines across diverse medical imaging tasks without task-specific redesign, while MIDOG25 results remained consistent across species, scanners, and tumor types.
Takeaways & Limitations
Experiments used a single random seed and lacked identical-budget comparisons with human-developed baselines, conventional hyperparameter optimization, or existing AutoML systems.
Abstract
from arXiv · showhide
Developing competitive deep learning baselines for medical imaging remains a highly iterative process requiring literature review, implementation, experimentation, and expert refinement. Existing automation approaches typically optimize isolated components, such as architecture search or hyperparameter tuning, rather than the complete baseline development process. We present an agentic AI Scientist workflow that combines literature-guided reasoning, automated code generation, and hypothesis-driven experimentation to generate competitive baseline models for medical imaging challenges. The framework is evaluated on four public benchmarks spanning segmentation, classification, and detection. Across all tasks, the Experimentation Pipeline consistently improves validation performance, achieving competitive leaderboard results, including 6th place on both PUMA tracks (15 teams) and 31st place on MILK10k (125 teams). On MIDOG25, the resulting model also demonstrates strong domain generalization across scanners, tumor types, and species. Using the same workflow across all challenges without task-specific redesign, we demonstrate that skill-based, literature-guided agentic workflows can substantially reduce the engineering effort required to develop competitive medical imaging baselines.
1 Introduction
Competitive medical imaging baselines require costly, expertise-intensive iteration across many interdependent modeling decisions. The paper addresses this gap with a unified, literature-guided workflow integrating planning, implementation, and evidence-driven experimentation under human supervision.
- Motivation: Developing a competitive medical imaging baseline requires interdependent decisions spanning preprocessing, architecture, optimization, augmentation, loss design, and evaluation.These decisions require both clinical and machine learning expertise, making development costly, time-consuming, and difficult to scale.
- Related Work: Existing automation mainly targets isolated stages such as hyperparameter optimization, neural architecture search, or data augmentation.LLM-based agents have also expanded automation to literature analysis, code generation, experiment execution, and scientific workflows.
- Approach: The proposed workflow integrates literature-guided planning, automated implementation, and evidence-driven experimentation into one supervised process.It is designed specifically for automatically developing competitive medical imaging baselines rather than serving only as a general-purpose research assistant.
- Contribution: Four public challenges spanning segmentation, classification, and object detection were evaluated without task-specific redesign.The workflow produced competitive leaderboard performance and strong baselines within 2–7 days.
2 Methods
The framework uses three sequential, skill-based pipelines to convert raw data and literature evidence into an executable baseline and iteratively refine it. Structured artifacts, a persistent experiment record, and researcher checkpoints support transparency and reproducibility throughout development.
- Workflow Overview: Independent LLM-driven skills exchange structured artifacts and maintain a persistent experiment plan and tree to record the baseline-development process.These artifacts provide a transparent, reproducible record while lightweight review checkpoints keep the researcher in the loop.
- Workflow Overview: Raw inputs become a structured task description and standardized dataset before the three development pipelines operate.The task description persists across phases, and the researcher invokes skills in sequence while reviewing and running proposed experiments.
- SOTA Knowledge Pipeline: The SOTA Knowledge Pipeline synthesizes clinical and machine learning task analysis, literature, challenge reports, and repository information into an initial development plan.Conflicting or uncertain recommendations become experimental hypotheses, while repository inspection identifies reusable and newly implemented components.
- Model Setup Pipeline: The Model Setup Pipeline converts the literature-guided plan into a reproducible training framework containing task-specific data, model, optimization, loss, and evaluation components.It configures the environment, implements dataset loading and preprocessing, instantiates the model, and integrates the components into training.
- Experimentation Pipeline: The Experimentation Pipeline tests hypotheses, interprets results, and prioritizes experiments by expected information gain within a limited computational budget.It establishes challenge and secondary metrics, then begins with learning-rate calibration and RandAugment-based policy optimization rather than unrestricted augmentation search.
3 Experimental Setup
The evaluation covers four benchmark tasks from three public challenges, using reproducible data splitting, official validation metrics, capped experimentation, and leaderboard or held-out testing. Implementation relies on a reusable PyTorch Lightning framework with human review and execution of generated experiments.
- Implementation: Claude Code and Claude Sonnet 4 orchestrated task-specific skills built on a shared PyTorch Lightning training framework.The framework generates task-specific data, model, loss, augmentation, optimizer, and evaluation components across two hardware platforms.
- Human Oversight: Human supervision remained part of the workflow through artifact review, implementation validation, experiment launching, and execution monitoring.The reported experiments required no manual modifications to the generated modeling code.
- Benchmarks and Splits: The workflow was evaluated on four tasks spanning semantic segmentation, multiclass classification, and object detection.Splits use group-aware identifiers when available and reproducible image-level random splits otherwise.
- Evaluation: 100 training runs per challenge was the maximum experimentation budget, with model selection based exclusively on official validation metrics.MIDOG25 evaluation reported Macro-F1 and F1,AMF, while final PUMA and MILK10k models used official challenge servers.
- Evaluation: Table 2 reports validation after learning-rate selection, augmentation, and experiment exploration, alongside final-baseline comparisons with similar and leading submissions.Leaderboard comparisons are subject to the stated test-set comparability caveat.
4 Results and Discussion
Across four tasks, literature-guided planning provided a strong starting point, while evidence-driven experimentation improved validation performance and produced competitive leaderboard results. The workflow also recorded positive and negative evidence, but its integrated design and predefined orchestration limit conclusions about individual components and broader autonomy.
- The initial development plan specified architecture, optimization, resolution, loss, augmentation, evaluation, and prioritized hypotheses from literature and challenge evidence.
- Across all four challenges, experimentation improved validation performance after each refinement stage, with the largest gains from experiment exploration.
- 6th of 15 teams in both PUMA tracks and 31st of 125 submissions on MILK10k were achieved in independent leaderboard evaluations.The comparisons were not controlled.
- 0.757 to 0.855 was the best validation F1 improvement across 30 MIDOG25 experiments.The framework tested hypotheses across optimization, augmentation, architecture, input resolution, and training strategies.
- 17 experiments were accepted and 13 rejected, with the experiment tree preserving negative evidence to guide subsequent decisions.This structured positive and negative evidence distinguishes the tree from tracking systems that primarily organize artifacts and metadata.
- Limitations: The study does not isolate individual pipeline components, uses one random seed without confidence intervals, and lacks matched comparisons with human baselines, conventional tuning, or AutoML.The reported results therefore concern the complete workflow rather than its individual modules.
5 Conclusion
The agentic framework combines literature-guided planning, automated implementation, and evidence-driven experimentation to develop competitive medical imaging baselines. Across diverse challenges, it improved validation performance and maintained competitive results while automating a substantial portion of model development.
- The workflow combines literature-guided planning, automated implementation, and evidence-driven experimentation to transform raw data into trained baselines with lightweight human oversight.
- Four public challenges spanning segmentation, classification, and detection were used to evaluate the framework.
- Across all tasks, the Experimentation Pipeline consistently improved validation performance.
- Automatically generated baselines achieved competitive leaderboard performance, and MIDOG25 results demonstrated generalization without dedicated domain-adaptation strategies.
- The findings indicate that structured AI Scientist workflows can automate a substantial portion of medical imaging machine learning development while maintaining competitive performance across diverse tasks.