PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

Junqi Liu, Selena Song, Yuhan Wang, Jiawei Mao, Hardy Chen, Xiaoke Huang, Tianhao Qi, Pengfei Guo, Yucheng Tang, Yufan He, Can Zhao, Andriy Myronenko, Dong Yang, Daguang Xu, Yuyin Zhou

arXiv:2606.01961v2cs.AI

TL;DR

Existing benchmarks provide limited visibility into how autonomous agents conduct end-to-end medical-AI research workflows. AutoMedBench evaluates both workflow stages and final outputs, finding that validation remains the weakest stage while agents often set up runnable pipelines.

  • Problem

    It remains unclear whether agentic AI systems can conduct complete medical-AI research workflows requiring domain understanding, engineering execution, and verification.

  • Method

    AutoMedBench evaluates agents across 24 medical imaging and multimodal tasks using five workflow stages, process-level scoring, deterministic metrics, and post-run error diagnosis.

  • Results

    Validation is consistently the weakest stage, while agents often set up runnable pipelines and engineering failures dominate over understanding errors.

  • Takeaways & Limitations

    The benchmark indicates that reliable medical-AI research agents must improve verification of intermediate outputs and recovery from workflow errors.

  • Takeaways & Limitations

    The benchmark excludes subjective-judgment, long-horizon clinical-dialogue, and training-time-adaptation tasks and enforces inference-only runs without training or fine-tuning.

Abstract

from arXiv · show

Autonomous agents are increasingly expected to support end-to-end medical-AI research workflows, moving beyond isolated prediction tasks or short-form clinical question answering. However, existing medical agent benchmarks primarily evaluate final outputs, providing limited visibility into agent behavior within the research process. To address this gap, we present AutoMedBench, a workflow-aware benchmark for autonomous medical-AI research across diverse medical imaging and multimodal inference tasks, organizing agent execution into a unified five-stage workflow (S1-S5): Plan, Setup, Validate, Inference, and Submit. It comprises long-horizon tasks with each run averaging 33 agent turns, spanning five research tracks: segmentation, image enhancement, visual question answering (VQA), report generation, and lesion detection. Each task is evaluated under two difficulty tiers, Lite and Standard, which use the same data and metrics but differ in the amount of task-brief scaffolding, and each run is scored using both final task performance and S1-S5 stage scores, enabling stage-level analysis from the initial task brief to the final submitted artifact. Across thousands of recorded runs, stage-level scoring reveals that Validate is the weakest workflow stage on average, whereas Setup is the strongest, suggesting that current agents are better at making pipelines executable than at verifying their reliability. Post-run error analysis further shows that verification and submission failures dominate tagged errors, accounting for 37.7% and 38.1% of fired codes respectively, whereas task-understanding errors are rare at 0.9%, and runs with one fired error code have a 48% lower overall score than runs with no error code on average.

1 INTRODUCTION

AutoMedBench addresses the lack of end-to-end evaluation for autonomous medical-AI research by organizing agent runs into five workflow stages and exposing process-level failures beyond final outputs. It evaluates heterogeneous imaging and multimodal tasks using interaction traces, stage-level scoring, controlled difficulty tiers, and diagnostic error codes.

  • Motivation: Medical-AI research requires agents to combine clinical understanding with engineering execution across data handling, model selection, dependency resolution, validation, inference, and submission.This makes medical AI a challenging testbed for autonomous research assistance.
  • Motivation: Existing medical and healthcare benchmarks largely emphasize question answering, clinical dialogue, EHR/FHIR interaction, administration, or final task success rather than complete research workflows.These benchmarks provide limited visibility into whether agents can execute end-to-end medical-AI research and evaluation pipelines.
  • Benchmark Design: AutoMedBench organizes each agent run into five stages: Plan, Setup, Validate, Inference, and Submit.The unified workflow mirrors practical medical-AI research processes from task understanding through final submission.
  • Evaluation: AutoMedBench combines final-performance evaluation with stage-level scoring, full interaction traces, and post-run cause-based error codes to diagnose hidden workflow breakdowns.This links progress through the workflow with explanations of where and why runs fail.
  • Contributions: The benchmark covers heterogeneous imaging and multimodal tasks and uses process-level scoring, deterministic held-out evaluation, controlled difficulty tiers, and post-run error diagnosis.These components form the proposed workflow-aware evaluation protocol for autonomous medical-AI research.

2 AUTOMEDBENCH

AutoMedBench is a workflow-aware benchmark for autonomous end-to-end medical-AI research, evaluating agents across five standardized stages and complementary workflow and artifact-quality dimensions. Its design emphasizes reproducible, inference-only evaluation with hidden references, structured submissions, and diagnostic error coding.

  • Shared workflow: Every run follows five stages: Plan, Setup, Validate, Inference, and Submit.Stages S1–S3 capture research decisions, while S4–S5 assess execution completeness and submission validity.
  • Task suite: 24 tasks span five medical research tracks: segmentation, image enhancement, VQA, report generation, and lesion detection.The tasks cover CT, MRI, X-ray, pathology, blood-smear microscopy, dental imaging, and medical video.
  • Task suite: 48 task-tier settings result from evaluating 24 tasks under Lite and Standard difficulty tiers.The tiers use the same task suite while varying the scaffolding provided to the agent, as described in the benchmark design.
  • Execution environment: 0 training or fine-tuning is permitted during a run under the inference-only protocol.Agents may use pre-trained models and approved inference APIs, while policy violations receive zero scores for all S1–S5 stages.
  • Scoring: Two complementary axes evaluate each run: workflow execution and final artifact quality.S1–S3 use LLM-judge evidence from artifacts and traces, whereas S4–S5 use deterministic checks for prediction files and submission schema validity.

3 EXPERIMENTAL SETUP

The experimental setup evaluates six agents across 48 task-tier settings under a fixed interface and unified replication protocol. Runs use controlled environments with fresh workspaces, and log workflow, performance, resource, and diagnostic outcomes for analysis.

  • Agents and coverage: Six frontier base models are evaluated across 48 LITE/STANDARD task-tier settings, using each model directly as the agent in a shared code-execution environment.The evaluation includes both hosted proprietary and open-weight models while controlling the system prompt and tool schema to isolate base-model effects.
  • Agents and coverage: The main evaluation covers 24 active tasks across five medical research tracks under both LITE and STANDARD tiers, yielding 48 reported task-tier settings.Tasks use a public-input/private-reference split, and restricted datasets must be credentialed by runners before benchmark execution.
  • Replication protocol: 288 evaluation cells comprise the main experiments, with each cell defined by an agent, task, and tier and replicated from a fresh workspace.Six agents, 24 tasks, and two tiers determine the cell count; runs share no cache, files, or conversation history, with a default cohort of N=10 runs per cell.
  • Logging and analysis: Every run logs five workflow-stage scores, TASK and OVERALL scores, conversational turns, runtime, token counts, inference cost, status, and the full interaction record.Post-run diagnostics extract all fired E1–E5 error codes from conversation.json and write results to a unified ledger.
  • Workflow scoring: Setup is the strongest workflow stage on average, whereas Validate is the weakest across the six evaluated agents.The stage-level pattern indicates stronger pipeline execution than reliability checking before inference and submission.

4 RESULTS AND ANALYSIS

AutoMedBench separates agents by overall and track-level performance, but no single profile dominates every task or deployment objective. Workflow and error analyses show that validation, verification, and submission remain major weaknesses despite relatively strong setup and workflow completion.

  • Leaderboard and task performance: 15.3 points separate the evaluated agents’ overall scores, ranging from 51.2 to 66.5, while different agents lead different medical tracks.The top overall agent leads segmentation, enhancement, report generation, and detection; another agent leads VQA.
  • Leaderboard and task performance: All evaluated agents score higher on workflow than final task performance, showing that workflow completion does not guarantee strong medical artifacts.Agents can progress through required stages while the resulting task outputs remain substantially weaker.
  • Workflow, tier, and cost analysis: S3 (Validate) has the lowest mean stage score, whereas S2 (Setup) has the highest, indicating stronger environment preparation than meaningful pilot checking.The five-stage breakdown exposes differences that a single final score can hide.
  • Workflow, tier, and cost analysis: 16.3% is GPT-5.4’s relative agentic-score drop under LITE, illustrating that additional task-brief scaffolding does not uniformly improve performance.Four agents improve under LITE, but two perform worse, despite fixed data, metrics, time limit, scoring code, and submission schema.
  • Workflow, tier, and cost analysis: Opus 4.6 has the highest overall score and average cost per run, whereas GLM-5 achieves the second-best overall score at lower average cost.The results support both capability-oriented comparison and resource-aware model selection.
  • Failure modes and recovery: 48% lower overall score is associated with runs containing one fired error code versus none, while two or more fired codes remain in a low-score regime.Verification and submission errors dominate tagged failures, and stronger agents more often recover after multiple errors.

5 RELATED WORK

Prior benchmarks increasingly evaluate agents in interactive, tool-using, execution-heavy environments, while research automation assesses agents as scientific workers. Medical benchmarks largely remain focused on fixed-input knowledge, clinical reasoning, or single-task prediction, motivating workflow-oriented medical research evaluation.

  • Agentic Evaluation: Agent benchmarks evaluate multi-step reasoning and execution through interactive environments with external state, tools, verifiable artifacts, or sandboxed grading.Examples span broad reasoning, browser and operating-system environments, and repository-based tasks.
  • Research Automation: Research-agent studies assess whether LLMs can write code, analyze data, run experiments, or reproduce research artifacts as scientific workers.Most settings target general scientific workflows, machine-learning engineering, or paper replication and reproducibility.
  • Medical Benchmarks: Medical AI benchmarks typically evaluate fixed-input knowledge, clinical reasoning, or single-task prediction across language-centric and multimodal tasks.Examples include exam-style and biomedical question answering, rubric-based health evaluation, and medical imaging benchmarks such as MedMNIST v2.

6 CONCLUSION · APPENDIX CONTENTS

AutoMedBench evaluates autonomous agents’ end-to-end medical-AI research processes and outputs, revealing persistent weaknesses in validation and workflow-error recovery. The appendix documents the benchmark’s scoring, models, resources, tiers, workflow, error codes, task details, and example traces.

  • 6 CONCLUSION: AutoMedBench combines process-level scoring, held-out task metrics, controlled difficulty tiers, and post-run error diagnosis across 24 tasks and five medical research tracks.It uses a shared five-stage workflow to evaluate both agent behavior and final outputs.
  • 6 CONCLUSION: Validation is consistently the weakest stage, while runnable pipeline setup is often successful and engineering failures outweigh understanding errors.The reported bottleneck is verifying intermediate outputs and recovering from workflow errors rather than medical knowledge alone.
  • APPENDIX CONTENTS: The appendix defines per-task scoring and detailed scoring rubrics for evaluating benchmark performance.These sections appear on pages 22–23.
  • APPENDIX CONTENTS: The appendix documents evaluated model details, run resource statistics, and an API price snapshot.These materials cover model configuration, execution resources, and pricing information on pages 24–25.
  • APPENDIX CONTENTS: The appendix provides difficulty-tier details and workflow-step details for interpreting benchmark conditions and execution stages.These sections are listed on pages 26–27.
  • APPENDIX CONTENTS: Error-code definitions explain the categories used for post-run error diagnosis.The error-code section is listed on page 28.
  • APPENDIX CONTENTS: Additional appendix sections provide per-task scoring details and example benchmarking traces.These sections are listed on pages 30–31.

A PER-TASK SCORING DEFINITIONS · B SCORING RUBRICS DETAILS · C EVALUATED MODEL DETAILS

The appendix defines normalized per-task metrics, workflow-scoring procedures, and a controlled evaluation setup spanning hosted proprietary and open-weight models. Metrics are averaged over held-out cases, while workflow rubrics combine artifact-based judgments with deterministic evaluator checks.

  • A PER-TASK SCORING DEFINITIONS: Every task metric is scaled to [0, 1] and averaged over N held-out cases, with missing or unreadable outputs receiving 0 for the affected case.This normalization applies across tracks and explicitly penalizes unavailable outputs at the case level.
  • A PER-TASK SCORING DEFINITIONS: Enhancement uses mean SSIM, while VQA uses normalized exact-match accuracy averaged across cases.Enhancement compares restored image ˆxi with private reference xi; VQA scores 1{ˆai = ai}.
  • A PER-TASK SCORING DEFINITIONS: Report scoring averages BLEU, METEOR, ROUGE-L, F1RadGraph, micro precision, micro recall, and micro F1 within each case before averaging cases.The seven metrics contribute equally through an unweighted case-level mean.
  • A PER-TASK SCORING DEFINITIONS: Detection uses mAP@0.5 under the PASCAL VOC protocol, averaged over the C task classes.The reported quantity is mean average precision at IoU 0.5.
  • B SCORING RUBRICS DETAILS: S1–S3 workflow scores use LLM judgments from saved artifacts and execution traces, whereas S4–S5 use deterministic evaluator checks.The segmentation rubric is presented as a concrete example, with other tracks documented in the public repository.
  • B SCORING RUBRICS DETAILS: Incomplete segmentation runs receive zero Dice credit when any expected patient output is missing, although S4 records partial completion for workflow diagnosis.This separates task-performance credit from diagnostic information about workflow completion.
  • C EVALUATED MODEL DETAILS: The main experiments evaluate six base models across hosted proprietary and open-weight deployment modes using one shared benchmark harness and evaluation configuration.The shared setup includes the prompt template, tool schema, workspace layout, stopping rules, and scoring scripts.

D RUN RESOURCE STATISTICS … F WORKFLOW STEP DETAILS

The supplementary sections quantify per-run resource use and cost accounting, distinguish Lite from Standard through task-brief changes, and detail the evidence and checks required across workflow stages. Costs vary substantially across agents, while the tiers retain identical evaluation infrastructure apart from the brief.

  • D RUN RESOURCE STATISTICS: $19.77, $2.73, $5.85, $3.94, $2.70, and $1.83 are the average overall costs per run for Opus 4.6, GLM-5, Gemini 3.1 Pro, ChatGPT-5.4, MiniMax-M2.5, and Qwen3.5, respectively.Averages exclude Kimi and are weighted by run counts across task-tier settings.
  • D RUN RESOURCE STATISTICS: Average resource-use statistics cover wall-clock time, conversational turns, total LLM tokens, and normalized USD cost per run.The reported averages come from public leaderboard run summaries and exclude Kimi.
  • D.1 API PRICE SNAPSHOT: Cost accounting uses platform-reported run charges rather than multiplying total tokens by text-only token rates.OpenRouter prices were queried on May 28, 2026 and fixed rates omit prompt-cache and negotiated discounts.
  • D RUN RESOURCE STATISTICS: Per-track averages summarize resource use across agents, tiers, and task settings, feeding the cost analysis in Figure 6.Values exclude Kimi and are weighted by run count.
  • E DIFFICULTY TIER DETAILS: Lite and Standard use identical data, held-out references, time limits, metrics, scoring code, submission schema, and workflow; only the task brief changes.The tier distinction is therefore implemented through task-brief scaffolding rather than evaluation infrastructure.
  • F WORKFLOW STEP DETAILS: Each workflow row specifies the agent’s expected work and the artifact or check used by the harness.The expanded workflow table corresponds to the compact workflow description in §2.
  • F WORKFLOW STEP DETAILS: S1–S3 use LLM judge scores, whereas S4–S5 use deterministic checks.This scoring split is consistent with Figure 3 and operationalizes the workflow-stage requirements.

G ERROR-CODE DEFINITIONS

The appendix defines diagnostic post-run error codes for analyzing observed breakdowns in agent traces. Runs may receive multiple codes, which classify non-exclusive failure types rather than serving as exclusive causal labels.

  • Error-code framework: Error codes are assigned after agent interaction ends by examining conversation.json, including prompts, messages, tool calls, outputs, and submitted-file history.The detailed report records every code fired in the trace.
  • Error-code framework: Runs may fire multiple codes when their traces contain multiple breakdown types.The taxonomy is diagnostic only and does not represent a single exclusive cause.
  • Error-code categories: E1 covers solving the wrong problem or choosing an incompatible high-level approach, while E2 covers blockers in data, models, dependencies, APIs, or runtime resources.These categories are applied according to observed evidence in the run.
  • Error-code categories: E3 covers undetected or unrepaired warning signs and invalid outputs, whereas E4 covers pipeline failure during input processing.The supplied rubric passage also begins defining E5 as packaging or submission failure.

H PER-TASK SCORING DETAILS

Figure 9 shows overall, agentic, and task scores across five medical-AI tracks, revealing that overall rankings conceal track-specific specialization. Opus 4.6 leads most tracks, while GLM-5 leads VQA and several agents remain competitive on detection.

  • Track-wise leaderboard breakdown: Overall rankings mask task-track specialization across segmentation, image enhancement, VQA, report generation, and lesion detection.Figure 9 reports overall, agentic, and task scores for each evaluated agent across these five tracks.
  • Track-wise leaderboard breakdown: Opus 4.6 leads most tracks, while GLM-5 leads VQA and several agents remain competitive on detection.

I EXAMPLE BENCHMARKING TRACES

This section presents two real kidney tumor segmentation runs, illustrating benchmark traces for each tier through task text, reports, API usage, and conversation examples. Local paths, backend names, and secret-like strings are redacted.

  • Example runs: Two real kidney tumor segmentation runs are presented as benchmark examples.The section covers one run for each tier.
  • Trace contents: Each tier’s trace includes the task text, run report, API usage, and a short conversation example.
  • Redactions: Local paths, backend names, and secret-like strings are redacted from the displayed traces.

I.1 LITE

The Lite kidney-tumor segmentation run produced valid organ and lesion mask files across 20 public CT cases, but weak lesion performance and an invalid CSV reduced the overall result. Workflow scoring was strongest in Plan, Setup, and Inference, while Validate and Submit were weaker.

  • Results: 0.5320 Overall score accompanied a 0.3073 Task score and 0.7567 Agentic score for the Lite kidney tumor segmentation run.The task used 20 public CT cases and required organ.nii.gz and lesion.nii.gz outputs for each case.
  • Results: 0.9160 Organ Dice contrasted with 0.1045 Lesion Dice, indicating substantially weaker lesion segmentation performance.The goal was to produce valid kidney and lesion masks for held-out scoring.
  • Workflow stages: 1.0000 scores were recorded for S1 Plan, S2 Setup, and S4 Inference, versus 0.5000 for S3 Validate and 0.3166 for S5 Submit.The run therefore received lower stage scores for validation and submission than for planning, setup, and inference.
  • Submission: CSV valid: false, although Submission valid: true and Mask files valid: true.The recorded error was E3 verification/recovery error.
  • Execution: 27 calls, 3 errors, and 718.1 s characterized S4 Inference, the most call-intensive and time-consuming stage.The run totaled 55 API calls, 54 code runs, and 1 submit call.
Loading 2606.01961v2…