MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction

1 INTRODUCTION

• Existing automation often generates items within predefined scopes, while medical benchmark specification must also determine evaluated capabilities, valid annotation support, reference construction, and assessment.Heterogeneous annotation schemas, units, terminology, and label semantics make manual specification time-consuming and expertise-intensive.Source: §1 · pp. 1–2
• MedBenchAgent treats benchmark construction as constrained compilation, using a Benchmark Intermediate Representation (BIR) to record and carry specification decisions from planning into item instantiation.Planning verifies the specification against annotations and medical references before locking the BIR; instantiation then constructs and audits items under that fixed specification.Source: §1 · p. 2
• Task-Space F1 reached 90.9%, versus 79.2–80.0% for direct task induction and 85.1% for prior-guided induction; 994 of 1,000 sampled items from correctly identified tasks passed human audit.Evaluation also found substantial variation across tasks, imaging settings, and metrics that aggregate scores can obscure.Source: §1 · p. 2
• The paper contributes a constrained-compilation formulation, the MedBenchAgent framework and BIR, and empirical analysis of construction and medical VLM evaluation with reusable artifacts.Its stated scope extends automation beyond predefined medical scopes or question structures while preserving annotation and medical constraints.Source: §1 · p. 2

2 RELATED WORK

• Medical VLM benchmarks progressed from manually authored question–answer datasets to scalable reuse of image–caption and classification supervision, then to task-structured evaluation over heterogeneous datasets.Examples named include VQA-RAD and SLAKE, PMC-VQA and OmniMedVQA, and GMAI-MMBench and MMBU, respectively.Source: §2 · p. 2
• General-purpose systems use declarative or agentic construction, whereas medical systems have favored domain-specific, more deterministic workflows.Examples include AutoBencher, BenchmarkAgent, and BenchAgents for general-purpose automation, and EHRBench and Liu et al. (2026a) for medical workflows.Source: §2 · p. 2

3.1 BENCHMARK FORMULATION

• A medical VLM benchmark comprises an evaluation scope, task space, item set, and protocol specifying response conventions and scoring rules.Each item includes an imaging case, evaluated task, model-facing query, and benchmark reference.Source: §3.1 · p. 3
• Source datasets pair imaging cases and annotation collections with schemas that specify units, granularity, label spaces, and field semantics; annotations become references only after semantic matching to a task.Benchmark-level reference provenance and semantics must be recorded rather than treating references as raw copied annotations.Source: §3.1 · p. 3

3.2 DESIDERATA FOR BENCHMARK CONSTRUCTION

• The three desiderata are evidence groundedness, medical alignment, and evaluative utility: tasks and references need annotation support, task expression should fit medical concepts, and assessment should preserve interpretable capability differences.Evaluative utility includes separate assessment across tasks and imaging settings.Source: §3.2 · pp. 3–4

3.3 BENCHMARK CONSTRUCTION AS CONSTRAINED COMPILATION

• Constrained compilation progressively establishes the specification, with each confirmed decision restricting later choices and an explicit intermediate state preserving decisions before item construction.Evaluation scope constrains task categories, annotation semantics constrain prediction targets and reference evidence, and medical knowledge and evaluation requirements constrain task expression and assessment.Source: §3.3 · p. 4

4 MEDBENCHAGENT

• MedBenchAgent separates five-agent planning from instantiation, with Figure 1 and Algorithm 1 formalizing the workflow and stage-wise BIR updates.Planning establishes the specification; instantiation constructs the item set under it.Source: §4 · p. 4

4.1 INPUTS AND BENCHMARK INTERMEDIATE REPRESENTATION

• Inputs include the intended evaluation scope, annotated medical imaging cases, medical references, and an initial task-category prior with annotation-support conditions.The scope comes from a natural-language user requirement; references include clinical guidelines and reporting standards.Source: §4.1 · p. 4
• Each dataset-specific BIR records retained task nodes, supporting evidence, evaluation protocols, sample allocations, and medically grounded item specifications before the state is confirmed and locked.Decomposable tasks expand into distinct prediction targets, and only retained atomic tasks count as evaluation tasks.Source: §4.1 · p. 5

4.2 PLANNING: COMPILING AND LOCKING THE BIR

• SpecificationAgent structures requirements, TaskPlanningAgent retains annotation-supported tasks and expands distinct answer targets, and MedicalAlignAgent designs task items.TaskPlanningAgent selects a compatible primary metric for each retained atomic task.Source: §4.2 · pp. 5–6
• MedicalAlignAgent uses retrieval-augmented generation over medical references to specify question semantics, response spaces, formulations, and reference-answer normalization before the BIR is fixed.Downstream stages may read but cannot revise the confirmed task definitions, evidence, protocols, allocations, or construction specifications.Source: §4.2 · p. 6

4.3 INSTANTIATION: CONSTRUCTING ITEMS UNDER THE LOCKED BIR

• GenerationAgent varies query wording within locked task semantics and authorized evidence, while deterministic code derives reference answers and review screens for item-level problems.The fixed reference is not modified during review.Source: §4.3 · p. 6
• AuditAgent checks annotation support, response-reference consistency, duplication, option-position balance, and task-stratified issues before finalizing the item set.The checks target task drift, answer leakage, ambiguity, and semantic duplication.Source: §4.3 · p. 6

5.1 EXPERIMENTAL SETUP

• The breast-imaging benchmark uses VinDr-Mammo, BrEaST, and BUS-BRA with ACR BI-RADS v2025 knowledge, and evaluates twelve general-purpose and medical VLMs.Planning checkpoints in the reported runs include researcher review; multiple-choice and spatial tasks use different metrics.Source: §5.1 · p. 7

5.2 BENCHMARK CONSTRUCTION ANALYSIS

• 90.9% Task-Space F1, versus 79.2–80.0% for direct induction and 85.1% for prior-guided induction, demonstrates stronger task-space construction with BIR mediation.Precision rises from 76.9% to 87.0% at the same 95.2% recall; all methods achieve 100% Evidence-Mapping Agreement on matched tasks.Source: §5.2 · p. 7
• 994 of 1,000 audited items pass all four criteria, a 99.4% consistency rate, with six failures localized to template semantics and evidence control.The failures comprise four VinDr-Mammo questions asking for BI-RADS categories and two BrEaST diagnosis questions with unsupported additions.Source: §5.2 · p. 8

5.3 MODEL EVALUATION

• Medical VLMs lead by 6.23 points in Overall Acc. but trail by 2.48 points on AC, while 12–24 of 66 model pairs reverse ordering across task and aggregate Acc. measures.Within tasks, 4–15 of 66 model pairs also reverse ordering between Acc. and F1.Source: §5.3 · p. 8, Table 3 · p. 8
• Lingshu-7B leads Qwen2.5-VL-32B in Overall Acc. (47.00% vs. 44.65%), but Qwen2.5-VL-32B performs better on all six spatial measures.Across spatial measures, 28–34 of 66 model pairs reverse ordering relative to Overall Acc.Source: §5.3 · pp. 8–9, Table 4 · p. 8
• Blurring evaluation images decreases Acc. by 5.86–17.97 percentage points and F1 by 6.33–22.61 percentage points across four medical VLMs.The intervention replaces each image with a Gaussian-blurred image while leaving other benchmark components unchanged.Source: §5.3 · p. 9, Table 5 · p. 9

5.4 CROSS-DOMAIN CASE STUDY

• The complete workflow is applied to ISIC 2018 under a distinct dataset and annotation schema to assess cross-domain portability.Figure 3 identifies this case study as dermoscopy instantiation.Source: §5.4 · p. 9, Figure 3 · p. 9

6 CONCLUSION

• MedBenchAgent treats medical VLM benchmark construction as constrained compilation and makes the specification an explicit, auditable object.Its Benchmark Intermediate Representation separates specification planning from downstream item instantiation, enabling stage-wise evaluation of construction decisions.Source: §6 · p. 9
• Open-ended task-space specification is identified as the primary source of construction uncertainty, while evidence mapping and item instantiation are more reliable once benchmark semantics are fixed.Evaluation also reveals task-, metric-, and imaging-dependent variation that aggregate scores can obscure.Source: §6 · p. 9
• Planning checkpoints still involve limited human input; supervisory agents are proposed as a future way to reduce that reliance.The paper also describes the framework as a basis for comparing controlled benchmark compositions and constructing larger disease-, anatomy-, and modality-specific benchmark collections.Source: §6 · p. 10

A.1 RAW TASK PROPOSALS AND EVIDENCE MAPPINGS

• Table 6 transcribes every task retained by TaskPlanningAgent in its initial plans and maps each proposed task to authorized annotation evidence.Count-affecting researcher review decisions are reported separately in Appendix B.2.Source: §A.1 · p. 13

A.2 AUDITED REPRESENTATIVE ITEMS

• Twenty audited items cover every atomic dataset–task pair instantiated across BrEaST, BUS-BRA, and VinDr-Mammo.Each entry shows its source dataset, task definition, representative image, generated query, and benchmark reference answer.Source: §A.2 · p. 13

B.1 REFERENCE TASK SPACE AND TASK-SPACE EVALUATION PROTOCOL

• Direct baselines construct task spaces in a single call from the same evaluation scope and dataset profile, while the MedBenchAgent proposal compiles an initial BIR task tree against annotation evidence.The DeepSeek direct-plus-T0 baseline also receives the initial task definitions; all proposals are compared before human confirmation.Source: §B.1 · pp. 13, 14, 15, 17
• The reference space organizes breast-imaging evaluation into eight workflow phases, but includes separate atomic targets only where public-dataset annotations provide a clear reference standard.The framework retains unsupported phases to show which broader workflow capabilities the available annotations do not cover.Source: §B.1 · pp. 15–16, Figure 6 · p. 17
• Task-space matching counts dataset-conditioned prediction targets one-to-one, treating duplicates as false positives and conflated or omitted targets as false negatives; Evidence-Mapping Agreement checks supporting fields on matched tasks only.Overall precision, recall, and F1 micro-aggregate counts across the three datasets before calculation.Source: §B.1 · pp. 16–17

B.2 TASK-SPACE RESULTS AND DISCREPANCY ANALYSIS

• Matched tasks had reference-consistent supporting-field mappings; MedBenchAgent errors centered on semantic boundaries, while direct baselines more often proposed metadata, shortcuts, or invalid spatial evidence.The account distinguishes task selection, which remained difficult, from evidence mapping after a target was identified.Source: §B.2 · p. 17

B.3 HUMAN ITEM AUDIT AND RESIDUAL ERRORS

• 6 of 1,000 audited items failed at least one criterion, with four task–query-alignment failures and six evaluation-integrity failures.There were no reference-groundedness or unambiguity failures; all six references remained supported by authorized evidence.Source: §B.3 · p. 19

C.1 DATASET, TASK SPACE, AND REALIZED ALLOCATION

• All seven proposed ISIC tasks and their evidence mappings were retained, while source-data eligibility reduced the requested 5,000-item allocation to 4,179 realized items.The realized allocation included all eligible diagnosis and lesion-localization cases, with the remainder spread across five dermoscopic-attribute tasks.Source: §C.1 · p. 19

C.2 ITEM CONSISTENCY AND REPRESENTATIVE EXAMPLES

• All 200 sampled ISIC items passed checks of reference–evidence consistency, task–query consistency, and evaluation-format integrity.Figure 7 shows representative audited items across the seven dermoscopy tasks.Source: §C.2 · p. 19, Figure 7 · p. 20

C.3 MODEL EVALUATION

• No single checkpoint performed uniformly across diagnosis, dermoscopic attributes, and localization, and categorical performance did not reliably predict spatial-localization performance.The relative behavior of general-purpose and medical VLMs also varied across task types.Source: §C.3 · p. 20, Table 10 · p. 21

D.1 EVALUATION PROTOCOL

• Reported Overall scores macro-average task-family scores after fine-grained question types and dataset-specific results are macro-averaged within their respective levels.The protocol separately reports dataset-level task results and BrEaST finding-attribute question types.Source: §D.1 · p. 21
• Multiple-choice responses are scored with Accuracy and Macro-F1, while spatial responses are parsed as bounding boxes and invalid outputs receive zero IoU.Spatial predictions are converted to original-image coordinates using model-specific conventions, and metrics include all evaluated items.Source: §D.1 · p. 21

D.2 PAIRWISE MODEL-ORDERING ANALYSIS

• 12–24 of 66 model pairs (18.2–36.4%) reversed ordering between Overall Accuracy and task-category Accuracy.The reversal counts ranged from 12 for diagnosis to 24 for assessment category.Source: §D.2 · p. 22, Table 11 · p. 22
• 6.1–22.7% of model pairs reversed ordering between Accuracy and F1 within task categories.The category-specific reversal counts ranged from 4 for pathology status to 15 for assessment category and diagnosis.Source: §D.2 · p. 22, Table 11 · p. 22
• 28–34 of 66 model pairs (42.4–51.5%) reversed their ordering between Overall Accuracy and spatial measures across the reported detection and visual-grounding settings.The comparisons cover BUS-BRA and VinDr-Mammo detection and VinDr-Mammo visual grounding.Source: §D.2 · p. 22

D.3 ADDITIONAL ANALYSIS OF MODEL BEHAVIOR

• The additional analyses examine how model comparisons change with evaluation target, metric, imaging setting, scale, and thinking variant.Comparisons refer to the evaluated checkpoints under the reported protocols.Source: §D.3 · p. 22

D.3.1 TASK GRANULARITY AND METRIC-DEPENDENT RANKINGS

• BreastGPT-8B’s BrEaST Macro-F1 ranged from 78.0 on margin and 63.7 on halo to 33.2 on posterior features and 35.0 on calcifications.The aggregate finding-attribute score therefore combines markedly different attribute-level results.Source: §D.3.1 · p. 22, Table 12 · p. 23
• On posterior features and halo, QoQ-Med-7B led BreastGPT-8B in Accuracy (57.9 versus 54.8; 72.0 versus 70.3), while BreastGPT-8B led in Macro-F1 (33.2 versus 25.8; 63.7 versus 41.8).The passage attributes the difference to Accuracy weighting individual items equally and Macro-F1 averaging class-specific F1 scores.Source: Table 12 · p. 23, §D.3.1 · p. 24

D.3.2 CHANGES ACROSS MODEL SCALES AND THINKING VARIANTS

• A 0.60-point Overall Accuracy difference between Qwen3-VL-8B-T and Qwen3-VL-8B-I combined a 27.09-point FA gain with declines in five other tasks.The FA gain contributed approximately 4.52 points to Overall Accuracy, while the other five tasks jointly contributed approximately −3.91 points.Source: §D.3.2 · p. 24
• Halo and skin-thickening gains contributed approximately 18.0 points of the roughly 27.1-point FA improvement, while posterior-feature Accuracy decreased from 37.3 to 22.2.The contribution estimate uses rounded attribute scores and seven equally weighted attributes.Source: Table 12 · p. 23, §D.3.2 · p. 24
• Lingshu-32B improved over Lingshu-7B in Overall Accuracy and FA Macro-F1 but declined in DX Macro-F1; Qwen3-VL-32B improved over Qwen3-VL-8B-I on aggregate multiple-choice performance and all three reported spatial settings.These comparisons show that checkpoint-level aggregate gains can have different task-level compositions.Source: §D.3.2 · p. 24

• 56.3 Accuracy and 34.4 Macro-F1 make Hulu-Med-7B the ISIC diagnosis leader, but Qwen3-VL-8B-I scores 80.3 mIoU and 91.4 Acc@IoU0.5 versus Hulu’s 0.7 and 0.2.Qwen3-VL-8B-I has lower diagnosis scores of 53.2 Accuracy and 22.8 Macro-F1.Source: §D.3.3 · pp. 24–25
• ISIC diagnosis and detection use different annotation subsets, so their comparison describes dataset-level performance, not paired outcomes on individual images.Spatial scores also include response validity under the evaluation protocol.Source: §D.3.3 · p. 25
• BreastGPT-8B leads Qwen3-VL-32B on BUS-BRA detection, 82.80 versus 62.08 Acc@IoU0.5, while Qwen3-VL-32B leads on VinDr-Mammo, 12.86 versus 0.90.The ranking reversal means a model’s advantage in one localization setting need not persist in another.Source: §D.3.3 · p. 25