UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

1 Introduction

• Current models exhibit a gap between semantic understanding and pixel-level localization, while existing benchmarks have not systematically evaluated this disconnect in ultrasound.Segmentation benchmarks measure masks, whereas VQA and report-generation benchmarks primarily assess generated answers or text.Source: §1 · p. 2
• UltraG-Bench builds on 40 public datasets across 13 anatomical categories to evaluate instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation.The benchmark includes 138,832 images and 181,952 segmentation masks.Source: §1 · pp. 2–3
• The benchmark evaluates models spanning closed- and open-source general-purpose, medical, and pixel-level grounded MLLMs.The introduction describes a unified evaluation framework for these model families.Source: §1 · p. 3
• UltraG-Agent combines an open-source MLLM’s instruction understanding and task planning with UltraSAM3’s ultrasound segmentation capability, achieving state-of-the-art performance among evaluated models.The paper reports complementarity between MLLMs’ semantic responses and specialized segmentation models’ pixel-level localization.Source: §1 · p. 3

Medical Evidence-Grounded Benchmarks

• Earlier medical evidence-grounded datasets connect text with image regions, but primarily cover radiology, and many datasets focus on a single task rather than ultrasound grounding across tasks.The paper contrasts these datasets with UltraG-Bench’s ultrasound focus and multi-task design.Source: §2 · p. 3
• UltraG-Bench is positioned as a multi-task ultrasound grounding benchmark, with its comparison to representative medical evidence-grounded datasets presented in Table 1.The supplied caption identifies the comparison’s scope but does not provide the table’s cell values.Source: §2 · p. 3, Table 1 · p. 3

Large Vision-Language Models for Medical Evidence Grounding

• Medical MLLMs offer visual understanding and language interaction, while segmentation-enabled models produce pixel-level masks; their ultrasound grounding capability remains underexplored.The related-work discussion describes instruction-driven segmentation approaches including LISA and UniBiomed.Source: §2 · pp. 3–4

3 UltraG-Bench

• UltraG-Bench is built from 40 public ultrasound segmentation datasets spanning 13 anatomical categories to evaluate pixel-level evidence grounding.The construction pipeline is illustrated in Figure 2.Source: §3 · p. 4

3.1 Task Definition

• The three tasks progress from instruction-based target segmentation to VQA with supporting masks and structured reports linked to pixel-level evidence.Grounded reports exclude diagnostic, pathological, and treatment-related inferences unsupported by the available annotations.Source: §3.1 · pp. 4–5

3.2.1 Dataset Collection and Preprocessing

• The collection standardizes source masks and target categories in COCO-style annotations, yielding 138,832 images and 181,952 segmentation masks.The standardized image, category, region, and annotation identifiers support downstream task construction.Source: §3.2.1 · p. 5

3.2.2 Evidence-constrained Annotation

• Evidence-constrained annotation uses existing target names and masks as its factual basis and excludes clinical claims unsupported by pixel-level evidence.Generated instructions, VQA pairs, and reports remain traceably connected to original masks.Source: §3.2.2 · pp. 5–6

3.2.3 Dataset Validation

• Annotation reliability is checked through automated completeness, format, correspondence, and task-alignment tests, followed by independent review of 10% of samples per anatomical structure.Two ultrasound-trained reviewers assess evidence consistency and jointly resolve disagreements.Source: §3.2.3 · p. 6

3.3 Statistics

• UltraG-Bench contains 331,125 segmentation instructions, 666,779 evidence-grounded VQA pairs, and 138,832 grounded reports.Together, the annotations form a multi-level framework from segmentation to question answering and report generation.Source: §3.3 · p. 6

4 UltraG-Agent

• UltraG-Agent combines a pretrained MLLM without task-specific fine-tuning with ultrasound-adapted, prompt-driven UltraSAM3, requiring no additional end-to-end training.The framework combines semantic understanding with fine-grained pixel-level localization.Source: §4 · p. 6
• The MLLM selects among segmentation, VQA, and report generation, then converts the selected target into a concise text prompt for UltraSAM3.It may select one or multiple targets depending on the task.Source: §4 · p. 6
• UltraSAM3 returns masks, and for VQA or report generation the agent uses mask-derived presence, category, location, and extent to constrain text outputs.The agent outputs visualized pixel evidence with the corresponding answer or structured report.Source: §4 · p. 7

5.1 Experimental Setting

• The unified evaluation covered three tasks and compared 14 representative MLLMs, segmentation baselines, and four UltraG-Agent variants.All models were evaluated under the same protocol.Source: §5.1 · p. 7

5.2 Evaluation Metric

• Segmentation uses IoU and Dice; VQA Grounded Accuracy requires a correct answer and sufficiently mask-consistent evidence at τ = 0.5; report SemAcc averages presence, category, location, and size checks.Report generation also uses ROUGE-L for lexical similarity.Source: §5.2 · p. 7

5.3.1 Main Results

• UltraG-AgentI+US3 outperformed the best baseline by 0.2911 IoU, 0.2666 Dice, 0.2292 Acc., 0.3922 G-Acc., 0.1563 ROUGE-L, and 0.3110 SemAcc, leading all six metrics.Compared with UltraSAM3 alone, it raised IoU/Dice from 0.5478/0.6190 to 0.5564/0.6384 while adding VQA and report generation.Source: §5.3.1 · p. 8, Table 2 · p. 8
• Replacing UltraSAM3 with SAM3 substantially reduced performance across segmentation, VQA, and report generation, while different MLLM backbones produced only modest differences.The results identify ultrasound-specific localization as a key factor alongside semantic reasoning.Source: §5.3.1 · pp. 8–9

5.3.2 Fine-grained Organ-level Analysis

• Performance varied across organs and tasks; UltraG-Agent variants with UltraSAM3 showed broader coverage across anatomical categories and more consistent trends across MLLM backbones.The reported advantage was not driven by only a few specific organs.Source: §5.3.2 · p. 9, Figure 4 · p. 9

6 Conclusion

• UltraG-Bench unifies evaluation of instruction-guided segmentation, evidence-grounded VQA, and evidence-grounded report generation; experiments show a gap between existing MLLMs' semantic understanding and pixel-level localization.UltraG-Agent consistently improves both semantic prediction and visual grounding.Source: §6 · p. 9

Reproducibility statement

• Appendices provide dataset and baseline details, benchmark construction and quality validation, agent implementation, evaluation protocols, metric analyses, and case studies; the benchmark and code are reported as anonymously released.The stated release address is anonymous.4open.science.Source: Reproducibility statement · pp. 9–10

The Use of Large Language Models

• ChatGPT was used only for language refinement and writing assistance, with generated content reviewed and revised.Source: The Use of Large Language Models · p. 10

A.1 Datasets Details

• The 40 source datasets contain 138,832 images and 181,952 segmentation masks across training and test splits.Source: §A.1 · p. 14
• Cardiac samples account for 38%–41% and thyroid samples for 26%–27% of annotations, the largest category shares across the three tasks.Fetal, muscle, and prostate categories follow; the remaining categories have smaller shares.Source: §A.1 · p. 14, Figure 5 · p. 14
• Despite differing annotation totals, the three tasks have broadly consistent anatomical compositions, providing comparable category coverage from segmentation through VQA and report generation.Source: §A.1 · p. 14

• The baselines span closed- and open-source general MLLMs, medical MLLMs, pixel-level grounded medical MLLMs, and prompt-driven segmentation models.Source: §A.2 · p. 14
• Claude-4.5 and GPT-5.4 represent closed-source general-purpose MLLMs evaluated for ultrasound understanding and grounding.Source: §A.2 · p. 14
• Open-source general-purpose baselines include Qwen3-VL 4B, 8B, and 32B, MiniCPM-V 2.6 8B, and InternVL3.5 8B and 14B.Source: §A.2 · pp. 14–15
• Table 3 summarizes statistics for the 40 ultrasound datasets retained in the Grounded-US annotations.Source: Table 3 · p. 15
• Medical MLLM baselines include LLaVA-Med, MedGemma 4B and 27B, and Lingshu 7B and 32B.Source: §A.2 · p. 16
• UniBiomed links findings to localized regions, BiomedParse unifies biomedical parsing tasks, and UltraSAM3 provides prompt-driven ultrasound segmentation.Source: §A.2 · p. 16

• Original COCO segmentation annotations yield mask-derived facts about target presence, category, identity, location, extent, area ratio, boundaries, and target count.Source: §A.3 · p. 16
• Mask facts generate segmentation instructions and reports deterministically, while VQA is filtered for clinical relevance and evidence validity, with invalid outputs replaced by rule-based fallbacks.Source: §A.3 · p. 16
• VQA addresses visible-target presence, size, or count; questions about visible targets require mask-geometry grounding, while absent targets permit only absence or not-applicable questions.Source: §A.3 · pp. 16–17
• Rule-based validation checks mask integrity, annotation counts and references, report consistency with masks, and unsupported clinical or metadata claims; critical or major issues flag a sample as failing.Source: §A.3 · p. 17

• An MLLM plans from the image, instruction, and predefined categories, identifying the task and required targets and generating a concise UltraSAM3 prompt for each.Source: §A.4 · p. 17
• UltraSAM3 generates candidate masks and confidence scores for each planned target; invalid candidates are filtered and the highest-scoring mask is selected.Source: §A.4 · p. 17
• Extracted mask evidence conditions responses: VQA answers pair with referenced-mask unions, report statements each link to an evidence mask, and mask facts remain authoritative.Source: §A.4 · pp. 17–18

A.5 Evaluation Details

• MLLMs are evaluated zero-shot on the three tasks with shared prompts and structured outputs; spatial predictions are rasterized and compared with ground-truth masks.Segmentation uses IoU and Dice; VQA Grounded Accuracy requires a correct answer and evidence-mask IoU above 0.5, while reports are scored for text similarity and evidence-supported attributes.Source: §A.5 · p. 18

A.6 Detailed Definition of Semantic Accuracy

• Report Semantic Accuracy equally weights binary judgments of target presence, category, location, and size.Location uses a 3 × 3 image grid; size allows a tolerance of the greater of one pixel or 20% of each reference dimension.Source: §A.6 · pp. 19–20

A.7 Attribute-wise Analysis of Semantic Accuracy

• 0.9614 presence accuracy for UltraG-AgentL+US3; UltraG-AgentI+US3 leads category, location, and size at 0.7254, 0.6462, and 0.4606, respectively.Replacing UltraSAM3 with SAM3 degrades these attributes; the reported gains chiefly reflect stronger alignment with fine-grained spatial evidence.Source: §A.7 · p. 20, Table 4 · p. 21

A.8 Case Study

• In the breast and fetal segmentation cases, UltraG-Agent produces more complete, spatially consistent masks than many standalone MLLMs, while retaining semantic instruction understanding.The examples contrast approximate boxes or incomplete contours from MLLMs with the fine-grained localization of prompt-driven segmentation models.Source: §A.8 · pp. 21–22
• Thyroid and ovarian VQA cases show UltraG-Agent pairing target answers with aligned masks, while other models give unsupported descriptions or inaccurate spatial grounding.In report-generation examples, it also grounds target presence, location, and pixel extent; competing models produce nonspecific, incorrect, or unsupported details.Source: §A.8 · pp. 22–23

A.9 Error Analysis

• A challenging abdominal case shows that weak boundaries and irregular visible extent can defeat segmentation; when the prompt-driven segmenter misidentifies a target, that error propagates into UltraG-Agent’s output.Other systems in the example produce fragmented masks, coarse boxes, shifted regions, or empty predictions.Source: §A.9 · pp. 23–24
• A lung VQA case shows that either incorrect semantic prediction or invalid spatial evidence can fail the joint evaluation, even when the other component appears plausible.The examples include a correct textual location without valid pixel evidence and a plausible mask paired with an incorrect location.Source: §A.9 · p. 24
• A cardiac report case exposes difficulty planning and grounding multiple structures together; target-planning errors can carry through to the final report.The case also includes unsupported clinical attributes without corresponding pixel-level evidence.Source: §A.9 · pp. 24–25