Source-linked AI summary
From Reasoning to Pixels: Grounded Medical Multimodal LLMs for VQA and Segmentation
Haowen Gu, Gensheng Pei, Junzhu Mao, Qiong Wang, Mingwu Ren, Yazhou Yao
TL;DR
Existing medical multimodal systems often lack pixel-level evidence, while segmentation methods struggle with complex clinical reasoning. MedREAL bridges reasoning and localization through SARP and R2V, achieving 68.49% gIoU and 70.47% cIoU on MedRAVS-13K.
Problem
Existing Med-VQA systems often provide answers without pixel-level evidence, while segmentation methods lack complex clinical reasoning needed for question-driven localization.
Method
MedREAL uses [SEG] tokens, SARP, and R2V fusion to distill reasoning-aware semantic evidence and guide mask decoding.
Results
68.49% gIoU and 70.47% cIoU are achieved on MedRAVS-13K, with evidence masks semantically consistent with predicted clinical answers.
Takeaways & Limitations
MedREAL provides a unified framework for aligning clinical reasoning with spatial evidence across four medical imaging modalities.
Takeaways & Limitations
MedSAM3’s high metrics benefit heavily from explicit ground-truth category prompts and extensive pretraining on the original source datasets.
Abstract
from arXiv · showhide
Although Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in Medical Visual Question Answering (Med-VQA), their reliance on global image features often lacks precise pixel-level grounding, thereby limiting clinical trustworthiness. To bridge the semantic gap between high-level clinical reasoning and spatial localization, we propose \textsc{\textsc{MedREAL}} (\textbf{Med}ical \textbf{RE}asoning-driven \textbf{A}nswering and \textbf{L}ocalization), a unified framework that seamlessly aligns linguistic reasoning with spatial grounding. Specifically, \textsc{MedREAL} introduces \textbf{S}eg \textbf{A}nchored \textbf{R}easoning \textbf{P}ooling (SARP) to distill task-relevant semantic evidence directly from \texttt{[SEG]} tokens within the MLLM's hidden states. Furthermore, a \textbf{R}easoning-to-\textbf{V}isual (R2V) fusion mechanism is proposed to effectively inject these reasoning-aware features into a segmentation pipeline for accurate mask decoding. To facilitate this paradigm, we construct MedRAVS-13K, a comprehensive dataset comprising 13,824 expertly validated samples across four diverse imaging modalities. Extensive experiments demonstrate that \textsc{MedREAL} significantly outperforms state-of-the-arts, achieving 68.49\% gIoU and 70.47\% cIoU on benchmark evaluations. By generating evidence masks that are strictly consistent with textual diagnoses, \textsc{MedREAL} provides a robust, interpretable framework for reasoning-driven medical image analysis.
1 Introduction
MedREAL addresses the gap between Med-VQA reasoning and pixel-level evidence by using explicit [SEG] reasoning anchors, reasoning-aware feature fusion, and a unified segmentation framework. MedRAVS-13K supports this paradigm with 13,824 samples, while experiments report state-of-the-art grounding performance and answer-consistent evidence masks.
- Introduction: Existing Med-VQA systems often answer clinical questions without pixel-level evidence, whereas segmentation methods generally localize spatial regions without complex clinical reasoning.Promptable models such as SAM3 and MedSAM3 [32] enable text-conditioned segmentation but rely largely on shallow lexical alignment.
- Introduction: MedREAL embeds explicit [SEG] tokens as reasoning anchors to distill answer-relevant semantic evidence from MLLM hidden states for pixel-level localization.This design targets the problem that MLLM representations are dominated by global context and broad linguistic patterns, obscuring specific visual evidence.
- Introduction: 68.49% gIoU and 70.47% cIoU: MedREAL achieves state-of-the-art performance while producing evidence masks semantically consistent with predicted clinical answers.These results are reported on MedRAVS-13K and address the interpretability and clinical-trust limitations of answers without pixel-level grounding.
- Introduction: SARP extracts reasoning-relevant features, while R2V fusion injects them into a SAM-based segmentation pipeline for precise, answer-consistent mask decoding.The framework bridges linguistic reasoning and spatial grounding rather than relying on shallow lexical alignment or direct text-embedding concatenation.
- Introduction: MedRAVS-13K contains 13,824 samples across four imaging modalities with clinical question-answer pairs and verified pixel-level masks.The dataset addresses the scarcity of benchmarks that jointly evaluate clinical reasoning and spatial grounding.
2 Related Work
Medical VQA has evolved from fusing static image and language features toward MLLMs with sophisticated cross-modal reasoning. Reasoning-guided segmentation extends semantic reasoning to pixel-level predictions, but prior work largely targets natural scenes and uses a single [seg] token for the target mask [36].
- Medical Visual Question Answering: Medical VQA initially fused static image features with linguistic encodings, whereas recent MLLMs use large-scale pretraining for sophisticated cross-modal reasoning.This shift motivates connecting clinical reasoning with spatial localization.
- Reasoning Guided Segmentation: Reasoning-guided segmentation incorporates high-level semantic reasoning into pixel-level predictions, but existing studies predominantly address natural scenes and represent the target mask with one [seg] token [36].The single-token design enables coarse alignment between language and the target semantic mask.
- Reasoning Guided Segmentation: MedREAL injects reasoning-derived features into segmentation through R2V, making masks directly driven by diagnostic reasoning.The framework combines reasoning-derived features with the segmentation pipeline to align masks with diagnoses.
3 Method
MedREAL unifies medical VQA and pixel-level segmentation in a reasoning-aware latent space, conditioning localized evidence on the MLLM’s diagnostic logic. SARP extracts task-relevant reasoning from [SEG]-anchored hidden states, while R2V fuses it with visual context for semantically guided mask decoding.
- Overall Architecture: MedREAL jointly generates clinical answers and segmentation masks, ensuring localized visual evidence remains semantically aligned with the MLLM’s diagnostic reasoning.The framework prompts a dedicated [SEG] token when localization is required and conditions mask decoding on the resulting reasoning representation.
- Seg-Anchored Reasoning Pooling (SARP): SARP uses the [SEG] hidden state as an attention anchor to aggregate diagnostically relevant semantics from the preceding reasoning sequence rather than relying on a single token.Masked average pooling computes hseg, after which query-key-value attention filters linguistic noise and produces the compact reasoning representation hreason.
- Reasoning-to-Visual Fusion (R2V): R2V fuses hreason with globally pooled multi-scale visual features and projects the combination into a conditional pseudoprompt for the SAM-based mask decoder.This design supplies anatomical context while replacing geometric prompts such as points or boxes with semantically informed guidance.
- Training Objective: MedREAL is trained end-to-end with a weighted objective combining autoregressive text cross-entropy with mask Binary Cross-Entropy and Dice losses.The Dice component addresses class imbalance in medical lesions, while the composite objective aligns linguistic logic with pixel-level boundaries.
4 Dataset
MedRAVS-13K combines heterogeneous medical images across modalities and organs to support reasoning-driven segmentation. Its generation pipeline grounds clinically generated QA rationales in annotated regions using [SEG] tokens and applies automated plus manual quality assurance.
- Dataset composition: MedRAVS-13K spans ultrasound, X-ray, dermoscopy, and endoscopy across breast, lung, skin, and colon imaging, creating a challenging multi-modality testbed.The data cover heterogeneous imaging physics and lesion morphologies, from low-contrast sonographic boundaries to intricate endoscopic mucosal textures.
- Dataset generation: The dataset pipeline combines raw medical images and ground-truth masks with modality- and organ-specific prompts to generate reasoning-aware QA pairs.The process integrates images and masks, generates reasoning-aware text, and includes rigorous quality assurance.
- Dataset generation: Each generated sequence explicitly embeds a [SEG] token to denote the region of interest and anchor textual rationale to annotated anatomy or lesions.The token mechanism is designed to keep the textual rationale grounded in the segmentation annotations.
- Quality assurance: Quality assurance combines automated filtering for medical plausibility, semantic consistency, and visual relevance with manual verification of 10% of samples for clinical and spatial alignment.The manual review checks both clinical accuracy and alignment between textual queries and segmentation masks.
5 Experiments
Experiments evaluate MedREAL on MedRAVS-13K against structural, agent-based, and reasoning-segmentation baselines, showing that reasoning-aware spatial grounding improves robustness and mask–diagnosis consistency. Ablations further identify spatial priors, SARP–R2V coupling, and larger reasoning backbones as important contributors.
- Comparative Evaluation: 55.71% overall gIoU for MedSAM3-Agent, versus 70.99% for oracle-prompted MedSAM3, exposes the severe cost of replacing explicit category labels with autonomous reasoning.On COVID-QU-Ex, MedSAM3-Agent falls from 71.56% to 46.88%, showing the vulnerability of the decoupled reasoning setup.
- Ablation Study: 45.02% gIoU with Box Guidance alone rises from 11.35% without the three components, but SARP without R2V reaches only 45.50% gIoU and remains weak in cIoU.These results show that coarse spatial priors help substantially, while precise semantic alignment requires coupling reasoning extraction with visual fusion.
- Scaling Behavior: 75.96% gIoU versus 68.49% follows from replacing Qwen3-VL-2B-Instruct with the frozen architecture’s 4B-Instruct backbone, confirming gains from MLLM scaling.The experiment attributes the improvement to stronger reasoning capabilities translated into localization.
- Robustness across Modalities: MedREAL produces precise, artifact-robust masks across BUSI, COVID-QU-Ex, ISIC-2018, and Kvasir-SEG, whereas competing models fail under low contrast, diffuse infections, visual noise, or irregular boundaries.MedREAL feeds diagnostic reasoning into the visual decoder through SARP and R2V, supporting artifact removal and boundary accuracy.
- Reasoning-to-Mask Consistency: MedREAL localizes pathological regions according to clinical logic rather than visual dominance, improving consistency between generated VQA text and segmentation masks over LISA.Figure 6 highlights LISA’s mismatch between incorrect text predictions and false-positive masks, whereas MedREAL maintains semantic consistency.
6 Conclusion
MedREAL bridges high-level diagnostic reasoning and pixel-level spatial grounding in medical multimodal large language models. It does so by combining SARP and R2V Fusion to extract localized semantic evidence and guide precise mask decoding.
- Conclusion: MedREAL bridges the semantic gap between diagnostic reasoning and pixel-level spatial grounding in medical multimodal large language models.It is presented as a unified framework for connecting high-level reasoning with spatial localization.
- Conclusion: SARP distills localized semantic evidence from the model’s internal reasoning sequence.The mechanism is explicitly Seg-Anchored Reasoning Pooling.
- Conclusion: R2V Fusion uses reasoning-aware evidence to guide precise mask decoding.Together with SARP, it connects internal reasoning to visual segmentation outputs.
- Conclusion: The framework is supported by the newly curated MedRAVS-13K benchmark.