PaperlayerView exact revision on arXiv

Source-linked AI summary

MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images

Ankan Deria, Komal Kumar, Adinath Madhavrao Dukre, Eran Segal, Salman Khan, Imran Razzak

arXiv:2602.06965v2cs.CV

TL;DR

Medical MLLMs remain constrained by limited domain coverage, imperfect modality alignment, and insufficient grounded reasoning. MedMO addresses these gaps with a multi-stage medical post-training pipeline combining multimodal supervision and verifiable spatial rewards. Across diverse tasks and modalities, it reports gains over strong open-source baselines, while stage-wise training can introduce minor task-level shifts.

  • Problem

    Medical MLLMs remain limited by domain coverage, modality alignment, grounded reasoning, and unified cross-modal generalization.

  • Method

    MedMO uses four-stage post-training with large-scale alignment, high-resolution medical fine-tuning, instruction tuning, and reinforcement learning with verifiable rewards.

  • Results

    MedMO reports substantial gains over strong open-source baselines across VQA, text QA, report generation, and grounding benchmarks.

  • Takeaways & Limitations

    MedMO establishes an open-source medical multimodal foundation model spanning visual grounding, clinical reasoning, and language understanding across diverse medical modalities.

  • Takeaways & Limitations

    Stage-wise training introduces minor task-level performance shifts attributed to catastrophic forgetting, motivating improved cross-task retention and broader modality coverage.

Abstract

from arXiv · show

Multimodal large language models have advanced rapidly, but their adoption in medicine is constrained by limited domain coverage, imperfect modality alignment, and insufficient grounded reasoning. We introduce MedMO, a medical multimodal foundation model built on a general MLLM architecture and trained exclusively on large-scale domain-specific data. MedMO uses a multi-stage training recipe that includes cross-modal pretraining to align heterogeneous visual encoders with a medical language backbone, instruction tuning with multi-task supervision spanning captioning, VQA, report generation, retrieval, and bounding-box disease localization, and reinforcement learning with verifiable rewards that combine factuality checks with a box-level GIoU signal to improve spatial grounding and step-by-step reasoning in challenging clinical settings. Across modalities and tasks, MedMO surpasses strong open-source medical baselines. MedMO-8B-Next achieves consistent gains on VQA benchmarks, improving by 6.6% on average over Fleming-VL-8B, including gains of 6.0% on MMMU-Med, 9.8% on PMC-VQA, and 21.3% on MedXpertQA. On text-based QA, it improves by 14.4% over Fleming-VL-8B, driven by gains of 8.4% on MMLU-Med and 30.1% on MedQA. For medical report generation, it improves by 6.7% on MIMIC-CXR. MedMO-8B-Next also demonstrates strong grounding performance, reaching 56.1 IoU on Bacteria, which is a 47.8 IoU gain over Fleming-VL-8B. At smaller scale, MedMO-4B-Next remains competitive and exceeds Fleming-VL-8B across VQA, QA, and report generation. Evaluations spanning radiology, ophthalmology, and pathology microscopy further confirm broad cross-modality generalization. Project is available at https://genmilab.github.io/MedMO-Page

1. Introduction

Medical MLLMs remain limited by domain-specific interpretation, grounding, structured supervision, and broad cross-modal coverage. MedMO addresses these gaps with an open-source, multi-stage foundation model trained on diverse medical data and evaluated across medical understanding tasks.

  • Medical images require precise domain-specific interpretation, expert contextualization, and grounding to clinical knowledge.
  • Existing medical MLLMs often rely on distilled data with weak domain grounding and generative-only supervision, increasing hallucination and inconsistency risks.
  • Current models commonly target individual tasks or narrow modalities rather than unified cross-modal generalization across healthcare imaging.
  • MedMO combines more than 26M samples from 45 datasets spanning diverse medical imaging modalities with aligned biomedical and visual-language data.
  • MedMO uses multi-stage post-training to improve visual grounding, clinical reasoning, and textual alignment in an open-source medical VLM.
  • Extensive evaluations report state-of-the-art performance across medical VQA, report generation, and diagnostic reasoning benchmarks.

2. Related Works

Medical vision-language research has progressed from foundational alignment systems toward explicit spatial grounding, but fine-grained clinical localization remains challenging. This motivates domain-aligned post-training for broader and more reliable medical grounding.

  • Early medical vision-language models aligned specialized visual encoders with pretrained language models, enabling foundational multimodal reasoning.
  • Recent vision-language models increasingly return spatial evidence such as bounding boxes or points aligned with textual queries.
  • Advanced general-purpose VLMs still face challenges on fine-grained, clinically realistic localization tasks, underscoring the need for domain-aligned post-training.

3. Methodology-MedMO

MedMO uses four sequential post-training stages that progress from general medical alignment to high-resolution grounding, instruction tuning, and reinforcement learning. The pipeline combines multimodal supervision with verifiable spatial rewards to improve clinical responses and localization.

  • Methodology-MedMO: MedMO’s methodology consists of general supervised fine-tuning, high-quality medical fine-tuning, instruction and grounding tuning, and reinforcement learning.
  • Overview of Supervised Fine-tuning: Supervised fine-tuning predicts target responses from image-text inputs using conditional next-token likelihood.
  • Overview of Supervised Fine-tuning: MedMO combines a vision encoder, DeepStack vision-language adapter, and language-model decoder to process images and generate textual responses.
  • Stage 1: General Medical SFT: Stage 1 trains on 18.5M multimodal samples spanning medical captioning, VQA, and general visual-language alignment across diverse imaging modalities.
  • Stage 2: High-quality medical image fine-tuning: Stage 2 uses expert-annotated image-text pairs and bounding-box datasets to add localized feature prediction while preserving global image-text alignment.
  • Stage 3: Instruction tuning: Stage 3 applies 4.3M multimodal instruction-response pairs covering captioning, diagnosis, report summarization, and retrieval-based reasoning.
  • Stage 4: Reinforcement Learning: The final reinforcement-learning stage compares sampled responses using reward functions that include label accuracy, bounding-box quality, tag count, and soft-overlap penalties.
  • Stage 4: Reinforcement Learning: The bounding-box reward provides a verifiable spatially grounded signal intended to directly enhance localization performance.

4. Experiments

MedMO is evaluated across medical QA, report generation, grounding, multimodal datasets, and staged post-training. Across these settings, its larger and Next variants generally outperform strong open-source baselines, with progressive training and bounding-box rewards improving performance and spatial grounding.

  • QA benchmarks: MedMO-8B-Next achieves a 72.7% VQA average, outperforming Fleming-VL-8B at 66.1% by +6.6%.It also sets best reported scores on MMMU-Med, VQA-RAD, SLAKE, and OMVQA.
  • QA benchmarks: MedMO-8B-Next achieves a 60.1% Text QA average, outperforming Fleming-VL-8B at 45.7% by +14.4%.It leads on MMLU-Med, MedQA, and MedXpertQA, while MedMO-8B reaches the highest QA average at 61.3%.
  • Report generation: MedMO-8B-Next achieves the highest CIDEr on MIMIC-CXR at 143.4 and CheXpert Plus at 88.3, while outperforming Fleming-VL-8B on most reported metrics.Fleming-VL-8B remains higher on MIMIC-CXR ROUGE-L, 35.7% versus 32.6%.
  • Medical grounding: MedMO-8B-Next achieves the best grounding average at 56.8%, versus 17.2% for Fleming-VL-8B.It reaches 40.5% IoU on DeepLesion and 56.1% IoU on Bacteria, where several baselines score substantially lower.
  • Medical grounding: MedMO-8B-Next scores 77.5% on MedSG multi-view, 78.8% on object tracking, and 71.9% on referring expression grounding.These results exceed the corresponding specialist MedSG-Bench scores of 55.0%, 62.1%, and 60.4%.
  • Ablation studies: Performance improves progressively across post-training stages, with Stage 2 providing the largest gain through high-resolution and diverse medical-data training.Stage 3 further boosts QA and VQA through instruction tuning, while reinforcement learning consistently improves spatial precision and grounding consistency.

5. Conclusion

MedMO unifies visual grounding, clinical reasoning, and language understanding across medical modalities through a four-stage post-training pipeline. Its grounding reward combines geometric localization signals with matching and penalties, while stage-wise training can cause minor task-level shifts.

  • MedMO uses a four-stage pipeline comprising large-scale alignment, high-resolution fine-tuning, instruction tuning, and reinforcement learning with verifiable rewards.
  • Stage-wise training introduces minor task-level performance shifts associated with catastrophic forgetting, motivating improved cross-task retention.
  • The grounding reward evaluates predicted boxes against ground truth using Hungarian matching and geometric metrics.
  • The per-match score blends normalized L1 distance and GIoU with weights wL1 = 5.0 and wG = 2.0.
  • The reward handles empty-ground-truth, missing-prediction, and failed-matching cases with explicit neutral, zero, or unmatched-box treatments.

B. Experimental Details

The experiments use reproducible mixed-precision training on 64× AMD Instinct MI210 GPUs and report stage-specific settings. Stage 1 converges rapidly, with accuracy stabilizing after an early sharp improvement.

  • Experiments use SFT_Trainer and GRPO reinforcement-learning trainers with bfloat16 mixed precision on 64× AMD Instinct MI210 GPUs.
  • Stage 1 training settings are documented in Table 5 and applied exclusively to Stage 1 MedMO.
  • Stage 1 loss drops from ∼11 to < 0.3 within ≈10 steps while mean token accuracy rises to ∼0.95 and then plateaus.
  • Stage 2 training settings are documented in Table 6 and applied exclusively to Stage 2 MedMO.

Training Dynamics

Training dynamics show rapid, stable optimization across high-resolution adaptation and instruction tuning. Stage 2 sharpens multimodal alignment and spatial reasoning, while Stage 3 improves instruction-following and cross-modal reasoning.

  • Stage 2: High-resolution fine-tuning: Stage 2 mean token accuracy increases from ∼0.86 to ∼0.95, loss decreases from ∼0.9 to ∼0.3, and entropy declines from ∼0.65 to ∼0.27.
  • Stage 2: High-resolution fine-tuning: Stage 2 combines VQA, report-generation, grounding, and bounding-box datasets for high-resolution medical image training.
  • Stage 2: High-resolution fine-tuning: This Stage 2 training combination improves fine-grained visual grounding and detailed report synthesis under high-resolution supervision.
  • Stage 3: Instruction tuning: Stage 3 mean token accuracy rises from ∼0.62 to ∼0.69 while training loss decreases from ∼1.7 to ∼1.4.

Datasets Used

MedMO training uses broad medical instruction, reasoning, report-generation, and spatial-supervision datasets. The combined corpus supports factual grounding, text–image comprehension, and bounding-box localization across medical scenarios.

  • Instruction and reasoning datasets: Stage 3 integrates diverse medical QA, reasoning, instruction-following, biomedical, and clinical datasets for instruction tuning.
  • Report-generation datasets: Medical-Meadow-Cord19 and MIMIC-IV-Ext-BHC provide long-form radiology and biomedical report-synthesis supervision.
  • Training effects: These datasets strengthen instruction-tuned reasoning, factual grounding, and text–image comprehension across medical instruction and report-generation scenarios.
  • Spatial-supervision datasets: DAPO training uses bounding-box IoU and label-accuracy rewards derived from these datasets to refine spatial grounding.
  • Spatial-supervision datasets: Stage 4 uses NIH Chest X-ray, DeepLesion, Bacteria Segmentation, CTC, SLAKE, GRAZPEDWRI-DX, and MedSG for explicit spatial supervision.

C. Dataset Collection

MedMO is trained on a unified, large-scale multimodal corpus spanning diverse medical modalities, anatomical systems, tasks, and progressive training stages.

  • Dataset composition: The corpus combines 45 datasets and more than 26M samples across radiology, pathology, ophthalmology, dermatology, and surgical imaging.MedTrinity contributes 18.5M publicly available instruction-following pairs.
  • Training stages: Training progresses from foundational multimodal understanding to high-resolution visual reasoning and grounding, medical text instruction tuning, and reinforcement-learning refinement.Stages 1–2 establish multimodal and spatial capabilities, Stage 3 strengthens clinical language understanding, and Stage 4 refines localization.
  • Modality and anatomical coverage: The dataset covers X-ray, CT, MRI, ultrasound, optical, and nuclear imaging across systems including the chest, brain, heart, liver, kidney, eye, colon, and tissue.Grounding supervision includes NIH Chest X-ray, DeepLesion, Bacteria, Wrist X-ray, CT, and Cell Microscopy datasets.
  • Dataset organization: Table 8 groups the training datasets by their primary role in each stage of MedMO’s pipeline.The roles include captioning, multimodal and text-based instruction tuning, and spatial grounding.
  • Dataset provenance: Several public datasets were already included in MedTrinity and were therefore not trained on separately.The listed datasets include TCGA, VALSET, MAMA-MIA, LLD-MMRI, CPD, CISC, CT-RATE, KIPA22, and PTCGA.

D. Qualitative Results

Figures 17–20 provide qualitative examples of MedMO predictions across diverse medical imaging scenarios, showing both successful cases and challenging clinical examples.

  • Qualitative evaluation: The qualitative visualizations compare representative model predictions with clinical ground-truth annotations across varied medical imaging conditions.They complement the paper’s quantitative analyses by illustrating successful and challenging examples.

E. Overall Training Summary

MedMO’s training pipeline progresses from multimodal alignment through spatial reasoning, instruction-tuned language understanding, and reinforcement-driven grounding refinement. Dataset overviews and qualitative examples support this broad, multi-stage design across medical imaging tasks.

  • Overall training summary: MedMO progresses from general multimodal alignment to high-resolution spatial reasoning and grounding, instruction-tuned language understanding, and reinforcement-driven grounding refinement.Together, these stages form a domain-aware foundation model for diverse medical imaging tasks.
  • Dataset organization: Table 8 organizes MedMO’s datasets by categories tied to captioning, multimodal and text-based instruction tuning, and spatial grounding objectives.The table summarizes how datasets contribute to distinct training goals.
  • Qualitative examples: Figures 17–20 compare medical-image inputs, model text outputs, and ground-truth annotations, highlighting correct predictions and areas for improvement.The examples span varied clinical conditions and complement quantitative analyses.
  • Clinical examples: A thoracic CT example describes lungs, the heart, and a localized abnormal region with ground-glass opacity or consolidation associated with COVID-19 infection.The abnormality is described as adjacent to normal lung tissue, while the image alone does not establish relationship directionality.
  • Clinical examples: A brain MRI example identifies a central abnormality involving the cerebral hemispheres and reports an area ratio of 0.4% of the image.The description places the possible abnormality near the cerebral cortex and adjacent brain tissue.
Loading 2602.06965v2…