MedUAG: Unified Understanding and Generation for Medical Multimodal Models

Zijie Meng; Yuncheng Zhang; Hualiang Wang; Yitian Tang; Xiaotang Gai; Chen Shen; Songtao Jiang; Shaosheng Cao; Jian Wu; Xian Wu; Zuozhu Liu

Source-linked AI summary

MedUAG: Unified Understanding and Generation for Medical Multimodal Models

Zijie Meng, Yuncheng Zhang, Hualiang Wang, Yitian Tang, Xiaotang Gai, Chen Shen, Songtao Jiang, Shaosheng Cao, Jian Wu, Xian Wu, Zuozhu Liu

arXiv:2608.18937v1Aug 19, 2026cs.CLcs.AI

TL;DR

Medical unified understanding and generation lacks comprehensive corpora, benchmarks, and broadly validated models across diverse tasks and modalities. MedUAG addresses these gaps with a large corpus, standardized benchmark, and unified model that performs strongly across understanding and generation tasks, establishing a competitive baseline.

Problem
Medical UAG lacks comprehensive training corpora, evaluation benchmarks, and broadly validated unified models spanning diverse tasks and imaging modalities.
Method
The paper constructs MedUAGCorpus and MedUAGBench, then develops MedUAG as a unified medical multimodal model for systematic training and evaluation.
Results
MedUAG performs strongly across understanding and generation tasks, achieving the best average accuracy of 71.3 against 16 baselines on medical understanding benchmarks.
Takeaways & Limitations
The work establishes a competitive baseline and provides a unified foundation for future medical multimodal research.
Takeaways & Limitations
The model has limitations in some understanding settings and faces a trade-off between structural fidelity and synthesis diversity.

Abstract

from arXiv · show

Recent Multimodal Large Language Models (MLLMs) are rapidly evolving into unified understanding and generation (UAG) frameworks. However, extending these unified paradigms to the medical domain is hindered by: the absence of comprehensive training and evaluation benchmarks, and the lack of broadly validated unified medical model. To address these gaps, we present a comprehensive foundation for medical UAG. First, we construct MedUAGCorpus, the largest unified medical understanding and generation dataset to date, comprising over 6 million instances across 14 imaging modalities. Second, we introduce MedUAGBench, a systematic benchmark that expands medical generation evaluation to 12 diverse tasks under standardized protocols. Finally, leveraging these resources, we develop MedUAG, an end-to-end trained unified medical model. Extensive experiments demonstrate that MedUAG achieves strong performance across a wide array of understanding and generation tasks, establishing a competitive baseline and paving the way for next-generation medical multimodal systems.

I. INTRODUCTION

Medical multimodal models have evolved toward unified understanding and generation, but medical UAG remains limited by incomplete corpora, benchmarks, and validation. MedUAG addresses these gaps with a 12-task benchmark, a corpus exceeding 6 million instances across 14 modalities, and an end-to-end unified model.

Motivation: MLLMs have evolved from isolated understanding or generation systems into unified understanding and generation frameworks across heterogeneous modalities.Unified training across diverse UAG tasks can learn universal representations within a single system and reduce reliance on task-specific fine-tuning.
Challenges: Medical UAG works are hindered by missing comprehensive training corpora and evaluation benchmarks, with limited modality, task, and clinical-application coverage.Existing studies often use narrow selections of common or readily accessible tasks and have limited empirical validation breadth and consistency.
MedUAGBench: 12 tasks are covered by MedUAGBench, extending UniMedVL’s 5 tasks across various modalities under standardized prompts, metrics, and evaluation settings.The benchmark is designed to enable systematic and reproducible assessment of medical generative capability.
MedUAGCorpus: Over 6M instances across 14 medical modalities comprise MedUAGCorpus, a large-scale corpus spanning unified understanding and generation.The corpus includes diverse task types for unified medical modeling.
MedUAG: MedUAG is an end-to-end unified medical model that achieves strong performance across a broad range of understanding and generation tasks.The work presents a unified foundation spanning benchmark, data corpus, and model, establishing a competitive baseline for future research.

II. METHODS · A. Definition of Tasks

The study frames unified medical multimodal modeling around two primary task domains: understanding and generation.

A. Definition of Tasks: The task definition comprises two primary domains: understanding and generation.These domains organize the unified medical multimodal modeling framework.

1) Understanding Tasks: · 2) Generation Tasks: · B. Construction of Datasets

The paper evaluates unified medical models through image understanding and medically meaningful image generation, supported by a systematic, extensible dataset and benchmark construction pipeline. Understanding centers on VQA and medical report generation, while generation spans four paradigms including synthesis.

1) Understanding Tasks:: Understanding tasks assess interpretation of medical images and extraction of clinically relevant semantics.They focus on two primary tasks: Visual Question Answering and Medical Report Generation.
1) Understanding Tasks:: Visual Question Answering answers natural-language questions grounded in specific medical images, evaluating localized diagnostic reasoning and response precision.
1) Understanding Tasks:: Medical Report Generation produces a complete clinical report from medical images, unlike VQA’s question-focused response setting.
B. Construction of Datasets: The dataset construction pipeline includes source data curation, task-specific construction, and unified sample standardization.
B. Construction of Datasets: MedUAG’s architecture uses dual image encoders and a mixture of transformer experts to support medical image understanding and generation within one framework.
2) Generation Tasks:: Generation tasks evaluate production of medically meaningful images under diverse conditions and are categorized into four paradigms.
2) Generation Tasks:: Synthesis creates or edits medical images under semantic guidance through text-based, mask-based, and counterfactual settings.It requires capturing anatomical structure and pathological semantics for plausible generation or clinically specified modification.
B. Construction of Datasets: Figure 3 presents a systematic and extensible pipeline for building the unified corpus and benchmark.

1) Source Data Curation: · 2) Task-Specific Construction: · 3) Unified Sample Standardization:

The corpus is built by curating and normalizing public medical datasets, deriving task-specific samples, and standardizing all data for patient-level training and evaluation splits. These procedures support unified multitask training while preventing leakage and accommodating fixed-size generative inputs.

1) Source Data Curation:: Public datasets are collected according to a task taxonomy and manually reviewed to remove unusable samples and harmonize file structures and annotations.Understanding tasks primarily use Hulu-Med, while generation tasks draw from public synthesis, translation, reconstruction, and prediction datasets.
1) Source Data Curation:: Understanding data mainly comes from Hulu-Med, whereas generation data covers synthesis, translation, reconstruction, and prediction.
2) Task-Specific Construction:: Task-specific construction derives samples matching downstream scenarios by exploiting information available in each dataset.The stage comprises data alignment, source emulation, and a third component not detailed in the supplied passage.
2) Task-Specific Construction:: Data alignment restructures labels and registers images, including cross-modality alignment of the same anatomical structures when necessary.
3) Unified Sample Standardization:: Processed samples are standardized into a unified format and divided into training and benchmark sets at the patient level.All images, slices, and derived samples from one patient are assigned exclusively to one split, preventing training–evaluation leakage.
3) Unified Sample Standardization:: Generation-task images are resized and padded to 512 × 512 to meet fixed input-size requirements of some generative models.
3) Unified Sample Standardization:: Each sample is packaged with its image path, metadata, unique identifier, and task-specific text prompt for consistent processing.This packaging supports unified multitask training and evaluation.

4) Dataset Statistics: · C. Implementation of MedUAG

MedUAGCorpus provides a large, diverse foundation for medical unified understanding and generation, while MedUAG combines modality-specific visual representations, task-specific objectives, and two-stage medical training. Its benchmark reports standardized averages across multiple generation categories and compares models on understanding benchmarks.

4) Dataset Statistics:: MedUAGCorpus contains over 6M training instances, including 1.76M domain-alignment and 4.61M instruction-tuning instances, with more than 5.3M images across 14 anatomical systems and 14 imaging modalities.MedUAGBench is more compact and evaluation-oriented, with smaller system and modality coverage than the corpus.
C. Implementation of MedUAG: MedUAG follows Bagel’s unified architecture, combining a ViT encoder for high-level understanding tokens with a VAE encoder-decoder for low-level image-generation latents.A decoder-only transformer backbone supports two task-specific branches.
C. Implementation of MedUAG: For understanding, MedUAG autoregressively predicts output tokens from text and ViT visual tokens using cross-entropy loss.Here, x denotes input text tokens, v ViT visual tokens, and y the target output sequence.
C. Implementation of MedUAG: For generation, MedUAG predicts target velocity in VAE latent space under a flow-matching objective.The noised latent feature is z_t at timestep t, c is conditioning information, and u_t is target velocity.
C. Implementation of MedUAG: MedUAG decodes text and image outputs through the LM head and VAE decoder, respectively, under an overall joint objective.The passage specifies the output decoders and states that the objectives are combined overall.
C. Implementation of MedUAG: MedUAG trains in two stages: domain alignment first adapts the pretrained backbone medically, followed by unified instruction tuning.The alignment corpus covers medical image reconstruction, text-to-image generation, and image captioning, providing complementary supervision for generation and captioning.
C. Implementation of MedUAG: Unified instruction tuning covers generation tasks including synthesis, translation, reconstruction, and prediction, plus understanding tasks including VQA and MRG.This stage emphasizes task diversity, compositional conditioning, and instruction responsiveness.
C. Implementation of MedUAG: Generation results report task-level macro averages of FID, GFID, BIOCS, LPIPS, MSE, PSNR, and SSIM across four generation categories.Synthesis uses FID, GFID, and BIOCS, while translation, reconstruction, and prediction use LPIPS, MSE, PSNR, and SSIM.

III. EXPERIMENTS · A. Implementation Details · B. Benchmarks, Baselines and Metrics

MedUAG is trained in two stages from BAGEL-7B-MoT and evaluated through broad medical generation and understanding benchmarks. The experiments compare unified and specialized baselines using task-specific generation metrics and accuracy for medical understanding.

A. Implementation Details: The alignment stage uses 1.76M samples: 1.14M reconstruction, 191K text-to-image, and 432K image-to-text samples.These correspond to 64.6%, 10.8%, and 24.5%, respectively.
A. Implementation Details: The SFT stage continues for 20k steps at 2 × 10−5 on 4.61M instruction-tuning samples using 32 NVIDIA H800 80GB GPUs.The instruction-tuning data contains 3.33M generation samples and 1.28M single-image understanding samples.
B. Benchmarks, Baselines and Metrics: MedUAGBench evaluates medical generation with 5,000 test instances across 12 tasks covering diverse imaging scenarios and heterogeneous generation objectives.Medical understanding is evaluated on VQA-RAD, SLAKE, PathVQA, and OmniMedVQA.
B. Benchmarks, Baselines and Metrics: The understanding benchmarks provide complementary reasoning demands across radiology and pathology, broadening assessment of medical visual understanding.The benchmark suite includes VQA-RAD, SLAKE, PathVQA, and OmniMedVQA.
B. Benchmarks, Baselines and Metrics: Generation comparisons include BLIP3-o, UniWorld-V1, Bagel, HealthGPT, and UniMedVL, spanning general-domain unified and medical-domain unified models.Understanding comparisons include proprietary systems such as GPT-4.1, Claude Sonnet 4, and Gemini-2.5-Fl…
B. Benchmarks, Baselines and Metrics: MedUAGBench uses FID, gFID, and BioCS for synthesis, while LPIPS, MSE, PSNR, and SSIM assess translation, reconstruction, and prediction.These metrics measure realism, fidelity, semantic consistency, perceptual similarity, pixel-wise error, reconstruction quality, and structural consistency.
B. Benchmarks, Baselines and Metrics: Medical understanding benchmarks use accuracy, with Qwen3-VL-30B-A3B-Instruct assessing open-ended questions.This provides a standardized metric for the four medical understanding benchmarks.

C. Main Results · 1) Comparison on MedUAGBench:

Table II shows that MedUAG consistently outperforms general-domain and prior medical unified models across most MedUAGBench generation settings, with particularly clear advantages in reconstruction and prediction.

1) Comparison on MedUAGBench:: MedUAG consistently outperforms general-domain unified models across most MedUAGBench generation settings.
1) Comparison on MedUAGBench:: MedUAG consistently outperforms prior medical unified models across most MedUAGBench generation settings.
1) Comparison on MedUAGBench:: Large-scale medical unified training is identified as the benefit underlying MedUAG’s performance.
1) Comparison on MedUAGBench:: MedUAG’s advantage is especially clear on reconstruction tasks.
1) Comparison on MedUAGBench:: MedUAG’s advantage is especially clear on prediction tasks.
1) Comparison on MedUAGBench:: Reconstruction and prediction require preserving anatomical structure and producing clinically grounded outputs.

2) Comparison on Understanding Benchmarks: · D. Ablation Studies · 1) Effect of Domain Alignment:

MedUAG performs strongly on medical understanding benchmarks, achieving the best average accuracy among 16 baselines. Domain alignment improves structure-preserving reconstruction but reduces synthesis performance, exposing a trade-off between anatomical fidelity and generative diversity.

2) Comparison on Understanding Benchmarks:: 71.3 average accuracy is the best result among 16 baselines on medical understanding benchmarks.
2) Comparison on Understanding Benchmarks:: 75.6 on VQA-RAD and 78.0 on SLAKE are MedUAG’s leading benchmark scores.
2) Comparison on Understanding Benchmarks:: MedUAG remains competitive on PathVQA and OmniMedVQA while demonstrating strong visual reasoning across radiology and pathology.
2) Comparison on Understanding Benchmarks:: Understanding data comprises less than 30% of the corpus in both training stages, despite MedUAG being jointly trained for generation and understanding.
1) Effect of Domain Alignment:: Domain alignment is evaluated by comparing models with and without the stage under both from-scratch and Bagel initializations.
1) Effect of Domain Alignment:: Figure 6 compares models trained with different SFT data ratios.
1) Effect of Domain Alignment:: 0.032 lower translation LPIPS and 0.881 higher reconstruction PSNR result from alignment with Bagel initialization.
1) Effect of Domain Alignment:: Alignment provides consistent gains across reconstruction benchmarks but decreases synthesis performance, indicating a trade-off between anatomical fidelity and generative diversity.

2) Effect of Base Model: · 3) Effect of Data Scale: · IV. APPLICATION EXPLORATION

The experiments show that Bagel initialization substantially improves medical understanding and generation, while larger SFT datasets yield broad but eventually saturating gains. In an application study, MedUAG-generated images effectively augment scarce real image-report pairs for medical report generation.

2) Effect of Base Model:: Bagel initialization consistently outperforms training from scratch, with especially large gains in synthesis and reconstruction.Without domain alignment, Bagel reduces gFID from 332.40 to 128.28 and increases reconstruction PSNR by 2.027; its advantage persists after domain alignment.
2) Effect of Base Model:: Bagel remains the optimal foundation after domain alignment for translation.
3) Effect of Data Scale:: Increasing instruction-tuning data improves performance across all tasks, but gains gradually saturate at larger scales.
3) Effect of Data Scale:: Understanding and synthesis benefit more persistently from added SFT data than reconstruction and prediction.The passage attributes this pattern to their smaller representation in the SFT set and greater benefit from increased diversity and coverage.
IV. APPLICATION EXPLORATION: 0.086 average improvement from synthetic augmentation narrows the gap to the 2K-real upper bound to 0.009.MedUAG generates 1K synthetic images from corresponding reports to augment the 1K-real baseline.

V. RELATED WORK · VI. CONCLUSION

Related medical multimodal models have advanced understanding and text generation but remain limited in unified medical generation, while existing medical UAG efforts face restricted data, task coverage, and evaluation. MedUAG addresses these gaps with a unified model, corpus, and benchmark, while identifying remaining limitations and future directions.

V. RELATED WORK: Medical vision-language models support VQA, report generation, and clinical reasoning through image-text alignment, instruction tuning, and domain-specific data.
V. RELATED WORK: Most medical multimodal models remain focused on visual comprehension and text generation rather than directly synthesizing, translating, reconstructing, or manipulating medical images.
V. RELATED WORK: General-domain UAG methods combine language models with generative modules, visual tokens, decoupled encoders, or expert architectures, but are primarily trained on natural images.
V. RELATED WORK: HealthGPT and UniMed-VL advance unified medical AI, yet existing efforts remain constrained by limited data scale, narrow generation coverage, and insufficiently standardized evaluation.
VI. CONCLUSION: MedUAG is presented with MedUAGBench and MedUAGCorpus for systematic training and evaluation of medical understanding and generation, achieving strong performance across both task families.
VI. CONCLUSION: The work reports clear gains in reconstruction and prediction, demonstrating the promise of unified modeling for general-purpose medical AI.
VI. CONCLUSION: Key challenges include limitations in some understanding settings and a trade-off between structural fidelity and synthesis diversity.
VI. CONCLUSION: Future work will improve data composition, task balancing, and training strategies toward more robust and clinically useful medical multimodal systems.