Source-linked AI summary
HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Tianwei Lin, Wenqiao Zhang, Sijing Li, Yuqian Yuan, Binhe Yu, Haoyuan Li, Wanggui He, Hao Jiang, Mengze Li, Xiaohui Song, Siliang Tang, Jun Xiao, Hui Lin, Yueting Zhuang, Beng Chin Ooi
TL;DR
Medical LVLMs have largely focused on text-based comprehension, while unified medical comprehension and generation remains constrained by limited data and task conflicts. HealthGPT progressively adapts a pretrained LLM with H-LoRA, hierarchical visual perception, staged learning, and VL-Health. Experiments report strong performance across multiple medical comprehension and generation tasks, including improvements over existing approaches.
Problem
Medical LVLMs mainly produce text for comprehension tasks, while unified medical comprehension and generation remains underexplored amid limited medical data and task conflicts.
Method
HealthGPT progressively adapts a pretrained LLM using H-LoRA, hierarchical visual perception, a three-stage learning strategy, and the VL-Health dataset.
Results
HealthGPT achieves performance comparable to or better than existing state-of-the-art models across multiple medical comprehension and generation metrics.
Takeaways & Limitations
HealthGPT unifies medical vision-language comprehension and generation within a single autoregressive Med-LVLM framework.
Abstract
from arXiv · showhide
We present HealthGPT, a powerful Medical Large Vision-Language Model (Med-LVLM) that integrates medical visual comprehension and generation capabilities within a unified autoregressive paradigm. Our bootstrapping philosophy is to progressively adapt heterogeneous comprehension and generation knowledge to pre-trained large language models (LLMs). This is achieved through a novel heterogeneous low-rank adaptation (H-LoRA) technique, which is complemented by a tailored hierarchical visual perception approach and a three-stage learning strategy. To effectively learn the HealthGPT, we devise a comprehensive medical domain-specific comprehension and generation dataset called VL-Health. Experimental results demonstrate exceptional performance and scalability of HealthGPT in medical visual unified tasks. Our project can be accessed at https://github.com/DCDmllm/HealthGPT.
1 Introduction
Medical LVLMs have mainly handled visual comprehension with text outputs, leaving medical visual generation underdeveloped. HealthGPT addresses this gap by progressively adapting a pretrained LLM with task-decoupled knowledge, hierarchical perception, staged learning, and VL-Health.
- Medical LVLMs primarily support comprehension tasks such as visual question answering and report generation, but lack medical visual “drawing” capability.
- Unified medical LVLMs face limited high-scale, high-quality data and conflicts between comprehension and generation tasks.
- HealthGPT progressively adapts a pretrained LLM using H-LoRA plugins that separately store comprehension and generation knowledge.
- Hierarchical Visual Perception preserves visual features from anterior and posterior ViT layers to accommodate different visual granularities across tasks.
- The three-stage strategy first trains task-specific knowledge, then builds fusion layers and output heads through minimal mixed-task training before further fine-tuning.
- VL-Health contains seven comprehension tasks and five generation tasks for training unified medical LVLMs.
2 Related Work
Medical vision-language models have improved medical-image understanding and query answering through specialized encoders, pretrained language models, and interdisciplinary data. However, language adaptability and dataset specificity remain unresolved concerns in prior systems.
- Medical vision-language models combine medical visual encoders with fine-tuned language models to answer questions about medical images.
- BiomedGPT uses BERT-style encoding, GPT-style decoding, and interdisciplinary pretraining while reducing model size relative to Med-PaLM.
- Prior medical vision-language systems still face language adaptability and dataset specificity issues.
3 Preliminaries
LVLMs encode images and text into token sequences and model their joint probability autoregressively. Related visual generation uses VQGAN to quantize images into discrete indices that can supervise image reconstruction.
- A LVLM converts an image through a visual encoder and maps text into a sequence of text tokens.
- The LLM models the joint probability of visual and textual tokens in the unified sequence U.
- LVLMs iteratively generate each response token conditioned on preceding response tokens and optimize cross-entropy loss.
- HealthGPT uses discrete tokens for both text and visual outputs, unifying comprehension and generation as an autoregressive task.
- VQGAN encodes images into latent representations, quantizes them with a codebook, and decodes predicted indices into images.
- LoRA adapts downstream tasks with low-rank adapters whose update impact is controlled by the scaling factor α/r.
4 HealthGPT
HealthGPT unifies medical visual comprehension and generation by representing both text and visual outputs as discrete tokens in one autoregressive framework. It combines hierarchical visual features with task-routed H-LoRA modules and staged training to support diverse medical tasks.
- Unified Autoregressive Generation: HealthGPT represents text and visual outputs with discrete tokens, treating medical comprehension and generation as autoregressive tasks.Comprehension produces text tokens, while generation produces VQGAN index tokens that are decoded into images.
- Hierarchical Visual Perception: Hierarchical ViT features are divided into concrete-grained representations for generation and abstract-grained representations for comprehension.The task type selects the corresponding feature set before image and text features are concatenated into the LLM input sequence.
- Heterogeneous Knowledge Adaptation: H-LoRA stores comprehension and generation knowledge in separate modules and dynamically routes task-relevant information.Its feature-level mechanism merges low-rank expert matrices and allocates routing weights based on the input hidden state.
- Dataset and Training: VL-Health covers seven comprehension tasks and five generation tasks, while later training uses task-specific data for medical comprehension and image-generation applications.The third stage trains H-LoRA and adapter modules after fine-tuning the word embedding layer and output head.
- Three-Stage Learning: The first training stage uses separate visual adapters and H-LoRA submodules to align medical visual features with text or support visual generation.Comprehension trains abstract-grained adapters, whereas generation trains concrete-grained adapters while retaining other components according to the staged setup.
- Experiments: HealthGPT is evaluated against LVLMs and unified multimodal models on medical comprehension and modality-conversion tasks.The supplied table descriptions identify medical visual comprehension comparisons and four modality-conversion experiments.
5 Experiments
Experiments evaluate HealthGPT on medical comprehension, generation, and ablation settings. Results report strong performance across tasks, with H-LoRA and task-specific visual perception contributing to the model’s behavior.
- Main Results: HealthGPT-M3 scores 61.3 on the medical multi-modal unified task, compared with 33.8 for Unified-IO 2.
- Main Results: HealthGPT-L14 reaches 66.4 in the larger model configuration, demonstrating scalability of the method.
- Generation: HealthGPT-M3 achieves an SSIM of 79.38 on CT2MRI-Brain, versus 71.09 for Pix2Pix and 71.47 for DiffMa.
- Generation: On IXI 4× super-resolution, HealthGPT-M3 records SSIM 78.19, PSNR 32.76, and ISE 34.47, with a lowest score of 12.34.
- Ablation Studies: H-LoRA outperforms LoRA and MoELoRA in most comprehension tasks and all generation tasks, while requiring approximately 50% more training time than LoRA.
- Ablation Studies: HealthGPT generates CXR images from instructions describing injury severity and location using a small amount of MIMIC-CXR data.
- Ablation Studies: Comprehension converges more efficiently with abstract-grained inputs, whereas generation performs better with concrete-grained inputs.
6 Conclusion
The paper concludes that HealthGPT unifies medical vision-language comprehension and generation through heterogeneous knowledge adaptation. Its appendices provide implementation, H-LoRA analysis, supplementary validation, and architectural details.
- Conclusion: HealthGPT is presented as a Med-LVLM unifying medical vision-language comprehension and generation through heterogeneous knowledge adaptation.
- Appendix Overview: The appendix documents experimental implementation details, HealthGPT training, and the VL-Health dataset.
- Appendix Overview: A dedicated appendix section systematically analyzes Heterogeneous Low-Rank Adaptation.
- Appendix Overview: Supplementary experiments validate the effectiveness of HealthGPT.
- Model Details: HealthGPT uses shallow and deep CLIP-L/14 features as concrete- and abstract-grained visual tokens aligned with text through two-layer MLP adapters.
- Model Details: The model expands the LLM vocabulary with 8192 VQ indices from VQGAN-f8-8192 for multimodal input understanding.
A.2 Training Details
Training uses a three-stage strategy supported by VL-Health data construction and unified instruction-response formatting. The appendix also reports occasional loss spikes associated with larger models and learning rates.
- Training Strategy: HealthGPT uses a three-stage learning strategy compatible with H-LoRA, with configurations documented in Table 7.
- Training Stability: Training sometimes exhibits loss spikes in medical visual comprehension and generation tasks.
- Training Stability: Larger model parameters and learning rates tend to produce the observed loss spikes, motivating different hyperparameters for HealthGPT-M3 and HealthGPT-L14.
- Dataset Construction: VL-Health construction comprises data collection and data processing, with distribution summarized in Figure 8 and Table 8.
- Dataset Construction: VL-Health includes 765,802 additional VQA training samples alongside LLaVA-558k and PubMedVision-PT alignment data.
- Dataset Format: All samples are converted into a unified instruction-response format for training and evaluation.
- Dataset Format: Each sample specifies visual-feature granularity and the corresponding H-LoRA submodule, while generation responses include multimodal tokens for VQ indices.
B Analysis of Heterogeneous Low-Rank Adaptation
H-LoRA uses task-specific routing and merged expert matrices to preserve heterogeneous knowledge while reducing the computational overhead of multi-expert adaptation. Its overhead remains constant with the number of experts, unlike MoELoRA.
- H-LoRA uses hard routing to allocate task-specific plugins for heterogeneous knowledge learning and representation.The method assigns comprehension- or generation-specific modules based on the task type and hidden state.
- H-LoRA’s algorithm combines task-specific visual features with text features, applies routed merged experts, and adds the result to pretrained weights.Generation selects concrete-grained visual features, while comprehension uses abstract-grained features.
- H-LoRA concatenates LoRA expert matrices, replacing independent expert operations with merged matrix computation.The merged matrices are combined before applying routing weights and the base linear layer.
- MoELoRA introduces O(5k+1) overhead, whereas H-LoRA adds fixed O(6) overhead independent of expert count k.The difference becomes more consequential as the number of experts increases.
C.1 Results: OmniMedVQA Benchmark
On OmniMedVQA, HealthGPT-M3 leads four of seven subtasks, while HealthGPT-L14 achieves the strongest overall performance with an average score of 74.4. The unified model remains relatively weak on this benchmark, but the approach mitigates degradation associated with generation tasks.
- OmniMedVQA evaluates medical visual question answering across diverse modalities and anatomical regions using images collected from multiple medical datasets.
- HealthGPT-M3 outperforms other models in 4 of 7 OmniMedVQA subtasks and exceeds cutting-edge medical and general LVLMs in average score.
- HealthGPT-L14 achieves optimal or near-optimal results across all subtasks, with an average score of 74.4 that significantly surpasses other models.
- The unified model performs relatively weakly on OmniMedVQA, but the approach mitigates performance degradation caused by generation tasks.
C.2 Stability Analysis of Number of Experts
The expert-count analysis shows that H-LoRA scales without the training-time penalty observed for MoELoRA. At eight experts, MoELoRA takes twice as long as LoRA, while H-LoRA adds no training delay and performs better.
- At n = 8, MoELoRA training takes twice as long as LoRA, whereas H-LoRA incurs no additional training delay and performs better.
- At n = 32, MoELoRA is unable to complete training.
- The three-phase learning strategy supports H-LoRA by mixing embedding-layer and output-head training for comprehension and generation tasks during phase two.
C.4 Human Evaluation.
Human evaluation and reconstruction experiments assess HealthGPT’s unified medical capabilities across comprehension and generation. HealthGPT is frequently preferred by clinicians, shows stable reconstruction performance, and produces visually faithful modality transformations and super-resolution results.
- Human evaluation: Five clinicians ranked anonymized responses from six models on 1,000 open-ended questions across VQA-RAD, SLAKE, and PathVQA, with HealthGPT frequently selected as best.
- Reconstruction evaluation: HealthGPT exhibits the most stable performance among evaluated models on four medical image reconstruction tasks.
- Modality transfer: Figure 11 presents CT-to-MRI and MRI-to-CT modality transformations whose generated results are highly close to the ground truth.
- MRI super-resolution: Figure 12 shows MRI super-resolution reconstruction that restores scan-image details and reconstructs essential details accurately.
- Human evaluation: Figure 10 reports the proportion of responses selected as best in human evaluation and identifies the corresponding human evaluation dataset.