PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Medical Large Vision-Language Models with Abnormal-Aware Feedback

Yucheng Zhou, Lingran Song, Jianbing Shen

arXiv:2501.01377v2cs.CLcs.AIcs.CVcs.LG

TL;DR

Medical LVLMs still struggle to localize abnormalities in medical images, despite this capability being important for reliable interpretation. UMed-LVLM addresses the gap with the MAU dataset and two-stage abnormality-aware training, outperforming existing methods across medical-image understanding, abnormality identification, and cross-modal generalization.

  • Problem

    Existing Med-LVLMs have limited abnormality-localization capability, while specialized training data for medical-image abnormalities remain scarce.

  • Method

    UMed-LVLM combines the 5,817-image MAU dataset with Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding to focus diagnosis on abnormal regions.

  • Results

    UMed-LVLM outperforms competing methods on abnormality understanding, medical visual question answering, and MedMNIST evaluation, including unseen ultrasound and dermatoscopy modalities.

  • Takeaways & Limitations

    The findings support improving abnormality detection as a route to stronger medical-image understanding and broader generalization in Med-LVLMs.

  • Takeaways & Limitations

    Limited computational resources prevented evaluation on the largest and most complex open-source LVLMs, restricting validation at larger scale.

Abstract

from arXiv · show

Existing Medical Large Vision-Language Models (Med-LVLMs), encapsulating extensive medical knowledge, demonstrate excellent capabilities in understanding medical images. However, there remain challenges in visual localization in medical images, which is crucial for abnormality detection and interpretation. To address these issues, we propose a novel UMed-LVLM designed to unveil medical abnormalities. Specifically, we collect a Medical Abnormalities Unveiling (MAU) dataset and propose a two-stage training method for UMed-LVLM training. To collect MAU dataset, we propose a prompt method utilizing the GPT-4V to generate diagnoses based on identified abnormal areas in medical images. Moreover, the two-stage training method includes Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding, comprising Relevance Reward, Abnormal Localization Reward and Vision Relevance Reward. Experimental results demonstrate that our UMed-LVLM significantly outperforms existing Med-LVLMs in identifying and understanding medical abnormalities, achieving a 58% improvement over the baseline. In addition, this work shows that enhancing the abnormality detection capabilities of Med-LVLMs significantly improves their understanding of medical images and generalization capability.

1 Introduction

Existing Med-LVLMs understand medical images and queries but remain limited in abnormality localization, especially where detector data are scarce. The paper proposes UMed-LVLM, using the MAU dataset and abnormality-focused training to improve abnormality identification and medical-image understanding.

  • Motivation: Existing Med-LVLMs remain limited in visual localization of abnormalities, while scarce training data—particularly for rare diseases—makes detector-free improvement necessary.The paper motivates enhancing inherent localization capabilities to improve medical-image understanding and response reliability.
  • Dataset: The MAU dataset is created with a GPT-4V prompt method and comprises medical images, queries, diagnoses, and abnormality annotations for training.The dataset contains 5,817 medical images.
  • Proposed approach: UMed-LVLM enhances medical diagnosis by improving visual localization to unveil abnormalities.The model is designed to focus on abnormal areas when generating responses.
  • Results: UMed-LVLM outperforms existing Med-LVLMs in identifying and understanding medical abnormalities on the MAU dataset.The experiments also analyze training and generalization capabilities, supporting the use of medical abnormalities to enhance Med-LVLMs.

2 Related Work

Recent LVLM advances have improved visual comprehension and contextual language understanding. Pre-trained image-text models and GPT-4 have propelled progress in vision-language tasks, including models such as CLIP, BLIP-2, and LLaVA.

  • 2 Related Work: Recent LVLM advancements have significantly improved visual comprehension and contextual language understanding.
  • 2 Related Work: CLIP and BLIP-2 achieved impressive vision-language results by leveraging pre-trained image-text pairs.
  • 2 Related Work: GPT-4 further propelled the field, alongside models such as LLaVA and its improved version.

3 Methodology

UMed-LVLM uses two-stage training that combines Abnormal-Aware Instruction Tuning with Abnormal-Aware Rewarding to improve abnormality understanding and localization. The rewarding stage integrates language relevance, bounding-box localization, and vision relevance signals within reinforcement learning.

  • Two-Stage Training: UMed-LVLM is trained in two stages: Abnormal-Aware Instruction Tuning followed by Abnormal-Aware Rewarding, with continued training on MedVInt.Abnormal-Aware Rewarding comprises LLM relevance, Abnormal Localization, and Vision Relevance Rewarding.
  • Abnormal-Aware Instruction Tuning: Abnormal-Aware Instruction Tuning trains the model on medical images, queries, diagnoses, and abnormal-region descriptions using cross-entropy loss.This improves abnormal-region understanding and diagnosis generation, but does not directly guide attention toward abnormal regions.
  • Abnormal-Aware Rewarding: Abnormal-Aware Rewarding uses reinforcement learning to optimize Med-LVLMs around abnormalities through Relevance, Abnormal Localization, and Vision Relevance Rewards.The relevance framework combines policy, value, and LLM reward models to guide response generation and optimization.
  • Abnormal Localization Reward: Abnormal Localization Reward scores predicted abnormal bounding boxes by their Intersection over Union with ground-truth boxes, directly encouraging precise localization.Higher IoU indicates greater overlap and more accurate abnormal-area localization.
  • Vision Relevance Reward: Vision Relevance Reward aggregates attention between abnormal-category tokens and image patches identified as abnormal regions, reinforcing focus on critical visual information.The reward uses transformer attention scores to quantify associations between textual abnormalities and corresponding image patches.
  • MAU Dataset Construction: The MAU dataset pipeline combines data collection, prompt design, GPT-4V response generation, output reflection, diagnosis construction, and human expert review.These stages are presented as the process for constructing the medical abnormalities unveiling dataset.

4 MAU Dataset

Section 4 introduces the MAU dataset, built from abnormality-annotated medical images and GPT-4V-generated diagnosis annotations. The dataset construction uses a two-stage prompt pipeline and human expert review to improve reliability.

  • Collecting Medical Image Datasets with Abnormal Areas: MAU combines abnormality-annotated medical images with GPT-4V-generated diagnosis annotations for training UMed-LVLM.The source data include five sub-datasets spanning X-rays, CT scans, and gross pathology, with abnormality types and bounding boxes.
  • Medical Abnormal Unveiling Dataset Construction: The GPT-4V prompt method constructs MAU through two stages: diagnosing abnormalities and reflecting on the previous diagnosis.The plug-and-play method integrates with other medical datasets and is disease-agnostic.
  • Review by Human Experts: Human experts reviewed and filtered all generated data, manually correcting the 13 samples found to contain errors.The reviewers were three doctoral students specializing in medicine.
  • Review by Human Experts: Providing GPT-4V with the abnormal-area location improves diagnosis performance over direct diagnosis without additional information.The supplied passage states that direct diagnosis yields limited performance, but does not provide the subsequent quantitative result.

5 Experiments

Experiments evaluate UMed-LVLM across abnormality, medical VQA, classification, and generalization benchmarks, showing strong performance and benefits from abnormal-aware training. Ablations and sensitivity analyses further link diagnostic gains to localization, rewarding components, instruction-tuning scale, and training diversity.

  • Overall Results: UMed-LVLM outperforms other methods on the MAU benchmark and significantly surpasses GPT-4V, while adding ground-truth abnormal-region boxes improves GPT-4V diagnosis.The comparison attributes gains over MedVInt variants to the MAU dataset and Abnormal-Aware Rewarding, underscoring the importance of abnormal-region localization.
  • Overall Results: UMed-LVLM outperforms other methods on MedMNIST Pneumonia, Breast, and Derma, despite continual training without ultrasound and dermatoscopy data.This result demonstrates generalization to modalities excluded from continual training.
  • Ablation Study: Removing ALR, VRR, or AAR progressively decreases diagnosis accuracy, supporting the critical role of abnormal localization, vision relevance, and their dual-reward combination.VRR aligns abnormal identification with attention regions to reduce missed subtle abnormalities, while the w/o AAR variant uses only Abnormal-Aware Instruction Tuning.
  • Impact of Abnormal Localization: More accurate abnormal localization improves diagnostic performance, although gains diminish when IoU exceeds 0.6.Bounding boxes with different IoU scores were injected during inference to assess localization effects.
  • Instruction-Tuning Sensitivity: Instruction-tuning performance rises through epoch 4 to approximately 0.75 and increases substantially as the training-data scale grows from 20% to 100%.The trends indicate benefits from both additional instruction-tuning epochs and larger training sets.
  • Generalization: UMed-LVLM generalizes to untrained categories, excluded datasets, and cross-modality evaluations better than comparison methods, whereas MedVInt scores close to zero on untrained categories.Performance is reduced on categories absent from training, but abnormal-aware learning enables transfer across unseen categories, datasets, and modalities.

6 Conclusion

The study introduces UMed-LVLM to improve medical diagnosis through visual localization of abnormalities. Its specialized dataset and abnormal-aware training improve anomaly detection, interpretation, diagnostic reliability, and interpretability.

  • Conclusion: UMed-LVLM is designed to enhance medical diagnosis by visually localizing abnormalities in medical images.
  • Conclusion: Its specialized process combines the Medical Abnormalities Unveiling dataset with Abnormal-Aware Instruction Tuning and Abnormal-Aware Rewarding.
  • Conclusion: UMed-LVLM surpasses existing Med-LVLMs in accurately detecting and interpreting medical anomalies.The AAR’s reward mechanisms sharpen focus on abnormal areas, improving diagnostic reliability and interpretability.

Limitations … A.2 Medical Large Vision-Language Models

The study is limited by computational resources that prevent validation on the largest open-source LVLMs, while related work reviews advances in general and medical LVLMs, including specialized multimodal systems.

  • Limitations: Limited computational resources prevented applying the methodology to the largest and most complex open-source LVLMs, restricting larger-scale validation.The authors identify computational capacity as the study’s limitation because increasingly large LVLMs require substantial training and deployment power.
  • A.2 Medical Large Vision-Language Models: Medical LVLM applications have increased significantly alongside rapid LVLM development.This establishes the broader motivation for medical-domain systems reviewed in the related work.
  • A.2 Medical Large Vision-Language Models: LLaVA-Med extended LLaVA to medicine with multimodal dialogue, whereas Visual Med-Alpaca combined image captioning, an LLM, and classifier-based model assignment.LLaVA-Med is described as an end-to-end model specifically designed for the medical field.
  • A.2 Medical Large Vision-Language Models: OphGLM combined visual and language capabilities to provide comprehensive ophthalmic assistance.The passage presents OphGLM as an ophthalmology-focused medical LVLM.
  • A.2 Medical Large Vision-Language Models: XrayGPT addressed open-ended chest-radiograph questions by aligning a medical visual encoder with a fine-tuned large language model through a linear transformation.The passage identifies the visual encoder as Wang et al. (2022) and the language model as Chiang et al. (2023).
  • A.2 Medical Large Vision-Language Models: Qilin-Med-VL was introduced as a Chinese large vision-language model in response to the scarcity of non-English language models.The supplied passage truncates before providing further details about the model.

A.3 Reinforcement Learning for Large Language Models

Reinforcement learning (RL) learns through interactions with environments and offers flexible reward functions beyond supervised and unsupervised learning. This flexibility makes RL a useful technique for developing large language models (LLMs), with WebGPT illustrating its application.

  • Foundations: RL is a training paradigm that learns from interactions with environments.This definition is attributed to Sutton and Barto (1998).
  • Foundations: Widely used RL approaches include Q-learning, TRPO, A3C, and PPO.The passage cites Mnih et al. (2015, 2016) and Schulman et al. (2015, 2017) for these methods.
  • RL for LLMs: RL supports generative AI because its reward functions are more flexible than the objectives used in supervised and unsupervised learning.This flexibility facilitates applying RL methods to generative AI models (Cao et al., 2023).
  • RL for LLMs: In LLM development, RL can help unlock substantial potential, as illustrated by WebGPT’s use of RL and rejection sampling to optimize its reward function.The supplied passage identifies WebGPT as a successful RL–LLM application, but its sentence is truncated before stating the optimization aim.

B MAU Dataset

The MAU dataset combines five diverse sub-datasets spanning CT and chest X-ray images, covering multiple abnormalities and pathological categories. It includes large-scale collections for lesions, kidney stones, and thoracic diseases, with dataset sizes ranging from 1,300 to 112,120 images or scans.

  • Dataset composition: MAU integrates five sub-datasets from diverse sources: DeepLesion, KidneyStone, NIH, TBX11K, and KVASIR.The collection spans medical imaging tasks involving lesions, kidney stones, chest diseases, tuberculosis, and endoscopic imagery.
  • Dataset composition: The dataset includes 32,120 axial CT slices from DeepLesion covering eight abnormality types across anatomical regions.Covered regions include the lungs, abdomen, mediastinum, liver, pelvis, soft tissue, kidneys, and bones.
  • Dataset composition: The collection contains 1,300 renal CT scans of kidney stones and 112,120 NIH chest X-rays spanning 14 pathological categories.The KidneyStone scans vary in stone size, shape, and urinary-system location, while NIH includes conditions such as pneumonia, tuberculosis, lung cancer, pulmonary edema, and pulmonary fibrosis.
  • Dataset composition: TBX11K contributes 11,200 chest X-ray images focused on tuberculosis classification and detection.

C Implementation Details · D Large Vision-Language Models · E Hyperparameter Choices for Equ. 11

The experiments initialize UMed-LVLM from MedVInT, specify separate instruction-tuning and AAR optimization settings, compare general-purpose and medical LVLMs, and search hyperparameters for Equ. 11. Supplementary figures show diagnosis comparisons, multi-turn interaction, and MAU examples.

  • C Implementation Details: UMed-LVLM is initialized from MedVInT and trained with Adam using a 1 × 10−5 learning rate, 0.01 weight decay, batch size 128, four epochs, and linear learning-rate decay.Instruction tuning uses the reported optimization configuration.
  • C Implementation Details: AAR uses γ = 0.99, c1 = 0.5, c2 = 0.5, c3 = 0.01, a 1 × 10−6 learning rate, and batch size 16.The passage specifies these AAR coefficients and optimization settings.
  • D Large Vision-Language Models: The evaluation covers general-purpose LVLMs MiniGPT-4 (Zhu et al., 2024a), mPLUG-Owl (Ye et al., 2023), LLaVA (Liu et al., 2023b), and Qwen-VL.These models are evaluated alongside specialized medical LVLMs.
  • D Large Vision-Language Models: The evaluated medical LVLMs include XrayGPT (Thawakar et al., 2023), LLaVA-Med (Li et al., 2023a), Med-Flamingo (Moor et al., 2023), and MedVInt (Zhang et al., 2023a).The study compares these specialized models with general-purpose LVLMs.
  • D Large Vision-Language Models: Figure 5 compares diagnosis results between UMed-LVLM and GPT-4V on randomly sampled samples, while Figure 6 presents a multi-turn conversation from UMed-LVLM.These visuals document diagnostic comparison and conversational behavior without supplying numerical results.
  • D Large Vision-Language Models: The compared LVLMs generally use two-stage training: vision-language feature alignment followed by instruction-dataset fine-tuning.UMed-LVLM extends the MedVInt framework with MAU training using Instruction Tuning and ARRL methods.
  • E Hyperparameter Choices for Equ. 11: Hyperparameters for Equ. 11 were based on PPO1 source code and further tuned through task-specific search, with explored ranges and selected values reported in Table 11.Figure 7 provides examples from the MAU datasets.

F Case Study

Figure 5 case studies show UMed-LVLM localizing a suspicious upper-lobe lung lesion in a chest X-ray, whereas GPT-4V failed to localize the abnormality.

  • F Case Study: In a chest X-ray example, UMed-LVLM localized a suspicious upper-lobe lung lesion despite a moderate IoU score, consistent with pulmonary tuberculosis findings, while GPT-4V failed to localize it.Figure 5 compares randomly sampled examples from UMed-LVLM and GPT-4V.
Loading 2501.01377v2…