PaperlayerView exact revision on arXiv

Source-linked AI summary

Boosting Medical Visual Understanding From Multi-Granular Language Learning

Zihan Li, Yiqing Wang, Sina Farsiu, Paul Kinahan

arXiv:2511.15943v2cs.CV

TL;DR

Medical images often carry multiple labels across annotation granularities, a setting not addressed by CLIP’s single-label, single-granularity alignment. MGLL combines multi-label supervision and multi-granular text with alignment losses, and the paper reports consistent state-of-the-art results across downstream datasets.

  • Problem

    CLIP’s single-label, single-granularity alignment does not match medical images that can have multiple labels across annotation levels.

    Source: §1 · p. 1

  • Method

    MGLL combines multi-granular text with soft-label and point-wise supervision and smooth KL divergence for multi-label and cross-granularity alignment.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    MGLL consistently outperforms state-of-the-art methods across multiple downstream datasets, particularly where multi-label understanding spans granularities.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The evaluations support MGLL’s use for downstream medical visual understanding involving multiple labels and granularities.

    Source: §5 · p. 10

Abstract

from arXiv · show

Recent advances in image-text pretraining have significantly enhanced visual understanding by aligning visual and textual representations. Contrastive Language-Image Pretraining (CLIP) has played a pivotal role in multimodal learning. However, its focus on single-label, single-granularity alignment limits its effectiveness in complex domains such as medical imaging, where images often correspond to multiple high-level labels (e.g., disease categories) across different annotation granularities (e.g., diagnostic description, clinical explanation). To address this, we propose Multi-Granular Language Learning (MGLL), a contrastive learning framework designed to improve both multi-label and cross-granularity alignment. MGLL leverages structured multi-label supervision, integrates textual descriptions across granularities, and introduces soft-label supervision with point-wise constraints to enhance alignment. MGLL employs smooth Kullback-Leibler (KL) divergence to ensure cross-granularity consistency while maintaining computational efficiency as a plug-and-play module for vision-language models. Pretrained on our constructed large-scale multi-granular datasets and evaluated across multiple datasets, MGLL outperforms other state-of-the-art methods in downstream tasks. The code is available at https://github.com/HUANGLIZI/MGLL.

1 INTRODUCTION

Medical images can carry multiple disease labels at different annotation granularity, challenging CLIP’s single-label image–text matching [Radford et al. (2021)]. MGLL combines multi-granular text with multi-label and cross-granularity alignment objectives.

1 INTRODUCTION

  • Medical images can represent multiple diseases and finer-grained diagnoses, while annotation costs and privacy concerns contribute to data scarcity.The introduction also describes their information as spanning diagnoses, anatomical structures, lesions, and textures.Source: §1 · p. 1
  • MGLL uses multi-granular descriptions, soft-label supervision, point-wise constraints, and smooth KL divergence for joint multi-label and cross-granularity alignment.The framework is described as requiring no granularity-sensitive encoders, with no additional computational cost, and as plug-and-play for vision foundation or large vision-language models.Source: §1 · p. 2
  • The paper constructs retinal and X-ray multi-granular datasets and reports consistent state-of-the-art results across more than ten downstream datasets.The stated contributions also emphasize generalization ability.Source: §1 · p. 2

2 RELATED WORK

Related work covers image–text contrastive pretraining, multi-label learning, and multi-granularity learning. The paper positions MGLL as a flexible approach to learning multi-label and cross-granularity visual–language representations.

2.1 IMAGE-TEXT CONTRASTIVE LEARNING

  • Medical contrastive methods address issues such as false negatives, but medical frameworks still face data scarcity and complex semantics.Related approaches also include local-to-global pairwise learning, higher-order multimodal relations, and extended-text embeddings.Source: §2.1 · pp. 2–3

2.2 MULTI-LABEL LEARNING

  • Multi-label methods include joint image-label embeddings and hierarchy-preserving losses, while fixed label spaces constrain semantic flexibility.The cited examples include Wang et al. (2016) and Zhang et al. (2022).Source: §2.2 · p. 3

2.3 MULTI-GRANULARITY LEARNING

  • Multi-granularity approaches use varied alignment and fusion strategies, but fixed pipelines can limit heterogeneous annotations; MGLL targets flexible learning across datasets.The passage describes MGLL as not requiring specific annotation formats or model architectures and as robust to mixed granularity and noised annotations.Source: §2.3 · p. 3

3 MULTI-GRANULAR LANGUAGE LEARNING

MGLL combines multi-granular image–text representations with objectives for multi-label and cross-granularity alignment. The paper describes its losses, analyzes their roles, and constructs fundus and X-ray datasets with multi-level text.

3.1 OVERVIEW

  • MGLL encodes images and multi-granularity text, then uses multi-label contrastive learning and smooth KL divergence to align representations across granularities.Vision Transformer and BERT are given as default encoder choices; the framework is described as capturing richer image–text correlations than CLIP without additional parameters.Source: §3.1 · pp. 3–4

3.2 THE MGLL OBJECTIVES

  • Soft CLIP loss supports alignment with multiple labels, point-wise binary cross-entropy supervises individual image–text pairs, and smooth KL aligns granularity distributions toward their mean.The point-wise loss improves fine-grained multi-label alignment and visual-feature discriminability; the KL objective encourages consistency across granularity levels.Source: §3.2 · pp. 4–5

3.3.1 EMPIRICAL ANALYSIS

  • MGLL’s separate granularity spaces and smooth KL consistency objective address limitations of CLIP’s single-label, shared-space alignment.The empirical analysis also describes soft CLIP and point-wise losses as aligning images with multiple correlated text labels on a shared manifold.Source: §3.3.1 · p. 5

3.3.2 THEORETICAL ANALYSIS

  • The theoretical analysis characterizes soft CLIP as aligning images with multiple text features, point-wise loss as strengthening valid matches and suppressing irrelevant ones, and smooth KL as enforcing shared distributions.The text explains that minimizing the KL objective yields equal predicted distributions across granularities.Source: §3.3.2 · pp. 5–6

3.4.1 MGLL-FUNDUS DATASET

  • MGLL-Fundus contains 246,389 image–text pairs from 49 public datasets, covering more than 50 disease categories with disease-category and clinical-explanation text.The dataset description says the hierarchy includes normal/abnormal classification, disease categorization, and detailed clinical descriptions.Source: §3.4.1 · p. 6

3.4.2 MGLL-XRAY DATASET

  • MGLL-Xray comprises 190,882 MIDRC X-ray images with text fields for modality, study description, and series description.The descriptions distinguish modality, exam-level study descriptions, and specific imaging sequences such as PA View or Lateral View.Source: §3.4.2 · p. 6

4 EXPERIMENTS

The experiments evaluate MGLL across retinal and X-ray datasets, multimodal language models, and ablations of its objectives and design choices. The reported results favor MGLL across these evaluations, with performance also depending on encoder, image resolution, and text quality.

4.1 SETUP

  • The evaluation spans eleven downstream datasets, using AUC, mAP, and ACC under linear-probe and full fine-tuning settings.The default encoders are ViT-L/14 and BiomedicalBERT, and the experiments follow official dataset splits with test sets excluded from pretraining.Source: §4.1 · pp. 6–7

4.2.1 EVALUATION ON RETINAL FUNDUS DATASETS

  • MGLL consistently improves results across nine fundus datasets, with particularly strong gains in linear probing and disease-specific localization in CAMs.CLIP instead assigns nearly uniform attention, while MGLL highlights hard exudates for chorioretinitis and the retinal pigment epithelium for age-related macular degeneration.Source: Figure 3 · p. 7, §4.2.1 · p. 7

4.2.2 EVALUATION ON X-RAY DATASETS

  • 2.23% and 3.81% AUC gains over second-best methods on MIDRC-XR and MIDRC-XR-Portable, respectively, widen under full fine-tuning.On ChestX-ray14, linear-probe MGLL reaches 82.94% AUC, 90.41% accuracy, and 28.53% mAP, exceeding CARZero by 5.62%, 6.47%, and 1.65%, respectively.Source: §4.2.2 · pp. 7–8, Table 1 · p. 8

4.3 PERFORMANCE WITH MGLL IN MLLMS

  • On 2,233 clinical cases across ten ophthalmological conditions, MGLL replacement improved average accuracy across tested MLLMs by 4.6% (InternVL) to 34.1% (LLaVA-Med), with all models fine-tuned on target data.LLaVA accuracy increased from 72.73% to 79.98%.Source: §4.3 · p. 8

4.4.1 ABLATION STUDY ON MGLL OBJECTIVES

  • Adding point-wise and soft CLIP losses improves RFMiD results over CLIP, their combination improves further, and soft-KL yields the best performance.The ablation attributes complementary effects to the point-wise and soft CLIP losses, with soft-KL refining consistency across objectives.Source: Table 3 · p. 9, §4.4.1 · p. 9

4.4.3 SELECTION OF IMAGE ENCODER AND TEXT ENCODER

  • On RFMiD, ViT-L/14 achieves the best results across metrics; ViT generally outperforms CNNs, while the larger ViT-H/14 performs slightly worse.BERT is the best tested text encoder across metrics, whereas CLIPtext is the weakest alternative but still improves on baseline CLIP.Source: Table 5 · p. 9, Table 6 · p. 9, §4.4.3 · p. 10

4.4.2 ABLATION STUDY ON GRANULARITY COUNT

  • Three granularity levels—modality, study description, and series description—outperform one or two levels across metrics, including linear-probe gains over CLIP of 12.43% AUC, 10.84% ACC, and 8.31% mAP.MGLL3 also exceeds MGLL1 by 3.32% AUC, 2.09% ACC, and 2.30% mAP in linear probing.Source: Table 4 · p. 9, §4.4.2 · p. 9

4.4.4 ABLATION STUDY ON IMAGE QUALITY AND TEXT QUALITY

  • On MIDRC-XR-Portable, performance rises with image resolution, while standard descriptions outperform degraded text; missing information is less harmful than errors.MGLL remains better than baseline CLIP at 64×64 resolution and under both text-degradation conditions.Source: Table 7 · p. 10, §4.4.4 · p. 10

5 CONCLUSION

MGLL combines multi-granular text with soft-label and point-wise supervision to improve multi-label and cross-granularity alignment. The authors report consistent downstream gains, while emphasizing that clinical deployment requires further validation and human oversight.

5 CONCLUSION

  • MGLL combines multi-granular text, soft-label supervision, point-wise constraints, and smooth KL divergence to improve alignment and cross-granularity consistency.Evaluations across multiple downstream datasets report consistent gains over state-of-the-art methods, particularly where multi-label understanding across granularities is needed.Source: §5 · p. 10
  • The authors state that MGLL is not ready for direct clinical use without further validation and regulatory approval, and should support rather than replace clinical judgment.They call for rigorous clinical validation and human oversight in real-world applications.Source: §5 · p. 11
  • The work provides source code, training configurations, hyperparameters, mathematical derivations, and public-dataset preprocessing details to support reproducibility.Experiments use identical settings, including for baseline models pretrained on the custom multi-granularity datasets.Source: §5 · p. 11

A DETAILED THEORETICAL ANALYSIS OF MGLL

The theoretical analysis characterizes MGLL’s multi-label alignment as a weighted combination of associated text features, while its point-wise and Smooth KL losses respectively refine pair-level matching and cross-granularity consistency. The section also describes the multi-granular fundus, X-ray, and MIMIC-CXR data used for pretraining and evaluation.

A.1 SOFT CLIP LOSS

  • Soft CLIP Loss weights an image’s multiple associated text features so its optimal representation is their weighted center.The weights are assigned to text labels associated with the image.Source: §A.1 · p. 19

(c) Rewriting in Normalized Form

A.2 POINT-WISE LOSS

  • Point-wise binary cross-entropy supervises each image–text pair, pulling valid matches closer and pushing non-matches apart.Its gradient is described as increasing similarity for valid pairs and decreasing it for invalid pairs.Source: §A.2 · pp. 20–21, (a) Loss for a Single Pair · p. 22

A.3 SMOOTH KL DIVERGENCE LOSS

  • Smooth KL divergence encourages predictions from different granularities to agree by moving each distribution toward their mean.The loss is zero if and only if all predicted distributions are equal.Source: §A.3 · p. 22, (b) Conditions for Zero Loss · p. 23

B.1 PRETRAIN DATASETS

  • MGLL-Fundus contains 246,389 image–text pairs from 49 public datasets and covers more than 50 disease categories.Its text spans disease-category and clinical-explanation granularities.Source: §B.1 · p. 24
  • MGLL-Xray comprises 190,882 images with text at modality, study-description, and series-description granularities.The X-ray images were converted from DICOM to PNG while preserving essential metadata.Source: §B.1 · p. 24
  • MIMIC-CXR experiments use structured disease labels and radiology reports as coarse- and fine-grained text, evaluating on ChestX-ray14.The described MIMIC-CXR collection has 377,110 images associated with 227,835 imaging studies.Source: §B.1 · pp. 24–25

B.2 DOWNSTREAM DATASETS

The evaluation spans radiographic and retinal datasets, alongside a multiple-choice ophthalmic benchmark. AUC is the primary metric, with mAP and accuracy also reported and category-wise average accuracy used for the imbalanced RFMiD dataset.

B.2 DOWNSTREAM DATASETS

  • The downstream evaluation uses multiple datasets, including ChestX-ray14’s 112,120 frontal-view chest radiographs annotated with 14 labels and FIVES’s 800 retinal images across four diagnostic classes, including diabetic retinopathy and glaucoma.Source: §B.2 · pp. 26, 28
  • The multiple-choice ophthalmic benchmark contains 2,233 images across 25 diagnostic categories and uses four-option forced-choice classification.Each image is paired with the correct diagnosis and three randomly sampled disease-name alternatives.Source: §B.2 · p. 28, §C.1 · p. 28

C.1 EVALUATION METRICS

  • AUC is the primary evaluation metric, with mAP and accuracy also reported; RFMiD uses category-wise average accuracy because it is imbalanced.The passage describes mAP as useful for long-tailed label distributions.Source: §C.1 · p. 28

C.2 IMPLEMENTATION DETAILS

Implementation uses ViT-L/14 and BiomedicalBERT encoders, with 224 × 224 images and AdamW optimization.

C.2 IMPLEMENTATION DETAILS

  • Experiments use ViT-L/14 and BiomedicalBERT encoders, 224 × 224 inputs, and AdamW with a 1e-4 learning rate.The setup specifies batch size 32, feature dimension 768, temperature 0.07, and an NVIDIA RTX A6000 GPU.Source: §C.2 · p. 28

D.1 DETAILED RESULTS ON RETINAL FUNDUS DATASETS

MGLL is reported to outperform existing approaches across nine retinal datasets in linear probing and full fine-tuning, with strong results also reported for zero-shot classification and region segmentation. The supplied comparisons include representative retinal values and outcomes across medical and natural-image settings.

D.1 DETAILED RESULTS ON RETINAL FUNDUS DATASETS

  • 90.02% AUC on ADAM and 92.42% AUC on REFUGE exceed the cited second-best methods in linear probing; MGLL also leads across nine retinal datasets in both evaluation settings.The reported comparisons are 90.02% versus UniMed-CLIP’s 79.33% on ADAM and 92.42% versus RET-CLIP’s 84.59% on REFUGE.Source: §D.1 · p. 29, Table 19 · p. 30, Table 21 · p. 31

D.2 ZERO-SHOT COMPARISONS ACROSS MEDICAL AND NATURAL DOMAINS

  • 39.0% accuracy on COVIDx and 52.7% on OrganAMNIST are reported for MGLL, which also achieves the best stated CC3M result at 23.5% accuracy.The cited comparison reports 34.3% for CheXAgent and 35.5% for MedVersa on COVIDx, and 47.9% for FG-CLIP on OrganAMNIST.Source: §D.2 · p. 32, Table 24 · p. 32

D.3 PERFORMANCE EVALUATION ON REGION SEGMENTATION

  • MGLL achieves the best performance among the compared methods on the COVID-Xray region-segmentation evaluation.The table caption identifies the comparison as COVID-19 lesion segmentation on the COVID-Xray dataset.Source: §D.3 · p. 32, Table 25 · p. 32

D.4 MORE DETAILED RESULTS OF MGLL IN MLLMS

MGLL improves diagnostic accuracy across evaluated multimodal large language models, including difficult retinal conditions, and produces more focused lesion localization than CLIP on retinal datasets.

D.4 MORE DETAILED RESULTS OF MGLL IN MLLMS

  • The multiple-choice evaluation covers 25 ophthalmological conditions, extending beyond the ten primary conditions in earlier experiments.The models receive standardized diagnostic queries.Source: §D.4 · p. 33
  • 21.76% overall improvement (49.17% to 70.94%) for Med-Flamingo; MGLL improves diagnostic accuracy across all seven evaluated architectures.LLaVA-Med improves by 20.78% (56.47% to 77.25%).Source: §D.4 · p. 33
  • MGLL integration corrects baseline misses in representative cases of choroidal neovascularization and chorioretinitis across several multimodal models.For chorioretinitis, six models diagnose the condition correctly after integration, while none did so at baseline.Source: Figure 5 · p. 36, §D.4 · p. 37

D.5 MORE CAMS ON RETINAL FUNDUS DATASETS

  • Across retinal datasets, MGLL produces more precise lesion localization than CLIP under both linear probing and full fine-tuning.The reported activation maps highlight pathological features in diabetic retinopathy, macular edema, and glaucoma rather than diffuse or nonspecific regions.Source: §D.5 · p. 37

D.6 ABLATION STUDY ON WEIGHT FACTORS OF LOSS

Ablations identify effective settings for the composite loss and temperature coefficient, while tests report robustness to missing labels and mixed-granularity annotations.

D.6 ABLATION STUDY ON WEIGHT FACTORS OF LOSS

  • 79.62% AUC, 92.84% ACC, and 34.08% mAP in linear probing; 92.83% AUC, 95.48% ACC, and 64.99% mAP in full fine-tuning are achieved with the optimal RFMiD weights.The selected weights are α1 = 0.5, α2 = 1.0, and α3 = 1.0.Source: §D.6 · p. 37, Table 35 · p. 37

D.7 ABLATION STUDIES ON THE TEMPERATURE COEFFICIENT (τ)

  • τ = 0.07 gives the best reported results, with performance first improving and then dropping as τ increases.The authors attribute the trade-off to sharper similarity distributions and training instability at smaller τ, versus weaker alignment at larger τ.Source: §D.7 · p. 38, Table 36 · p. 38

D.8 PERFORMANCE ON DATASETS WITH ARTIFICIALLY INTRODUCED NOISE

  • With 30% of granularity labels missing, MGLL achieves 79.61% AUC in linear probing and 96.74% in full fine-tuning, above CLIP trained with complete labels.The corresponding CLIP AUCs are 71.43% and 91.83%.Source: §D.8 · p. 38, Table 37 · p. 38

D.9 PERFORMANCE ON DATASETS WITH MIXING GRANULARITY LEVELS

  • MGLL achieves comparable performance on two subsets with different granularity levels and on their combination, supporting robustness to heterogeneous annotation structures.The pretraining dataset was split into equal-sized subsets: Set A had two granularity levels, while Set B had one.Source: §D.9 · p. 38, Table 38 · p. 38

E DISCUSSION AND FUTURE WORK

The discussion links MGLL’s performance gains to multi-granular descriptions and input-data quality. It identifies broader inputs, domain generalization, and non-medical applications as future directions.

E DISCUSSION AND FUTURE WORK

  • Reported performance gains scale with the number of granularity levels and the quality of input data.The discussion attributes MGLL’s performance to complementary information in multi-granular textual descriptions.Source: E DISCUSSION AND FUTURE WORK · p. 39
  • Future work could improve generality by incorporating patient metadata or temporal information, adapting to unseen medical conditions or modalities, and integrating MGLL with large language models.Source: E DISCUSSION AND FUTURE WORK · p. 39
  • Satellite imagery and scientific visualization are proposed as non-medical domains for further validation of MGLL’s broader applicability.Source: E DISCUSSION AND FUTURE WORK · p. 39

Related papers

Loading 2511.15943v2…