PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging

Zhenyang Cai, Junying Chen, Rongsheng Wang, Weihong Wang, Yonglin Deng, Dingjie Song, Yize Chen, Zixu Zhang, Benyou Wang

arXiv:2412.20070v2cs.CVcs.AIcs.CLcs.LG

TL;DR

The factors behind MLLMs’ medical-image generalization remain unclear despite reported benefits of multi-task training. This paper uses compositional generalization and Med-MAT, built from 106 medical datasets, to investigate that gap. It finds that CG supports unseen-image understanding and is a key factor in multi-task generalization, with reported relevance to limited-data and cross-task settings.

  • Problem

    Although multi-task training has been reported to outperform single-task training, the factors underlying MLLM generalization in medical imaging remain insufficiently explored.

    Source: §1 · p. 1

  • Method

    The authors use compositional generalization to study MLLM generalization with Med-MAT, constructed from 106 medical datasets grouped by image elements.

    Source: §1 · pp. 1–2

  • Results

    MLLMs used CG to understand unseen medical images, and CG was identified as a key factor in multi-task generalization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report that CG supports limited-data medical scenarios and applies across classification and detection tasks.

    Source: §8 · p. 8

    The study uses data from medical challenges and open-source datasets, and further experiments are needed to mitigate potential risks in real-world deployment; CG is also only one aspect of MLLM generalization.

    Source: Potential Risks · p. 9, Limitations · p. 9

Abstract

from arXiv · show

Medical imaging provides essential visual insights for diagnosis, and multimodal large language models (MLLMs) are increasingly utilized for its analysis due to their strong generalization capabilities; however, the underlying factors driving this generalization remain unclear. Current research suggests that multi-task training outperforms single-task as different tasks can benefit each other, but they often overlook the internal relationships within these tasks. To analyze this phenomenon, we attempted to employ compositional generalization (CG), which refers to the models' ability to understand novel combinations by recombining learned elements, as a guiding framework. Since medical images can be precisely defined by Modality, Anatomical area, and Task, naturally providing an environment for exploring CG, we assembled 106 medical datasets to create Med-MAT for comprehensive experiments. The experiments confirmed that MLLMs can use CG to understand unseen medical images and identified CG as one of the main drivers of the generalization observed in multi-task training. Additionally, further studies demonstrated that CG effectively supports datasets with limited data and confirmed that MLLMs can achieve CG across classification and detection tasks, underscoring its broader generalization potential. Med-MAT is available at https://github.com/FreedomIntelligence/Med-MAT.

1 Introduction

The paper examines what drives MLLMs’ generalization in medical imaging by testing compositional generalization (CG), the recombination of learned elements to understand unseen combinations. It introduces Med-MAT and reports that CG supports unseen-image understanding and is a key factor in generalization during multi-task training.

1 Introduction

  • The authors investigate why multi-task training generalizes, since its underlying factors remain insufficiently explored despite reported benefits over single-task training.Medical images provide visual insights into body structures for diagnosis.Source: §1 · p. 1
  • CG describes understanding unseen combinations by learning fundamental elements and recombining them in novel ways.The paper uses this framework to investigate generalization among MLLMs analyzing medical images.Source: §1 · p. 1
  • Med-MAT was built by collecting 106 medical datasets and grouping data according to three image elements, yielding 53 subsets.The subsets cover 11 modalities, 14 anatomical regions, and 13 medical tasks.Source: §1 · pp. 1–2
  • Experiments found that MLLMs can use CG to understand unseen medical images, with consistent findings across different MLLMs.The authors describe CG as one of the main forms of generalization for medical MLLMs.Source: §1 · p. 2
  • Disrupting CG reduced generalization, identifying it as a key factor in MLLM generalization observed during multi-task training.The paper also reports CG benefits for limited-data medical scenarios and spatial awareness across classification and detection tasks.Source: §1 · p. 2

2 A Pilot Study on Generalization

This pilot study introduces Med-MAT as a platform for exploring MLLM generalization and asks what drives the gains observed in multi-task training. Its dataset preparation standardizes image-label data for VQA experiments, while initial comparisons report multi-task benefits on some in-distribution tasks and on out-of-distribution prediction.

2.1 Data Collection (Med-MAT)

  • Med-MAT combines 106 public-challenge or annotated medical image-label datasets, grouping data with identical MAT-triplets into subsets and manually clustering synonymous labels.The described coverage includes 11 medical modalities and 13 medical tasks.Source: §2.1 · p. 2
  • Training sets follow original splits or a 9:1 ratio, are capped at 3,000 samples with label balance maintained where possible, and undersized subsets are treated as OOD.Test sets balance samples per label to support reliable accuracy measurement.Source: §2.1 · p. 2
  • Image-label pairs are converted to VQA samples with six subset-specific instructions and single-choice questions of up to four options.Distractors are drawn from labels within the subset; ImageWikiQA is included in training to mitigate evaluation bias from differing option counts.Source: §2.1 · p. 3

2.2 Observation

  • Multi-task training outperformed single-task training on specific tasks and improved out-of-distribution prediction, motivating investigation of what drives these gains.The comparison evaluated models on all datasets after training on all in-distribution datasets or on individual in-distribution datasets.Source: §2.2 · p. 3, Table 1 · p. 3, Table 2 · p. 3
  • The pilot study asks what drives the generalization observed in MLLMs during multi-task training and frames its investigation through CG.The preceding comparison of multi-task and single-task training motivates this question.Source: §2.2 · p. 3

3 Proof of Concept on CG

The section tests whether MLLMs can recombine learned medical-image elements to understand unseen images, and whether this generalizes across model backbones. Its experiments examine two-element combinations, related-data controls, and other backbones.

3.1 Experiment Setup

  • The setup varies two MAT-Triplet elements while holding the third constant, then evaluates trained models on Target data against untrained and unrelated-data controls.Baseline is untrained; Baseline+ uses an equal number of randomly sampled Unrelated examples.Source: §3.1 · pp. 3–4

3.2 Results

  • Almost all tested two-element CG combinations generalize to downstream classification tasks, with similar strength reported for three-element combinations.The study also reports this pattern across all Direction Types.Source: §3.2 · p. 4
  • Related-data training supports unseen-target generalization, while models trained on unrelated data perform near randomly on the test set.The authors interpret this contrast as evidence that training data sharing at least partial MAT elements can support learning and generalization to new target tasks.Source: §3.2 · p. 5

3.3 Extending CG to other Backbones

  • Qwen2-VL and Llama3.2-Vision also show generalization across most selected tasks, despite differences in pretraining data and vision encoders.The selected backbones were used to test whether CG extended beyond LLaVA.Source: §3.3 · p. 5, Table 4 · p. 5

4 Scaling Combination in CG

The scaling experiments compare related and unrelated training combinations and test the effect of disrupting CG while holding data volume constant. Related combinations approach the performance of training on all data, whereas unrelated data alone remains close to baseline.

4.1 Experiment Setup

  • The scaling study asks whether more unrelated combinations help in a multi-task setting and whether disrupting CG changes multi-task generalization.These are framed as the study’s two sub-questions.Source: §4.1 · p. 5
  • The experiment uses Subsets 03 and 28 as Targets for balanced Related-to-Unrelated ratios, and caps each training condition at 15,000 samples.It compares All Data, All Related, All Unrelated, and Related variants that omit one shared MAT element.Source: §4.1 · pp. 5–6

4.2 Analysis of Scaling Experiment

  • All Related performs comparably to All Data, while All Unrelated remains close to Baseline even as unrelated data volume and task diversity increase.Figure 4 reports accuracy on the Target dataset, which is excluded from training.Source: Figure 4 · p. 6, §4.2 · p. 6
  • Removing datasets that share the Target’s modality, area, or task produces significant accuracy drops versus All Related despite equal training volume.The authors interpret the drops as reduced generalization when CG combinations are disrupted.Source: §4.2 · p. 6

5 Potential Applications of CG

The applications experiments examine generalization to targets without training data and training efficiency when target data are limited. Related combinations outperform unrelated combinations in the first setting and help models reach peak performance sooner in the second.

5.1 Generalization without Target Data

  • The no-Target-data experiments progressively increase composition-dataset size to assess transfer to selected Target datasets.Combinations were selected from cases where trained results exceeded both Baseline and Baseline+ by at least 10.Source: §5.1 · p. 6
  • Related combinations outperform Unrelated combinations across tasks without Target data, and their advantage continues to improve as composition data increases.Figure 5 presents accuracy curves as composition-dataset size increases without Target data in training.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2 Generalization with Limited Target Data

  • The limited-data study targets combinations with weak prior generalization and adds 2,000 Target examples to assess training efficiency.The section motivates this setting with rare conditions and tasks lacking dedicated training data.Source: §5.1 · p. 6, §5.2 · p. 7
  • With limited Target data, adding Related combinations helps the model reach peak performance sooner, even when CG brings no significant direct generalization gain.Figure 6 shows results as Target-data volume increases.Source: §5.2 · p. 7, Figure 6 · p. 7

6 CG across Detection and Classification

The section tests whether CG transfers between classification and detection tasks. It reports benefits in both directions, using detection data to support classification and classification data to support detection.

6 CG across Detection and Classification

  • The experiments ask whether classification data can support detection and detection data can support classification through CG.The motivation distinguishes classification’s visual knowledge from detection’s spatial information.Source: §6 · p. 7

6.1 Experiment Settings

  • Each training combination pairs one detection dataset with one classification dataset to test transfer between the two vision tasks.The study uses Next-Chat and MiniGPT-v2 as detection baselines and converts segmentation masks to bounding boxes when needed.Source: §6.1 · p. 7

6.2 Benefits for Classification (Q1)

  • All tested CG combinations let the model use detection data to generalize to Target classification data.Figure 7 reports classification accuracy for untrained and CG-trained models.Source: §6.2 · p. 8, Figure 7 · p. 8

6.3 Benefits for Detection (Q2)

  • All selected CG combinations help MLLMs achieve better detection performance when a fixed amount of Target data is included.Table 5 reports cIoU gains for Next-Chat and MiniGPT-v2; C and D denote classification and detection tasks.Source: §6.3 · p. 8, Table 5 · p. 8

7 Related Work

Prior work adapts multimodal models to medical tasks and studies generalization through data manipulation and representation learning. Detection-focused MLLM research explores several ways to encode regions and positions.

Medical MLLMs

  • Medical MLLMs have been adapted to specialized tasks through medical-data fine-tuning and instruction tuning.Examples include Med-Flamingo, Med-PaLM, LLaVA-Med, Med-Gemini, and HuatuoGPT-Vision.Source: §7 · p. 8

Generalization on Medical Imaging

  • Medical-imaging generalization research has used data augmentation and representation learning, while some studies examine mutual benefits between classification and segmentation.The cited examples include Li et al. (2022), Zhang et al. (2022), Le-Khac et al. (2020), and Ren et al. (2024).Source: §7 · p. 8

Detection with MLLMs

  • Detection methods for MLLMs encode regions as features, bounding boxes as text tokens, or position information through image-token hidden states.Some approaches also use unique task identifiers to improve learning efficiency.Source: §7 · p. 8

8 Conclusion

The paper uses Med-MAT to investigate compositional generalization in medical MLLMs, finding that it supports generalization to unseen data and limited-data conditions across multiple tasks.

Conclusion

  • Med-MAT provides a research platform for experiments on MLLM generalization in medical imaging.The experiments confirmed compositional generalization and identified it as a key factor in generalization observed in multitask learning.Source: §8 · p. 8
  • Compositional generalization helps MLLMs handle limited-data conditions, supporting its relevance to low-data medical tasks.Source: §8 · p. 8
  • MLLMs can apply compositional generalization across detection and classification tasks, indicating broader generalization potential.Source: §8 · p. 8

Limitations

Compositional generalization contributes to MLLM generalization in medical imaging, but does not account for it entirely.

Limitations

  • Compositional generalization supports MLLM generalization to unseen medical images and data-efficient training.Source: Limitations · p. 9
  • Disrupting compositional generalization reduces generalization but leaves some effectiveness intact.Source: Limitations · p. 9
  • The remaining effectiveness indicates that compositional generalization is only one aspect of MLLM generalization in medical imaging.Source: Limitations · p. 9

Potential Risks

The paper identifies further work needed before applying its findings in real-world medical settings. Additional experiments examine compositional generalization in segmentation and with more flexible medical elements.

Potential Risks

  • Further experiments are needed to mitigate potential risks before applying this work in real-world medical settings.The research uses data from medical challenges and open-source datasets.Source: Potential Risks · p. 9

A.1 Benefits for Segmentation

  • MLLMs still leverage compositional generalization across segmentation and classification tasks.The reported segmentation results are in Table 6, where cIoU gain represents the gain brought by compositional generalization.Source: §A.1 · p. 15, Table 6 · p. 15

A.2 More Complex Medical Elements

  • The paper explores flexible medical elements beyond predefined MAT-Triplet categories, including population groups and finer disease categories.The population-group experiment uses child and adult X-ray images; the finer-disease setup distinguishes COVID from common pneumonia.Source: §A.2 · p. 15, Table 7 · p. 15, Table 8 · p. 15

A.3 Statistical Tests of the Generalization Results

Supplementary experiments report stable CG results across runs, backbones, and medical-expert models, while probing data composition, task scope, dataset construction, and difficult cases.

A.3 Statistical Tests of the Generalization Results

  • Across three runs, the CG results showed low variance and continued to support the original conclusions.The experiments used LLaVA and the same data combinations as Section 3.3; Table 9 reports mean accuracy and standard deviation.Source: §A.3 · p. 15, Table 9 · p. 16

A.4 CG with All MAT-Triplet Elements from Different Sources

  • Most combinations using three distinct dataset sources generalized to targets, though subtle visual similarities caused leakage for two new attributes.CG combinations still improved downstream tasks for the new attributes despite that leakage.Source: §A.4 · pp. 15–16, Table 10 · p. 16

A.5 Details of Section 3.3: Exploring CG on different MLLM Backbones

  • To check whether results depended on model choice, researchers tested several other models and observed similar results on some subsets of Med-MAT.Source: §A.5 · p. 16

A.6 Details of Section 6: Exploring CG across Detection and Classification

  • Experiments tested CG across detection and classification using Next-Chat and MiniGPT-v2, which represent bounding boxes differently.The authors describe this validation with distinct box-modeling methods as evidence of CG’s broad applicability.Source: §A.6 · p. 16

A.7 CG with Medical Multimodal LLM

  • HuatuoGPT-Vision, a medical-expert MLLM, also leveraged CG to improve performance on novel tasks.The experiment reused the data combinations from Section 3.3.Source: §A.7 · p. 18, Table 15 · p. 18

B.1 Details of MAT-Triplet

  • Med-MAT defines samples by MAT-Triplet—Medical Modality, Anatomical Area, and Medical Task—and integrates datasets sharing identical triplets into subsets; it encompasses 11 modalities, 14 anatomical areas, and 13 distinct tasks.Because COVID Diagnosis and Cancer Diagnosis have limited options, the paper groups them under the broader category State.Source: §B.1 · p. 18

B.2 QA construction method

  • The dataset converts samples into multiple-choice QA using six prompts per subset and up to four options, with 3,000 training samples per subset and balanced test sets.QA datasets are released according to access licenses, with restricted datasets available through application links.Source: §B.2 · p. 19, §B.4 · p. 19, §B.3 · p. 19

C.1 Bad case analysis

  • Weak cases involved fine-grained targets and mismatched grading criteria; few-shot CG combinations helped downstream performance, while target-data augmentation accelerated adaptation.The authors also note that tasks such as bone-age assessment and cancer staging require relevant knowledge and defined grading standards.Source: §C.1 · p. 19, §C.2 · p. 19

Related papers

Loading 2412.20070v2…