PaperlayerView exact revision on arXiv

Source-linked AI summary

M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models

Fan Bai, Yuxin Du, Tiejun Huang, Max Q. -H. Meng, Bo Zhao

arXiv:2404.00578v1cs.CV

TL;DR

Medical MLLM research has focused mainly on 2D images, leaving 3D analysis comparatively under-explored despite its richer spatial information. The paper introduces a dataset, model, and benchmark for diverse 3D medical tasks, and reports robust performance across them, including gains over PMC-CLIP in retrieval.

  • Problem

    Previous medical MLLM research has primarily focused on 2D images, leaving 3D medical image analysis under-explored despite its richer spatial information.

  • Method

    The paper builds M3D-Data, proposes the 3D MLLM M3D-LaMed, and introduces M3D-Bench for automatic evaluation across eight tasks.

  • Results

    The method is reported as robust across 3D medical image-analysis tasks, outperforming existing solutions, including PMC-CLIP by 54% at IR R@10 and 17.95% at IR R@1.

  • Takeaways & Limitations

    M3D-LaMed provides a generalist foundation spanning retrieval, report generation, VQA, positioning, and segmentation for 3D medical scenarios.

Abstract

from arXiv · show

Medical image analysis is essential to clinical diagnosis and treatment, which is increasingly supported by multi-modal large language models (MLLMs). However, previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. This paper aims to advance 3D medical image analysis with MLLMs. To this end, we present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs specifically tailored for various 3D medical tasks, such as image-text retrieval, report generation, visual question answering, positioning, and segmentation. Additionally, we propose M3D-LaMed, a versatile multi-modal large language model for 3D medical image analysis. Furthermore, we introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks. Through comprehensive evaluation, our method proves to be a robust model for 3D medical image analysis, outperforming existing solutions. All code, data, and models are publicly available at: https://github.com/BAAI-DCAI/M3D.

1. Introduction

Medical MLLM research has largely focused on 2D images, while 3D analysis remains difficult despite the richer spatial information of 3D data. This work addresses that gap with M3D-Data, M3D-LaMed, and M3D-Bench for broad 3D medical analysis.

  • Existing medical MLLMs mainly adapt 2D models for medical image tasks, while 3D methods can require costly slice-by-slice analysis or fail outright.
  • M3D-Data contains 120K image-text pairs and 662K instruction-response pairs covering diverse 3D medical diseases and tasks.
  • M3D-LaMed directly understands and reasons about 3D images through a pretrained 3D vision encoder and an efficient 3D spatial pooling perceiver.
  • M3D-LaMed supports retrieval, report generation, VQA, positioning, and segmentation, including 3D referring expression segmentation through combination with a promptable segmentation model.
  • M3D-Bench evaluates 3D medical analysis across eight tasks spanning image-text retrieval, report generation, VQA, positioning, and segmentation.

2. Related Work

Prior medical MLLM research and datasets have largely centered on 2D images, while M3D-Data expands 3D coverage across multimodal tasks and evaluation settings.

  • Existing medical MLLMs are commonly fine-tuned from powerful 2D models, and large-scale datasets initially enabled training primarily for 2D images.
  • M3D-Data combines image-text pairs with instruction-response pairs and covers more 3D medical tasks than the compared datasets.
  • The dataset supports both language tasks and vision tasks, including 3D vision-language positioning and segmentation for identification and localization.

3. Dataset

M3D-Data assembles 3D CT image-text and instruction-response resources from public medical sources, using automated generation, filtering, and expert validation to support diverse tasks.

  • 120K image-text pairs and 662K instruction-response pairs cover eight 3D medical tasks in M3D-Data.
  • Image-text data: Publicly accessible medical websites provide image-report cases and expert-reviewed captions, enabling construction of the 120K-pair M3D-Cap dataset for retrieval and report generation.
  • Instruction data: The instruction-response collection supports VQA, vision-language positioning, and segmentation, totaling 662K pairs.
  • VQA generation: Public LLMs generate VQA instruction-response pairs from diagnostic reports through prompting, reducing reliance on costly specialized medical expertise.
  • VQA generation: Self-filtering removes noisy VQA data, while LLM and expert validation yields a 99.4% pass rate, or 13729/13814.
  • Positioning and segmentation: Positioning and segmentation use unified image-mask-text triplets, combining public densely annotated CT datasets with a Totalsegmentator-derived annotated subset.

4. Method

M3D-LaMed combines a 3D vision encoder, spatial pooling perceiver, LLM, and promptable segmentation module to process 3D medical images across multimodal tasks. Its training uses contrastive image-text pretraining followed by staged end-to-end instruction tuning.

  • 3D Perceiver: A 3D spatial pooling perceiver reconstructs encoder tokens into 3D space, reduces token count, and aligns embedding dimensions with the LLM.The design reduces computational cost while retaining spatial information.
  • LLM Integration: The model inserts refined 3D image embeddings into LLaMA-2-7B for multimodal generation and understanding.LLaMA-2-7B is used directly as the base LLM.
  • Promptable Segmentation: An output [SEG] token is mapped through an MLP into a prompt for a promptable 3D segmentation module, producing a 3D mask.The implementation uses SegVol as the promptable segmentation module.
  • 3D Image Encoder: The vision encoder is pretrained from scratch with a CLIP-like cross-modal contrastive strategy on M3D-Cap image-text pairs.
  • Training: Training first tunes the perceiver with frozen vision and language components, then fine-tunes the vision encoder, perceiver, LLM, and segmentation module with instruction data.LoRA is used for parameter-efficient LLM fine-tuning.

5. Evaluation and Benchmark

M3D-Bench evaluates 3D multimodal models across retrieval, report generation, VQA, positioning, and segmentation. Its task-specific protocols use curated test sets, multiple difficulty or question categories, and metrics suited to each output type.

  • Benchmark Scope: M3D-Bench covers 8 tasks across image-text retrieval, report generation, VQA, positioning, and segmentation.
  • Image-Text Retrieval: Retrieval evaluation measures image-to-text and text-to-image recall at ranks 1, 5, and 10 across difficulty levels.The test set contains 2,000 image-text pairs divided into four difficulty levels.
  • Report Generation: Report generation is evaluated on 2,000 test pairs using traditional metrics and Qwen-72B-based scoring of content overlap.
  • VQA: VQA evaluation uses multiple-choice questions with four answer choices and includes basic and small test sets for closed-ended and open-ended settings.The basic set contains 13,791 questions, while validation combines LLM and expert review.
  • Positioning: Positioning tasks localize targets or generate descriptions for regions, with masks converted into 3D bounding-box coordinates.The coordinates use the maximum bounding rectangle representation.
  • Segmentation: Segmentation covers semantic and referring-expression segmentation, with Dice used to evaluate masks.

6. Experiments

Experiments evaluate M3D-LaMed across retrieval, report generation, VQA, positioning, and segmentation, with ablations of its training and architecture. The model outperforms comparison methods on several tasks, while unlocking the vision encoder particularly benefits positioning REG.

  • 6.2. Results on Image-Text Retrieval: The model outperforms PMC-CLIP by 54% in image-to-text retrieval under the R@10 metric.Under the most difficult setting, it exceeds PMC-CLIP by 17.95% in IR at R@1.
  • 6.3. Results on Report Generation: M3D-LaMed with MLP exceeds RadFM by 2.92% in BLEU and by 4.17% under Qwen-72B evaluation for report generation.MLP performs slightly better than the Linear perceiver.
  • 6.4. Results on VQA: M3D-LaMed significantly outperforms RadFM on both closed-ended and open-ended VQA.Closed-ended performance is higher across all five problem types.
  • 6.5. Results on Positioning: Unlocking the vision encoder increases REG accuracy by 29.25%, while improvements on REC are not consistent.REG generates a box from the 3D image, whereas REC takes a box as input.
  • 6.6. Results on Segmentation: The model surpasses SegVol in several semantic-segmentation aspects and adds referring-expression segmentation capabilities absent from SegVol.The evaluation covers semantic segmentation and referring-expression segmentation.
  • 6.7. Ablation Study: Ablations indicate that vision pre-training, spatial pooling, the MLP, and an unlocked vision encoder each contribute to closed-set VQA performance.The authors identify visual pre-training followed by vision-encoder unlocking as a better training solution.

7. Conclusion

The paper concludes that M3D-Data, M3D-LaMed, and M3D-Bench advance 3D medical image analysis across multiple vision-language tasks. It presents the approach as a robust foundation for 3D medical MLLM research and application.

  • 7. Conclusion: M3D-LaMed is a generalist model for retrieval, report generation, VQA, positioning, and segmentation, while M3D-Bench evaluates eight tasks.The authors state that the approach provides a robust foundation for understanding 3D medical vision and language.

A.1. Modules

The 3D spatial pooling perceiver reduces and aligns 3D image tokens before injecting them into the LLM. M3D-LaMed combines this module with a 3D ViT, LLaMA-2-7B, and SegVol.

  • A.1. Modules: The 3D spatial pooling perceiver reconstructs tokens into 3D space for pooling, deconstructs them into sequence tokens, and aligns dimensions for the LLM.The architecture reduces token count and embedding dimensions before multimodal integration.

A.2. Datasets

The appendix documents the model configuration, prompt resources, and M3D-Seg dataset used for 3D positioning and segmentation. M3D-Seg contains labeled CT volumes collected from public datasets.

  • A.2. Datasets: The 3D spatial pooling perceiver reconstructs input tokens in 3D space before pooling and converts the pooled sequence to the LLM dimension.The architecture is presented as the mechanism for reducing token count and aligning dimensions.
  • A.2. Datasets: The appendix provides prompts and templates for data generation, data checking, model evaluation, task instruction, and a term dictionary.Figures 2 and 3 cover data generation prompts, while later figures provide evaluation and task templates.
  • A.2. Datasets: M3D-Seg contains 5,772 labeled 3D CTs from 25 public datasets for positioning and segmentation tasks.The dataset includes corresponding masks, with data, download links, and processing scripts designated for public release.

C. Qualitative Analysis

Qualitative analyses span eight 3D medical tasks, comparing retrieval, generation, question answering, positioning, and segmentation behavior. The visualizations report strong performance while clarifying task-specific inputs, outputs, and limitations.

  • Task coverage: Qualitative analysis covers eight tasks, including image-text retrieval, report generation, closed- and open-ended VQA, positioning, and segmentation.The positioning tasks are referring expression comprehension and generation; the segmentation tasks are semantic and referring expression segmentation.
  • Data generation: The VQA generation pipeline inserts image filenames and report text into an LLM prompt, then post-processes the output into VQA data.Qwen-72B was selected over ChatGPT because the experiments found similar performance and lower cost.
  • Positioning and segmentation: Instruction templates map categories, descriptions, and bounding boxes to inputs or outputs for referring-expression comprehension, generation, and segmentation.Segmentation answers include the special token [SEG], which guides the segmentation module.
  • Positioning and segmentation: The term dictionary supplies multiple descriptions for medical terms, supporting semantic-category conversion for vision-language positioning and segmentation.The descriptions are generated through ChatGPT.
  • Comparative results: Qualitative retrieval examples show top-ranked samples similar to ground truth, while report-generation comparisons favor the proposed model over RadFM.For report generation, GPT-4V was not included because it struggled to produce medical diagnostic recommendations.
  • Comparative results: Across closed- and open-ended VQA, positioning, and segmentation visualizations, the proposed model is described as superior or effective, with GPT-4V restrictions noted for some medical questions.Green boxes or masks denote ground truth, and red boxes or masks denote predictions.
Loading 2404.00578v1…