PaperlayerView exact revision on arXiv

Source-linked AI summary

Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysis

Yu Xin, Gorkem Can Ates, Kuang Gong, Wei Shao

arXiv:2503.20047v3cs.CVeess.IV

TL;DR

Med3DVLM addresses the computational demands and image-text alignment difficulties of applying VLMs to 3D medical images. It combines DCFormer, SigLIP, and a dual-stream MLP-Mixer projector, achieving state-of-the-art performance across retrieval, report generation, and VQA benchmarks. The model is presented as computationally efficient for real-world clinical workflows, although hallucinated report content remains a challenge.

  • Problem

    Extending VLMs from 2D to 3D is limited by volumetric computational demands and difficulty aligning 3D spatial features with clinical text.

  • Method

    Med3DVLM combines DCFormer for efficient volumetric encoding, SigLIP for pairwise image-text alignment, and a dual-stream MLP-Mixer projector for multimodal fusion.

  • Results

    Med3DVLM achieves state-of-the-art performance across image-text retrieval, radiology report generation, and visual question answering on the M3D dataset.

  • Takeaways & Limitations

    The results support Med3DVLM as a computationally efficient model for 3D medical image analysis and real-world clinical workflows.

  • Takeaways & Limitations

    Med3DVLM occasionally generates hallucinated radiology-report content, including references to unrelated anatomical findings.

Abstract

from arXiv · show

Vision-language models (VLMs) have shown promise in 2D medical image analysis, but extending them to 3D remains challenging due to the high computational demands of volumetric data and the difficulty of aligning 3D spatial features with clinical text. We present Med3DVLM, a 3D VLM designed to address these challenges through three key innovations: (1) DCFormer, an efficient encoder that uses decomposed 3D convolutions to capture fine-grained spatial features at scale; (2) SigLIP, a contrastive learning strategy with pairwise sigmoid loss that improves image-text alignment without relying on large negative batches; and (3) a dual-stream MLP-Mixer projector that fuses low- and high-level image features with text embeddings for richer multi-modal representations. We evaluate our model on the M3D dataset, which includes radiology reports and VQA data for 120,084 3D medical images. Results show that Med3DVLM achieves superior performance across multiple benchmarks. For image-text retrieval, it reaches 61.00% R@1 on 2,000 samples, significantly outperforming the current state-of-the-art M3D model (19.10%). For report generation, it achieves a METEOR score of 36.42% (vs. 14.38%). In open-ended visual question answering (VQA), it scores 36.76% METEOR (vs. 33.58%), and in closed-ended VQA, it achieves 79.95% accuracy (vs. 75.78%). These results highlight Med3DVLM's ability to bridge the gap between 3D imaging and language, enabling scalable, multi-task reasoning across clinical applications. Our code is publicly available at https://github.com/mirthAI/Med3DVLM.

1. Introduction

Med3DVLM addresses the computational, spatial, and alignment challenges of extending vision-language models to 3D medical imaging. It combines an efficient volumetric encoder, SigLIP alignment, and a dual-stream projector for richer image-text representations.

  • Motivation: 3D VLMs face lost global context from slice-wise analysis, higher computational complexity, and limited public image-report pairs.These challenges arise because scans contain hundreds of slices and volumetric data have higher dimensionality.
  • Limitations of Existing Models: Existing 3D VLMs remain limited by computational cost, task specialization, and insufficient cross-modal interaction modeling.M3D-LaMed is described as costly on high-resolution volumes, while its MLP fusion may not capture complex interactions sufficiently.
  • Med3DVLM Innovations: DCFormer decomposes 3D convolutions into parallel 1D convolutions along depth, height, and width to capture volumetric features efficiently.The decomposition reduces computational complexity and supports richer, more scalable representations.
  • Med3DVLM Innovations: SigLIP independently optimizes each image-text pair with a pairwise sigmoid loss instead of relying on large batches of negative samples.This removes global batch similarity normalization and makes training more stable and less sensitive to batch size.
  • Med3DVLM Innovations: The dual-stream MLP-Mixer projector fuses detailed low-level and abstract high-level image features with LLM text embeddings.Stacked MLP layers are used to capture richer cross-modal interactions than simple linear projection.

2. Related Work

Related work spans contrastive medical image-text learning, generalist 2D/3D VLMs, report generation, and medical VQA. These approaches progressively incorporate generative architectures and large language models for multimodal tasks.

  • Medical Vision-Language Learning: ConVIRT and BioViL use biomedical image-text data to improve multimodal representations, classification, and zero-shot retrieval.ConVIRT applies contrastive learning, while BioViL refines representations using large-scale biomedical datasets.
  • 3D Medical VLMs: RadFM combines contrastive learning and generative modeling across 2D and 3D images, modalities, and radiology reports.It provides a unified radiology representation across X-ray, CT, and MRI data.
  • Report Generation: Report-generation systems evolved from CNN-LSTM encoder-decoder models toward transformer-based and pretrained-LLM architectures.Later systems such as R2GenGPT and Med-Flamingo integrate visual features with language models for more coherent or clinically useful reports.
  • Medical VQA: Medical VQA began primarily as classification over predefined answers and increasingly adopted generative sequence-prediction architectures.Generative systems support open-ended responses by extending language models with visual inputs through visual instruction tuning.

3.1. Dataset

The study uses the publicly available, de-identified M3D dataset for image-text retrieval, radiology report generation, and visual question answering. It draws on M3D-Cap and M3D-VQA subsets with standardized image volumes.

  • Dataset: The M3D dataset was collected with informed consent and ethical approval by the original investigators.The secondary analysis used de-identified data and required no additional consent or approval.
  • Dataset: M3D-Cap includes 120K image-text pairs, while M3D-VQA contains 662K instruction-response pairs.These subsets support retrieval, report generation, and VQA tasks.
  • Preprocessing: All image volumes were resampled to a fixed size of 128x256×256.The dataset was divided into training and other splits, although the supplied passage truncates the split description.

3.2. Med3DVLM

Med3DVLM combines an efficient 3D encoder, pairwise sigmoid alignment, and a dual-stream MLP-Mixer projector to connect volumetric image features with clinical text. Its training uses contrastive pretraining, projector pretraining, and parameter-efficient fine-tuning for report generation and VQA.

  • Architecture and training: Med3DVLM consists of a vision encoder, multimodal projector, and large language model trained through multiple stages.The vision encoder extracts volumetric features, the projector integrates visual and textual embeddings, and the LLM generates outputs.
  • Efficient 3D feature encoding: DCFormer decomposes 3D convolutions into parallel 1D depthwise convolutions along depth, height, and width to reduce computational complexity.The approach supports larger 3D volumes while preserving fine-grained spatial details.
  • Vision-language alignment: SigLIP independently optimizes image-text pairs with a pairwise sigmoid loss instead of relying on large batches of negative pairs.This removes batch-wide similarity normalization and reduces sensitivity to batch size, making the approach suitable for small-batch medical imaging settings.
  • Multi-modal projector: The dual-stream MLP-Mixer projector fuses low-level spatial and high-level semantic image features with text embeddings through token- and channel-mixing operations.Its low-high hybrid design preserves diverse information without exceeding the LLM token-length limit and produces richer joint representations.
  • Parameter-efficient fine-tuning: LoRA fine-tunes the multimodal projector and LLM using low-rank weight updates while keeping the remaining LLM parameters frozen.This reduces trainable parameters, training cost, and memory usage while preserving general knowledge.

3.3. Evaluation Metrics

The evaluation uses retrieval metrics for ranking image-text matches and text-generation metrics for lexical, recall-oriented, flexible, and semantic similarity assessment.

  • Image-text retrieval: Recall@K measures the percentage of correct matches within the top K retrieved items.R@1 requires the correct match to rank first, while R@5 and R@10 allow ranks within the top five or ten.
  • Report generation and open-ended VQA: BLEU measures generated-report precision through n-gram overlap with reference text.Its calculation uses brevity penalty, n-gram weights, and n-gram precision.
  • Report generation and open-ended VQA: ROUGE emphasizes recall by measuring how much reference text appears in the generated output.The metric uses recall of generated-report n-grams against reference n-grams.
  • Report generation and open-ended VQA: METEOR balances precision and recall while accounting for synonyms and stemming.Precision and recall are based on matching words between generated and reference reports.
  • Report generation and open-ended VQA: BERTScore evaluates semantic similarity using contextual embeddings of generated and reference words.The score compares deep learning-based contextual representations rather than only surface word overlap.

3.4. Implementation Details

Med3DVLM is implemented with a DCFormer vision encoder, multimodal projector, and language model across contrastive pretraining, VLM pretraining, fine-tuning, and inference stages.

  • Training setup: Med3DVLM uses PyTorch, Hugging Face Transformers, DeepSpeed ZeRO2, BF16 precision, and eight NVIDIA A100 80GB GPUs.Optimization uses AdamW with a 0.03 warmup ratio and cosine learning-rate scheduling.
  • Contrastive pretraining: Contrastive pretraining pairs resized 3D CT volumes with radiology reports using DCFormer-small and ClinicalBERT embeddings.Image features use mean pooling, while text features are obtained from the [CLS] representation.
  • VLM pretraining: VLM pretraining uses image-question-answer triplets from M3D-Cap and M3D-VQA, while freezing the DCFormer-small encoder.A hybrid MLP-Mixer projector fuses low-level image features with language-model inputs.
  • VLM fine-tuning: Fine-tuning includes yes/no questions and updates only LoRA modules and the projector while keeping other weights frozen.The configuration uses LoRA rank 16, scaling factor α = 32, and dropout 0.05.
  • Inference: Inference evaluates report generation and VQA from a 3D CT volume plus a natural-language prompt.Open-ended VQA generates free-form answers, whereas closed-ended VQA selects among answer options.

4. Results

Med3DVLM outperforms M3D-LaMed across retrieval, report generation, and VQA benchmarks, while qualitative examples show stronger clinical alignment alongside remaining hallucination and spatial-reasoning errors.

  • Image-text retrieval: Med3DVLM achieves the highest Recall@1 across retrieval test sets of 100, 500, 1000, and 2000 samples.The reported gains apply to both image-to-text and text-to-image retrieval.
  • Report generation: Med3DVLM outperforms all report-generation baselines across BLEU, ROUGE, and METEOR.Its METEOR rises from 14.38% to 36.42%, while ROUGE rises from 19.55% to 40.25%; BERTScore remains similar to M3D-LaMed.
  • Qualitative report analysis: Med3DVLM correctly identifies multiple chest-CT abnormalities, whereas M3D-LaMed fails to detect the liver pathology in the qualitative comparison.The example includes hepatic masses, portal vein thrombosis, and occlusive filling defects.
  • Qualitative report analysis: Med3DVLM still produces incorrect or hallucinated findings, including hepatic steatosis, contrast enhancement, and iliac arteriovenous malformation.These errors suggest overgeneralization of common radiological patterns and a need for better factual grounding.
  • VQA performance: Med3DVLM achieves state-of-the-art performance in open-ended and close-ended VQA, particularly for organ and abnormality identification.Open-ended METEOR improves from 33.58% to 36.76%, and ROUGE improves from 52.39% to 56.31%.
  • VQA performance: 79.75% accuracy in close-ended VQA surpasses M3D-LaMed’s 75.78%.Qualitative cases show correct identification of pleural effusion, stomach mass, and portal venous phase where M3D-LaMed misclassifies them.
  • Qualitative VQA analysis: Med3DVLM provides only partially correct answers for some cranial-fossa lesion-localization cases.The qualitative results identify remaining limitations in fine-grained spatial reasoning and uncertainty calibration.

4.4. Ablation Study

The ablations show that DCFormer, SigLIP, and the multi-scale MLP-Mixer each contribute to Med3DVLM, with their importance varying by task. DCFormer is especially important for visual understanding, while the projector and SigLIP provide gains in multimodal reasoning and alignment.

  • Core components: 36.42% METEOR is achieved by the full model for report generation, compared with 23.78% without MLP-Mixer, 13.48% without SigLIP, and 8.88% without DCFormer.Removing DCFormer also removes the dependent MLP-Mixer because it requires DCFormer’s multi-scale outputs.
  • Core components: 36.76% mean METEOR falls to 1.38% in open-ended VQA when both DCFormer and MLP-Mixer are removed, whereas removing only MLP-Mixer yields 34.65%.Removing SigLIP produces only a small decrease to 36.36%.
  • Core components: 79.75% mean accuracy is achieved in closed-ended VQA, declining to 1.26% without both DCFormer and MLP-Mixer, but only to 79.71% without SigLIP.Removing MLP-Mixer causes a smaller decline to 78.58%.
  • Vision encoder and loss: 61.00% IR R@1 and 63.90% TR R@1 are achieved by DCFormer-S with SigLIP on 2,000 samples, versus 38.65% and 39.70% with ViT3D.Switching from CLIP to SigLIP consistently improves retrieval across encoder types and test sizes.
  • Multi-modal projectors: 36.42% METEOR is achieved by the 2×MLP-Mixer-H projector for report generation, versus 15.10% with 2×MLP and 23.25% with single-scale 1×MLP-Mixer-H.The same projector reaches 36.76% open-ended VQA METEOR and 79.75% closed-ended VQA accuracy, exceeding 2×MLP results of 34.39% and 78.65%.

5. Discussion

Med3DVLM addresses major 3D VLM limitations through efficient encoding, small-batch-friendly alignment, and dual-stream feature fusion. Its efficiency gains are substantial, but hallucinations, spatial-reasoning errors, dataset scope, and non-equivalent runtime comparisons remain important boundaries.

  • Addressing existing limitations: DCFormer decomposes 3D convolutions to reduce complexity while preserving spatial detail, supporting processing of high-resolution image volumes.SigLIP avoids reliance on large negative batches, while the dual-stream MLP-Mixer fuses low- and high-level image features with text embeddings.
  • Challenges and future directions: Med3DVLM occasionally hallucinates unrelated anatomical findings in radiology reports, indicating a factual-grounding challenge for generative outputs.The authors suggest structured clinical knowledge as one possible direction for reducing hallucinations and improving interpretability.
  • Challenges and future directions: Errors in lesion localization and answer precision persist in open-ended VQA, indicating room for improved spatial reasoning and visual-text alignment.In a complex cranial-fossa case, Med3DVLM is partially correct while M3D-LaMed misidentifies the region entirely.
  • Challenges and future directions: The M3D benchmark primarily contains CT images and covers retrieval, report generation, and VQA, limiting evidence for broader modality, demographic, protocol, and task generalization.The authors recommend evaluation on MRI, ultrasound, varied clinical settings, and additional tasks such as synthesis, translation, and segmentation.
  • Efficiency: 18.2M parameters and 21.59G FLOPs replace 87.4M parameters and 253.23G FLOPs in the vision backbone, although Med3DVLM uses a larger projector.The total model size increases slightly, while the authors attribute the main efficiency gains to the encoder replacement.
  • Evaluation boundary: Direct inference-time and memory comparisons are not reported because M3D-LaMed includes a segmentation module absent from Med3DVLM, making runtime comparisons non-equivalent.The comparison therefore focuses on theoretical complexity metrics for the vision-language components.

6. Conclusion

Med3DVLM combines efficient volumetric encoding, sigmoid-based contrastive learning, and dual-stream feature fusion for 3D medical image analysis. Experiments on M3D report state-of-the-art retrieval, report-generation, and VQA performance while maintaining computational efficiency.

  • Conclusion: Med3DVLM integrates DCFormer, SigLIP, and a dual-stream MLP-Mixer projector for 3D medical image analysis.These components target volumetric encoding, image-text alignment, and multimodal projection, respectively.
  • Conclusion: Experiments on the M3D dataset demonstrate state-of-the-art performance in image-text retrieval, radiology report generation, and visual question answering.The conclusion presents these results as evidence of the model’s promise for automated, interpretable, and multimodal medical AI systems.
  • Conclusion: Med3DVLM maintains high accuracy while remaining computationally efficient, supporting its suitability for real-world clinical workflows.The paper frames 3D VLMs as potential foundational tools for automated and multimodal medical AI systems.
Loading 2503.20047v3…