Source-linked AI summary
Localization Lens for Improving Medical Vision-Language Models
Hasan Farooq, Murtaza Taj, Mehwish Nasim, Arif Mahmood
TL;DR
Med-VLMs can struggle to understand anatomical structures and spatial positioning, which matter for medical reasoning. The paper adds expert-validated localization representations, pixel shuffle, and decoupled contrastive alignment to the Med-VLM pipeline; experiments report improved localization context and performance across Med-VLMs.
Problem
Med-VLMs often struggle to understand anatomical structures and spatial positioning, despite their importance for medical reasoning.
Source: §1 · pp. 1–2Method
The approach adds expert-validated localization representations and pixel shuffle, then aligns the representations with decoupled contrastive learning before Med-VQA integration.
Source: §1 · p. 2Results
Experiments on Med-VQA datasets report improved localization context and performance across Med-VLMs.
Source: §4 · p. 7Takeaways & Limitations
The localization-aware enhancement is described as model-agnostic and easily integrable into existing Med-VLMs.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Medical Vision-Language Models (Med-VLMs) have demonstrated strong capabilities in clinical tasks. However, they often struggle to understand anatomical structures and spatial positioning, which are crucial for medical reasoning. To address this, we propose a localization-aware enhancement to the Med-VLM pipeline, introducing improvements at three levels: data,architecture, and alignment. First, we introduce localization lens, a set of expert-validated representations that provide richer anatomical and positional context. However, as these representations increase input complexity, we integrate pixel shuffle within the model architecture to filter and refine representations, enhancing spatial information processing while preserving anatomical continuity. Lastly, to effectively align the localization lens representations with textual features, we incorporate decoupled contrastive loss (DCL) alongside the standard loss function. This ensures better feature discrimination and robustness, particularly in data limited medical settings. Through extensive evaluations on medical visual question answering (Med-VQA) datasets, we show that our methodology improves localization-driven performance across different Med-VLM architectures. Our analysis of localization-based questions further reveals that improvements in anatomy and spatial reasoning directly enhance the overall accuracy of Med-VQA upto 6.2%. The proposed approach is model-agnostic and can be seamlessly integrated into existing Med-VLM pipelines. The dataset, code, and trained models will be made publicly available at https://github.com/CVLABLUMS/localizationlens.
1 Introduction
Med-VLMs can struggle to encode the fine-grained anatomical and spatial cues needed for clinical reasoning, particularly in smaller models. The paper proposes expert-validated localization representations, pixel shuffle, and vision-language alignment to improve anatomical and spatial reasoning across Med-VLM architectures.
1 Introduction
- Smaller Med-VLMs often struggle to encode fine-grained localization cues needed for disease detection, lesion identification, and reporting.Anatomical structures and spatial relationships are essential for accurate clinical reasoning.Source: §1 · p. 2
- Expert-validated, clinically meaningful representations act as a localization lens to enhance anatomical and positional understanding in medical images.Source: §1 · p. 2
- Pixel shuffle addresses the localization lens's increased input complexity while improving capture and refinement of spatial and anatomical details.Source: §1 · p. 2
- The proposed vision-language pipeline aligns localization-lens representations using decoupled contrastive learning before integrating them into Med-VQA tasks.Source: §1 · p. 2
- The localization analysis reports significant improvements in spatial reasoning and anatomical understanding across different Med-VLM architectures.Source: §1 · p. 2
2 Methodology
The method enriches Med-VQA images with expert-validated localization representations, refines their processing with pixel shuffle, and trains visual and question-answer alignment in two phases. It then performs vision-language tasks using the original image alone.
2.1 Preparation of Localization Lens
- Localization lens combines original, single-color segmented, multi-color segmented, and masked images to add anatomical and positional context.Experts at a hospital and research center validated the representations' quality and clinical relevance.Source: §2.1 · p. 2
2.1.1 Uni and Multi-color Segmented Representations
- SAM produces coarser segmentation maps that may be medically imprecise but support generalization, while Med-SAM captures anatomical structures and granular image details.The paper attributes SAM's generalization benefit to ablation studies in Table 3.Source: §2.1.1 · p. 3
- The method unions segmentation masks into a full map and blends them with the original image using single- or multi-color coding.The blending weight α is sampled from a normal distribution centered at 0.50 with standard deviation 0.10; each mask receives a color.Source: §2.1.1 · pp. 3–4
2.1.2 Masked Representations
- Masked representations use expert-assisted segmentation maps to obscure background regions, helping the model focus on relevant features.The paper also describes a VLM-based promptable pipeline for automatically extracting masks on other datasets.Source: §2.1.2 · p. 4
2.2 Architectural Changes
- For each image-text pair, the model randomly selects two segmented or masked representations, encodes them independently, and fuses their embeddings with the original image embedding.The fusion uses an attention-based vision encoder mechanism.Source: §2.2 · p. 4
- Pixel shuffle rearranges visual tokens into a compact form, reducing patch count while preserving local dependencies and fine-grained continuity across representations.The paper says this filtering can reduce redundant tokenization and help select semantically rich tokens for attention, computational efficiency, and vision-text alignment.Source: §2.2 · pp. 4–5
2.3 Training Details
- Training proceeds in two phases: DCL aligns visual representations using organ prompts, then InfoNCE aligns localization-lens representations with question-answer pairs.DCL removes explicit negative repulsion during pre-training; the authors say this helps preserve meaningful medical variations and supports low-batch efficiency, while the second phase uses standard InfoNCE.Source: §2.3 · p. 5
2.4 Inference
- After training with localization lens, the model can perform vision-language tasks using only the original image.Source: §2.4 · p. 5
3 Experiments
Experiments evaluate the approach across medical and non-medical VLMs on VQA-RAD and SLAKE, then examine localization-question performance and representation ablations. The reported results show improvements across model types and localization questions.
3.1 Medical VLMs
- The evaluation uses four medical and two non-medical VLMs on VQA-RAD and SLAKE with a 70:30 train-test split; closed-category questions use accuracy and open-category questions recall.Source: §3.1 · pp. 5–6
- Both medical and non-medical VLMs improve; models trained from scratch show more improvement, possibly because pretraining aligns the localization lens with organ labels [3].Source: §3.1 · p. 6
3.2 Localization Analysis
- Up to 8.0%: the method improves Med-VLM performance on localization Q/A pairs in combined VQA-RAD and SLAKE, measured by recall.The authors report that these localization-question improvements contribute to overall performance improvements across Med-VLMs.Source: §3.2 · p. 6, Table 2 · p. 7
3.3 Ablation Studies
- Up to 6.2%: combining SAM and Med-SAM segmentation maps improves results, while adding masked representations to segmentation maps achieves the best results.The ablation compares original images, segmentation maps, and masked representations.Source: Table 3 · p. 7, §3.3 · p. 7
4 Conclusion
The conclusion presents localization lens representations, pixel shuffle, and decoupled contrastive alignment as a model-agnostic enhancement to Med-VLMs. Experiments on Med-VQA datasets show improved performance across models, and the approach can be integrated into existing pipelines.
- Localization lens representations paired with pixel shuffle and decoupled contrastive alignment improve Med-VQA performance across models and can be integrated into existing pipelines.Source: §4 · p. 7