PaperlayerView exact revision on arXiv

Source-linked AI summary

ACE-LoRA: Graph-Attentive Context Enhancement for Parameter-Efficient Adaptation of Medical Vision-Language Models

M. Arda Aydın, Melih B. Yilmaz, Aykut Koç, Tolga Çukur

arXiv:2603.17079v1cs.CV

TL;DR

Medical VLMs must balance specialist detail against generalist coverage, while existing PEFT methods often miss localized cues and rely on labeled few-shot data. ACE-LoRA adapts a frozen generalist model with LoRA, hypergraph-based context enhancement, and label-guided InfoNCE, achieving strong zero-shot transfer across medical benchmarks with only approximately 0.95M trainable parameters.

  • Problem

    Medical VLMs struggle to balance specialist models’ domain-specific detail and poor generalization with generalist models’ broad semantics and reduced fine-grained diagnostic cues.

  • Method

    ACE-LoRA inserts LoRA into frozen image-text encoders, uses ACE-HGNN for higher-order local-global interactions, and applies label-guided InfoNCE to reduce false negatives.

  • Results

    ACE-LoRA consistently outperforms medical VLM and PEFT baselines across zero-shot classification, segmentation, and detection benchmarks with approximately 0.95M trainable parameters.

  • Takeaways & Limitations

    The results support parameter-efficient adaptation of generalist medical VLMs for robust zero-shot transfer to unseen domain-specific datasets and data-limited clinical settings.

Abstract

from arXiv · show

The success of CLIP-like vision-language models (VLMs) on natural images has inspired medical counterparts, yet existing approaches largely fall into two extremes: specialist models trained on single-domain data, which capture domain-specific details but generalize poorly, and generalist medical VLMs trained on multi-domain data, which retain broad semantics but dilute fine-grained diagnostic cues. Bridging this specialization-generalization trade-off remains challenging. To address this problem, we propose ACE-LoRA, a parameter-efficient adaptation framework for generalist medical VLMs that maintains robust zero-shot generalization. ACE-LoRA integrates Low-Rank Adaptation (LoRA) modules into frozen image-text encoders and introduces an Attention-based Context Enhancement Hypergraph Neural Network (ACE-HGNN) module that captures higher-order contextual interactions beyond pairwise similarity to enrich global representations with localized diagnostic cues, addressing a key limitation of prior Parameter-Efficient Fine-Tuning (PEFT) methods that overlook fine-grained details. To further enhance cross-modal alignment, we formulate a label-guided InfoNCE loss to effectively suppress false negatives between semantically related image-text pairs. Despite adding only 0.95M trainable parameters, ACE-LoRA consistently outperforms state-of-the-art medical VLMs and PEFT baselines across zero-shot classification, segmentation, and detection benchmarks spanning multiple domains. Our code is available at https://github.com/icon-lab/ACE-LoRA.

1 Introduction

Medical VLMs face a specialization–generalization trade-off: specialist models capture domain-specific cues but generalize poorly, while generalist models retain broad semantics but may lose fine-grained diagnostic details. ACE-LoRA addresses this challenge with lightweight adaptation, hypergraph context enhancement, and label-guided contrastive learning.

  • Motivation: Specialist medical VLMs capture domain-relevant patterns but are constrained by limited modality-specific data and weak generalization beyond training distributions.MIMIC-CXR contains about 377K image-report pairs, compared with CLIP’s 400M pairs.
  • Motivation: Generalist medical VLMs broaden semantic coverage through large multimodal corpora but can lose fine-grained anatomical cues needed for domain-specific evaluation.The passage gives subtle opacity variations in chest X-rays as an example.
  • Motivation: Existing PEFT methods freeze the backbone and update few parameters, but often overlook localized diagnostic patterns and depend on explicitly labeled few-shot data.Large-scale medical datasets more commonly provide paired images and reports than curated task labels.
  • Approach: ACE-LoRA combines LoRA modules in frozen image-text encoders with ACE-HGNN to bridge specialist expertise and generalist capabilities through efficient adaptation.ACE-HGNN integrates local and global embeddings using hypergraph message passing based on attention affinities and token similarity.
  • Approach: ACE-HGNN models higher-order interactions among semantically related image regions and text tokens, while label-guided InfoNCE mitigates false negatives in medical contrastive learning.These components target structured cross-modal alignment rather than isolated pairwise interactions.
  • Evaluation: ACE-LoRA demonstrates robust zero-shot transfer with approximately 0.95M trainable parameters and is evaluated across zero-shot classification, segmentation, and detection tasks.The reported parameter count is approximately 0.48% of full fine-tuning requirements.

2 Related Work

Prior medical VLMs divide into specialist models trained on limited modality-specific data and generalist models trained on broader multimodal corpora. Existing PEFT methods mainly use prompts or adapters and often target labeled few-shot classification, leaving image-report-based medical adaptation underexplored.

  • Medical VLMs: Specialist medical VLMs use relatively limited modality-specific image-text datasets, whereas generalist models expand coverage across multiple imaging modalities.The related work discusses ConVIRT as an early specialist contrastive-learning approach.
  • Medical VLMs: Generalist systems such as PMC-CLIP and BiomedCLIP build on large PubMed or PubMed Central collections, including datasets with 1.6M and 15M image-text pairs.BiomedCLIP adapts CLIP to medicine while extending context length.
  • PEFT Methods: Natural-image PEFT methods primarily rely on prompt learning or lightweight adapters added within or on top of frozen encoders.The passage contrasts learnable prompt tokens with residual-style adapters and text-feature biases.
  • PEFT Methods: CLIP-LoRA inserts LoRA modules into query, key, and value projections of both encoders across all layers.This distinguishes LoRA-based adaptation from prompt-learning and adapter-based approaches.
  • Medical PEFT: Medical PEFT remains underexplored because methods such as BiomedCoOp target few-shot classification with labeled medical data, while large datasets predominantly provide image-report pairs.This creates a mismatch between common PEFT supervision and available medical multimodal data.

3 Method

ACE-LoRA combines frozen-encoder LoRA adaptation with ACE-HGNN hypergraph message passing and label-guided contrastive learning. The method uses transformer affinities and local token similarities to enrich representations with higher-order contextual information.

  • LoRA adaptation: ACE-LoRA inserts LoRA modules into BiomedCLIP’s self-attention projections while keeping the image-text encoders frozen.The trainable low-rank update is applied to the query, key, and value projection matrices.
  • ACE-HGNN construction: ACE-HGNN treats encoder outputs as vertices and constructs hyperedges from transformer-derived token affinities.The module is applied to both image and text encoders, with image embeddings described as one global and N local representations.
  • ACE-HGNN construction: Each hyperedge captures a token’s contextual neighborhood, enabling aggregation across semantically related groups of patches and the global representation.This extends beyond isolated pairwise connections to structured interactions among multiple tokens.
  • Affinity construction: ACE-HGNN forms global-to-local affinities from averaged transformer attention and local-to-local affinities from cosine similarity between normalized patch features.The raw affinity matrix combines global attention context with local patch similarity.
  • Hypergraph construction: Top-k filtering retains the strongest connections, softmax normalizes hyperedge weights, and strict self-connections preserve pretrained features.The incidence matrix sets diagonal entries to one and nonselected connections to zero.
  • Hypergraph message passing: Hypergraph message passing aggregates node information into hyperedges and maps hyperedge features back to vertices through bottleneck projection functions.The resulting representation is enriched with attention-guided context and local patch-similarity structure.
  • Label-guided contrastive learning: The label-guided InfoNCE formulation avoids separating nonmatching image-text pairs that share the same disease label.This addresses false negatives produced when clinically similar reports are treated as negatives.

4 Experiments

ACE-LoRA is evaluated across zero-shot classification, semantic segmentation, and object detection in radiology and histopathology, using comparisons with medical VLMs and PEFT methods. It achieves leading performance across these tasks while retaining parameter and data efficiency, and its ablations and visualizations examine the contributions of its components and localization behavior.

  • Zero-shot image classification: ACE-LoRA surpasses other PEFT strategies across all zero-shot classification datasets using approximately 0.95M trainable parameters.Full fine-tuning updates approximately 197M backbone parameters, whereas ACE-LoRA uses approximately 0.95M.
  • Zero-shot image classification: ACE-LoRA improves over specialist and generalist medical VLMs on the CheXpert 5×200, RSNA, and SIIM zero-shot classification datasets.The experiments report inconsistent cross-dataset generalization for specialist models and weaker chest-X-ray performance for generalist models.
  • Zero-shot image classification: ACE-LoRA substantially improves over BiomedCLIP and outperforms PLIP and QuiltNet on three histopathology benchmarks while training with 140K image-text pairs.PLIP and QuiltNet use 210K and 1M image-text pairs, respectively.
  • Dense prediction: ACE-LoRA remains the top performer in semantic segmentation and object detection, with local-context and high-order interaction design improving BiomedCLIP and pneumonia detection.The strongest competing detection results come from methods emphasizing local alignment, including GLoRIA and MGCA.
  • Ablation and analysis: Ablations show that LoRA, ACE-HGNN, and label-guided InfoNCE jointly produce the best overall classification results, while ACE-LoRA also improves BMC-CLIP.ACE-HGNN restores RSNA performance after LoRA’s slight degradation, and attention-based hypergraph modeling is linked to stronger higher-order relationship modeling.
  • Ablation and analysis: Cross-modal similarity maps show ACE-LoRA highlighting abnormal regions in distributed and localized bilateral pathology cases where BiomedCLIP misses relevant findings.The visualization compares ACE-LoRA and BiomedCLIP against RSNA ground-truth detection labels.

5 Conclusion

ACE-LoRA adapts a generalist medical VLM for robust zero-shot transfer to unseen domain-specific datasets while refining local and global embeddings. Its lightweight, locality-aware design supports strong performance in data-limited clinical settings.

  • ACE-LoRA adapts a generalist medical VLM for robust zero-shot transfer to unseen domain-specific datasets.
  • By refining both local and global embeddings, ACE-LoRA captures the semantic complexity of medical images more effectively.
  • ACE-LoRA enables BiomedCLIP to achieve strong performance in data-limited clinical settings.
  • Strong zero-shot generalization of PEFT-based methods over medical VLMs trained from scratch underscores this research direction's promise.

A Additional Method Details

This section compares attention formulations used in GAT and GATv2 for computing graph attention scores. GATv2 modifies the operation order to address GAT’s limited expressive power.

  • Table 6 compares attention-map edge weights with attention coefficients learned from scratch using GAT and GATv2 formulations.The comparison evaluates alternative attention designs for the approach.
  • GAT: GAT computes attention scores from transformed node features, concatenation, a learnable weight vector, and LeakyReLU normalization.The LeakyReLU activation uses a negative slope of 0.2.
  • GATv2: GATv2 modifies the attention operation order to enhance expressive power during attention computation.Its formulation applies the linear transformation to concatenated node features before LeakyReLU and scoring.

B.1 Additional Implementation Details for Downstream Tasks

The downstream-task implementation uses task-specific optimization, augmentation, loss, and feature-extraction settings for semantic segmentation and object detection. Dataset specifications and statistics are summarized separately in Table A.1.

  • Semantic Segmentation: Semantic segmentation training uses flips and random rotations, AdamW, and a cross-entropy–Tversky loss combination to address class imbalance.The learning rate is 1e−3 and weight decay is 1e−2.
  • Dataset Statistics: Table A.1 summarizes dataset specifications and statistics, including class counts and total image counts.“Classes” denotes the number of categories, while “Total Size” denotes the number of images.
  • Semantic Segmentation: Segmentation features come from frozen image-encoder layers {2, 4, 7, 9, 11}, with batch size 4 on one NVIDIA RTX 4090 GPU.The listed layers are zero-indexed.
  • Object Detection: Object detection uses AdamW with warmup during the first epoch followed by polynomial learning-rate decay.The learning rate is 1e−4 and weight decay is 1e−2.
  • Object Detection: Detection features come from frozen image-encoder layers {3, 7, 11}, with batch size 8 on one NVIDIA RTX 4090 GPU.The listed layers are zero-indexed.

B.2 Dataset Details

The paper describes dataset curation, class-name references, prompt templates, and a multi-dataset PEFT benchmark. The benchmark evaluates parameter efficiency, computational cost, and zero-shot accuracy across three datasets.

  • Histopathology Dataset Curation: Histopathology data were curated from PMC-OA using keyword filtering over image captions to isolate relevant samples.The source corpus contains multiple medical imaging modalities, motivating targeted filtering.
  • Class Names: Table A.2 lists the class names used for zero-shot image classification across datasets.
  • Prompt Templates: Zero-shot classification uses CLIP-style prompt templates tailored to chest X-ray and histopathology datasets.Chest X-ray prompts include disease-description templates, while histopathology prompts are adopted from QuiltNet.
  • PEFT Benchmark: PEFT methods are compared using trainable parameters, forward-pass computational cost, and average zero-shot accuracy across CheXpert 5×200, RSNA, and SIIM.The comparison is reported in Fig. A.1.
  • PEFT Benchmark: ACE-LoRA achieves state-of-the-art performance with competitive parameter counts and computational overhead, while CoCoOp is more parameter-efficient but performs suboptimally.The passage notes that ACE-HGNN adds only marginal overhead, but the sentence is truncated before specifying the amount.

D Additional Ablation Studies

Additional ablations show that ACE-HGNN improves LoRA-only adaptation across encoders and remains robust across inference strategies, neighborhood sizes, ranks, and pre-training data scales.

  • ACE-HGNN integration: ACE-HGNN applied to either encoder consistently outperforms the LoRA-only baseline, while applying it to both encoders produces the most balanced results across benchmarks.Image-only integration gives substantial gains across all datasets, and text-only integration achieves the best result on CheXpert 5×200.
  • Transductive inference: TransCLIP yields marginal gains over BiomedCLIP on two benchmarks but slightly worsens performance on RSNA, suggesting unsupervised domain adaptation alone may be insufficient.The comparison evaluates a transductive method that uses unlabeled target-domain samples during inference.
  • Neighborhood selection: k = 5 yields optimal results, while performance remains consistently high across tested k values and benchmarks.Even k = 1 produces substantial zero-shot gains over the LoRA-only baseline.
  • LoRA rank: r = 4 delivers the best zero-shot performance across the evaluated datasets.The rank sensitivity analysis compares zero-shot accuracy for different r values.
  • Data scaling: Pre-training data scale has a positive correlation with downstream performance from 1% to 100%, with even 1% improving over the baseline.Performance continues increasing as the data scale approaches 100%.

E Qualitative Results

Qualitative comparisons on SIIM segmentation and RSNA detection show ACE-LoRA outperforming PRIOR and BiomedCLIP across different image encoder backbones.

  • Cross-backbone comparison: ACE-LoRA outperforms PRIOR and BiomedCLIP on both semantic segmentation and object detection tasks.The comparisons use SIIM for segmentation and RSNA for detection, with GT denoting ground truth.
Loading 2603.17079v1…