PaperlayerView exact revision on arXiv

Source-linked AI summary

MMRL: Multi-Modal Representation Learning for Vision-Language Models

Yuncheng Guo, Xiaodong Gu

arXiv:2503.08497v2cs.LGcs.CV

TL;DR

Few-shot adaptation of pretrained VLMs can overfit and weaken generalization to new tasks. MMRL introduces a shared learnable multimodal representation space and separates representation-token adaptation from class-token knowledge during inference. Across 11-dataset base-to-novel evaluation, it establishes a new state of the art while balancing adaptation and generalization.

  • Problem

    Few-shot VLM adaptation can overfit because existing methods primarily optimize class-token features with task-specific objectives, reducing generalization and zero-shot capabilities.

  • Method

    MMRL maps a shared learnable representation space into image and text tokens, adapts representation features at higher encoder layers, regularizes class features toward zero-shot features, and decouples inference.

  • Results

    MMRL establishes a new state of the art across 11 datasets, gaining 2.48% in Base, 0.36% in Novel, and 1.33% in HM over MMA.

  • Takeaways & Limitations

    MMRL balances task-specific adaptation with generalization by using representation features for base classes and preserved class features for new tasks.

  • Takeaways & Limitations

    Some alternative adaptation approaches using LLM text augmentation or whole-dataset distillation require computational resources beyond efficient transfer learning's intended scope.

Abstract

from arXiv · show

Large-scale pre-trained Vision-Language Models (VLMs) have become essential for transfer learning across diverse tasks. However, adapting these models with limited few-shot data often leads to overfitting, diminishing their performance on new tasks. To tackle this issue, we propose a novel Multi-Modal Representation Learning (MMRL) framework that introduces a shared, learnable, and modality-agnostic representation space. MMRL projects the space tokens to text and image representation tokens, facilitating more effective multi-modal interactions. Unlike previous approaches that solely optimize class token features, MMRL integrates representation tokens at higher layers of the encoders--where dataset-specific features are more prominent--while preserving generalized knowledge in the lower layers. During training, both representation and class features are optimized, with trainable projection layer applied to the representation tokens, whereas the class token projection layer remains frozen to retain pre-trained knowledge. Furthermore, a regularization term is introduced to align the class features and text features with the zero-shot features from the frozen VLM, thereby safeguarding the model's generalization capacity. For inference, a decoupling strategy is employed, wherein both representation and class features are utilized for base classes, while only the class features, which retain more generalized knowledge, are used for new tasks. Extensive experiments across 15 datasets demonstrate that MMRL outperforms state-of-the-art methods, achieving a balanced trade-off between task-specific adaptation and generalization. Code is available at https://github.com/yunncheng/MMRL.

1. Introduction

VLM adaptation can overfit scarce downstream data because existing prompt and adapter methods primarily optimize class-token features. MMRL addresses this with a shared high-layer representation space, preserved class-token knowledge, and decoupled inference.

  • VLMs capture complementary visual and textual information and support transfer across diverse downstream tasks.
  • Few-shot adaptation can reduce VLM generalization because prompt and adapter methods mainly optimize class-token features for task-specific objectives.
  • MMRL introduces a shared, modality-independent learnable space that maps tokens into image and text representation tokens for multimodal interaction.
  • MMRL preserves generalized knowledge by integrating representation tokens at higher encoder layers and retaining the original class token.
  • MMRL trains representation features while keeping the class-token projection fixed, regularizes class features toward zero-shot features, and decouples inference across base and new classes.Both representation and class features are used for base classes, whereas only class features are used for unseen classes or new datasets.

2. Related Work

Prior work adapts VLMs through multimodal representations, prompts, adapters, and auxiliary language-model or distillation methods. These approaches improve flexibility or cross-modal alignment, but some require substantial computation or may compromise generalization.

  • Vision-Language Models: VLMs learn joint image-language representations from large-scale image-text data and support diverse multimodal tasks.
  • Prompt Learning: Prompt-learning methods replace fixed templates with learnable vectors or distributions, incorporate visual cues, and align vision and text modalities.
  • MMRL Framework: MMRL training frameworks insert representation tokens into both encoders while optimizing only the representation space, mapping function, and representation-token projection layer.
  • Adapter-Style Learning: Adapter methods refine frozen VLM features with lightweight modules, cache training features, or integrate image and text branches through shared representations.
  • Efficiency Boundary: LLM-based text augmentation and whole-dataset distillation can increase computational requirements beyond efficient transfer learning's intended scope.

3. Method

MMRL extends CLIP with a shared representation space whose modality-specific tokens are inserted into higher encoder layers. It separately preserves class-token knowledge and combines or decouples features according to whether classes are seen or novel.

  • CLIP preliminaries: MMRL builds on CLIP’s image and text encoders, projecting their outputs into a shared vision-language latent space for classification.The image encoder uses a class token and patch tokens, while the text encoder uses the EOT token; cosine similarities with text classifiers produce class probabilities.
  • Learnable representation space: A shared, learnable space projects tokens into visual and textual representation tokens that are integrated into the encoders.The representation space contains K tokens of dimension dr, and a learnable mapping function projects them into the two modalities.
  • Learnable representation space: Representation tokens enter the image and text encoders only from layer J onward, leaving lower layers unchanged to preserve generalized knowledge.The text branch retains the original text-token sequence and inserts representation tokens before it; its attention mask is adjusted for the longer sequence.
  • Representation learning: MMRL optimizes class and representation features separately, while keeping the class-token projection frozen and training the representation-token projection.The class-feature objective is additionally regularized toward frozen CLIP features to preserve generalization.
  • Inference: For base classes, MMRL combines class and representation probabilities; for novel classes or datasets, it relies only on class-token features.The base-class probability uses α to weight class features and 1 − α to weight representation features.

4. Experiments

MMRL is evaluated across base-to-novel generalization, cross-dataset transfer, domain generalization, few-shot learning, and component ablations. Results show stronger adaptation and generalization, while ablations identify the importance of multimodal representation tokens, decoupled inference, insertion depth, and token count.

  • Evaluation Settings: MMRL is evaluated on base-to-novel generalization, cross-dataset transfer, domain shifts, and few-shot learning.Except for few-shot learning, experiments use 16 examples per category; few-shot settings use 1, 2, 4, 8, and 16 shots.
  • Base-to-Novel Generalization: 2.48%, 0.36%, and 1.33% gains over MMA are reported in average Base, Novel, and HM metrics across 11 datasets.MMRL establishes a new state-of-the-art based on these averages.
  • Cross-Dataset Evaluation: 1.03% accuracy improvement over MMA is achieved on ImageNet, with the highest average accuracy across target datasets in cross-dataset evaluation.The comparison trains on ImageNet and applies the model to unseen datasets without further fine-tuning.
  • Domain Generalization: MMRL attains top performance on 2 of 4 domain-shifted datasets, demonstrating generalization across diverse domains.The domain-generalization evaluation uses ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
  • Few-Shot Learning: MMRL achieves the best average performance across 11 datasets under all few-shot settings, with margins increasing as shot number rises.The figure compares MMRL with previous state-of-the-art methods on few-shot learning.
  • Ablation Analysis: Removing either modality’s representation tokens, the representation space, or decoupling strategy reduces performance, confirming their contributions.Using only class features harms base performance, while using representation and class features for novel classes reduces transfer performance.
  • Ablation Analysis: Higher-layer insertion improves performance, whereas excessive token counts eventually reduce novel-class accuracy through overfitting.Lower layers retain more generalizable features, while higher layers capture dataset-specific discriminative features; performance also declines when insertion is too high.

5. Conclusion

MMRL targets VLM adaptation across diverse downstream datasets by combining multimodal representation learning with decoupled inference. Its evaluations support a balance between task-specific adaptation and generalization.

  • Conclusion: MMRL bridges image and text modalities through a shared, unbiased representation space while preserving pre-trained knowledge in class tokens.The framework uses representation tokens for downstream adaptation and class tokens for retained generalization.
  • Conclusion: Decoupling representation and class tokens during inference mitigates overfitting risks and reinforces adaptability.The conclusion characterizes this design as balancing task-specific adaptation with generalization.
  • Conclusion: Extensive evaluations establish MMRL as a benchmark for efficient transfer learning.The reported conclusion emphasizes an optimal balance between adaptation and generalization.

A. Implementation Details

The implementation uses a CLIP ViT-B/16 backbone, 16-shot training for most experiments, and AdamW optimization with mixed precision. Representation-token initialization and insertion settings vary by dataset as specified.

  • Training Setup: MMRL uses CLIP ViT-B/16 as the visual backbone and adopts a 16-shot setting except for few-shot experiments.Hand-crafted text prompts from prior methods are also used.
  • Training Setup: Optimization uses AdamW with an initial learning rate of 0.001 and mixed-precision training.Mixed precision is used to speed training.
  • Representation Settings: Representation tokens use zero-mean Gaussian initialization with standard deviation 0.02 and begin at transformer layer 6.The representation-space dimension is 2048 for EuroSAT and 512 for other datasets.

B. Dataset Details

Dataset details are documented in the supplementary materials, which cover 14 datasets.

  • Dataset Details: Details of 14 datasets are provided in Table 7.

C. Computational Cost

MMRL provides a favorable computational trade-off, combining faster training with competitive inference and performance under reduced representation dimensionality. Its efficiency was evaluated under a standardized single-GPU setup.

  • Computational efficiency: MMRL and MMA exhibit significantly faster training, reducing overall computational costs.MMRL also offers faster inference than MMA and MetaPrompt, although MaPLe and PromptSRC achieve higher inference speeds.
  • Resource-constrained evaluation: Reducing MMRL’s representation dimensionality from 512 to 32 yields parameter counts comparable to MMA while still significantly outperforming the previous state-of-the-art model.
  • Evaluation setup: The computational comparison trains all methods on a single NVIDIA RTX 4090 GPU using ImageNet and publicly available implementations with default configurations.The evaluation distinguishes multimodal vision-language interaction from separate modality-specific fine-tuning.

D. Ablation Analysis on λ

The λ ablation shows that increasing regularization strength generally improves performance, with results typically becoming optimal or near-optimal at moderate-to-high values. The study evaluates these settings using harmonic mean across 11 datasets.

  • λ sensitivity: Increasing λ generally improves performance, with optimal or near-optimal results typically occurring between 4 and 6 across most datasets.The ablation evaluates performance using the harmonic mean metric across 11 datasets.
  • λ sensitivity: At higher λ values, additional increases have diminishing effects within the same dataset, indicating reduced sensitivity to precise tuning.
  • Evaluation scope: The broader dataset summary covers 14 datasets, while the λ ablation specifically reports results across 11 datasets.

E. Ablation Analysis on Regularization Strategies

Cosine regularization best preserves performance among the evaluated strategies, whereas L1 and MSE regularization degrade it, with MSE producing the strongest decline. Across few-shot evaluations, MMRL also achieves the highest average performance among compared methods.

  • Regularization comparison: Cosine regularization achieves the best performance among the evaluated strategies for retaining pre-trained knowledge.
  • Regularization comparison: L1 and MSE regularization degrade performance, with MSE causing a significant decline.
  • Regularization comparison: The authors attribute cosine regularization’s advantage to its more relaxed and flexible constraints, which preserve generalizability while capturing task-specific knowledge.
  • Few-shot performance: MMRL achieves the highest average performance across all shots in few-shot comparisons spanning 11 datasets.The comparison includes prior state-of-the-art methods; MMA results were reproduced from open-source code.
Loading 2503.08497v2…