PaperlayerView exact revision on arXiv

Source-linked AI summary

HeBA: Heterogeneous Bottleneck Adapters for Robust Vision-Language Models

Md Jahidul Islam

arXiv:2603.16653v1cs.CV

TL;DR

Few-shot adaptation of large vision-language models must address the mismatch between homogeneous adapters and the distinct structures of visual and textual tokens. HeBA uses modality-specific bottleneck adapters with active Kaiming initialization, and reports state-of-the-art performance across 11 few-shot benchmarks.

  • Problem

    Homogeneous, wide adapters treat spatially structured visual tokens and semantically dense text tokens uniformly, while few-shot adaptation must preserve frozen CLIP knowledge during domain adaptation.

  • Method

    HeBA freezes CLIP and combines modality-specific visual convolutions, textual linear bottlenecks, D → D/4 compression, and active Kaiming initialization.

  • Results

    81.35% Harmonic Mean across 11 benchmarks establishes HeBA as a new state-of-the-art for Base-to-Novel generalization.

  • Takeaways & Limitations

    HeBA’s architecture aligns adapter processing with modality structure while using compression as an intrinsic regularizer for frozen-backbone adaptation.

  • Takeaways & Limitations

    Compared with expanding adapters, inverse-bottleneck designs can increase parameter count and overfitting risk in data-scarce regimes, while unconstrained high-capacity designs are sensitive to hyperparameter tuning.

Abstract

from arXiv · show

Adapting large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks often suffers from a "one-size-fits-all" architectural approach, where visual and textual tokens are processed uniformly by wide, generic adapters. We argue that this homogeneity ignores the distinct structural nature of the modalities -- spatial locality in images versus semantic density in text. To address this, we propose HeBA (Heterogeneous Bottleneck Adapter), a unified architectural framework that introduces modality-specific structural inductive biases. HeBA departs from conventional designs through three key architectural innovations: (1) Heterogeneity: It processes visual tokens via 2D depthwise-separable convolutions to preserve spatial correlations, while distinctively processing text tokens via dense linear projections to capture semantic relationships; (2) Bottleneck Regularization: Unlike standard expanding adapters, HeBA employs a compression bottleneck (D -> D/4) that explicitly forces the model to learn compact, robust features and acts as a structural regularizer; and (3) Active Gradient Initialization: We challenge the restrictive zero-initialization paradigm, utilizing a Kaiming initialization strategy that ensures sufficient initial gradient flow to accelerate convergence without compromising the frozen backbone's pre-trained knowledge. Extensive experiments demonstrate that HeBA's architecturally specialized design achieves superior stability and accuracy, establishing a new state-of-the-art on 11 few-shot benchmarks. Code is available at https://github.com/Jahid12012021/VLM-HeBA.

1. Introduction

Few-shot adaptation of frozen vision-language models must balance preserving pretrained knowledge with adapting to new domains. HeBA addresses homogeneous adapters by combining modality-specific processing, compressive bottlenecks, and active initialization.

  • Motivation: Few-shot VLM adaptation faces a stability-plasticity dilemma because full fine-tuning is costly, risks catastrophic forgetting, and can degrade performance on novel classes.Parameter-efficient methods freeze the backbone and add lightweight learnable modules.
  • Motivation: Homogeneous adapters process spatially correlated visual tokens and semantically dense textual tokens as uniform 1D vectors, discarding structural cues.This spatial amnesia can limit adaptation for textures, shapes, and other fine-grained visual details.
  • HeBA: HeBA introduces separate visual and textual streams, using 2D depthwise-separable convolutions for spatial locality and dense linear bottlenecks for semantic integrity.The architecture encodes modality-specific priors directly into the adapter.
  • HeBA: HeBA compresses features from D to D/4, restricting capacity and forcing compact representations that filter task-irrelevant noise as structural regularization.This replaces the expanding adapter design with a compressive bottleneck.
  • HeBA: Active Kaiming initialization supplies early gradient flow, while dynamic scaling and label smoothing stabilize adaptation of the frozen backbone.The strategy is presented as an alternative to zero initialization, which can produce vanishing early gradients.
  • Results: 81.35% Harmonic Mean across 11 benchmarks establishes HeBA as a new state-of-the-art, alongside 78.62% Novel Accuracy.The reported results support the framework’s performance claim across the evaluation suite.

2. Related Work

Prior VLM adaptation methods include prompt learning, adapters, regularization, and cross-modal approaches, but commonly use homogeneous or parameter-heavy designs. HeBA instead aligns adapter structure with modality-specific inductive biases and compresses features for stability.

  • Prompt Learning: Prompt-learning methods optimize learnable text or multimodal tokens, but CoOp can overfit Base classes and perform worse on Novel classes.Other methods condition, distribute, or regularize prompts to improve generalization.
  • Prompt Learning: Regularization-based approaches preserve knowledge through prompt discrepancies, masked read-only tokens, or language-aware soft prompting with distinct visual-language losses.These methods target retention and multimodal alignment through different constraints.
  • Adapter Tuning: Adapter methods add lightweight residual modules or bottleneck MLPs, while Tip-Adapter uses a key-value cache for training-free few-shot adaptation.Other approaches incorporate auxiliary knowledge or cross-modal interactions through hierarchical descriptions and dual pathways.
  • Adapter Tuning: LwEIB reintroduces spatial inductive bias with depthwise convolutions but uses an inverse bottleneck that expands internal features to 4×, increasing parameter count and overfitting risk.Its parameter-heavy design also motivates heuristic optimization schedules.
  • HeBA: HeBA inverts this expansion by compressing features to D/4, making structural regularization intrinsic and supporting active initialization without severe divergence risk.The design separates spatial locality in vision from semantic globalism in text.
  • HeBA: HeBA uses depthwise-separable convolutions for visual spatial structure and linear projections for textual semantics, achieving superior efficiency over modality-agnostic or purely prompt-based approaches.The framework is presented as a unified architecture for frozen CLIP adaptation.

3. Methodology

HeBA adapts frozen CLIP with modality-specific processing, a compressive bottleneck, and active initialization. Visual tokens retain spatial structure, while text tokens use dense projections for semantic processing.

  • Heterogeneous Bottleneck Architecture: HeBA is a unified framework for adapting the frozen CLIP backbone through modality-specific processing and strict dimension compression.The framework departs from expansive spatial adapters by combining heterogeneous processing with a bottleneck.
  • Heterogeneous Bottleneck Architecture: The adapter projects features from D to D/4, acting as a structural regularizer that learns a low-rank representation of domain shift.This compressive bottleneck contrasts with LwEIB’s internal expansion to 4D.
  • Visual Stream: Spatial-Aware Convolution: Visual tokens are reshaped into a 2D grid and processed with specialized convolutions to model spatial locality in textures and shapes.The design addresses spatial correlations that are lost when visual tokens are treated as flat sequences.
  • Visual Stream: Spatial-Aware Convolution: The visual adapter uses 3 × 3 depthwise-separable convolutions, whereas the text adapter uses dense linear projections to preserve semantic integrity.The text pathway treats spatial locality as irrelevant and operates directly on the token sequence.
  • Active Gradient Initialization Paradigm: HeBA uses Kaiming initialization for up-projections to stimulate gradient flow from the first iteration while the CLIP backbone remains frozen.The authors contrast this with zero-initialized adapters, which they argue can prolong vanishing gradients and delay adaptation.
  • Optimization and Regularization: Dynamic scaling and Label Smoothing Cross-Entropy complement active initialization by modulating adapter influence and penalizing peaky predictions.The smoothing parameter is ε = 0.1.

4. Experiments

Experiments evaluate HeBA across diverse few-shot classification settings, including base-to-novel generalization, cross-dataset transfer, and domain generalization. The implementation uses frozen ViT-B/16 CLIP and updates only adapter parameters.

  • Base-to-Novel Generalization: The base-to-novel protocol trains on 16 shots per Base class and evaluates accuracy on both Base and Novel classes plus their Harmonic Mean.The benchmark covers 11 datasets spanning objects, fine-grained categories, scenes, textures, satellite imagery, and actions.
  • Cross-Dataset Evaluation: Cross-dataset evaluation trains on ImageNet with 16 shots per class and tests directly on 10 other datasets without fine-tuning.This setting assesses transferability to new datasets.
  • Domain Generalization: Domain generalization evaluates an ImageNet-trained model on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.These four variants are used to test robustness against distribution shifts.
  • Implementation: The experiments keep the image and text encoders frozen and update only HeBA adapter parameters.The implementation is based on the ViT-B/16 CLIP backbone.
  • Architecture: HeBA uses a heterogeneous adapter with 3 × 3 depthwise-separable convolutions for visual tokens, linear projections for text, and a D → D/4 bottleneck.The implementation uses modality-specific structures and a reduction ratio of r = 4.
  • Experimental Protocol: Results are averaged over three independent runs with different random seeds.The reported seeds are 1, 2, and 3.

5. Results

HeBA reports strong performance across base-to-novel, cross-dataset, structure-sensitive, and out-of-distribution evaluations. The results include the highest average Harmonic Mean in base-to-novel evaluation and competitive transfer robustness.

  • Base-to-Novel Generalization: 81.35% Harmonic Mean surpasses LwEIB’s 81.21% and MMA’s 79.87% in base-to-novel generalization.HeBA also achieves 78.62% Novel accuracy versus LwEIB’s 78.21%.
  • Structure-Sensitive Datasets: +2.37% Novel accuracy over LwEIB on DTD corresponds to 70.20% versus 67.83%.HeBA also achieves 88.16% Harmonic Mean on EuroSAT versus LwEIB’s 86.86%.
  • Cross-Dataset Evaluation: 68.71% average accuracy across 10 target datasets exceeds LwEIB’s 68.61% in cross-dataset evaluation.On EuroSAT, HeBA reaches 58.99%, compared with LwEIB’s 55.37% and HPT’s 47.36%.
  • Structure-Sensitive Datasets: The structure-sensitive comparison covers textures, satellite imagery, fine-grained flowers, and actions, whose domains require local spatial correlations.The figure compares HeBA with CoOp, MaPLe, PromptSRC, HPT, and LwEIB.
  • Domain Generalization: 60.26% average accuracy on four OOD ImageNet variants is comparable to MaPLe’s 60.27% and PromptSRC’s 60.65%.HeBA achieves its highest OOD result on ImageNet-A at 51.36%.

6. Ablation Study

The ablation study evaluates HeBA’s components, showing that active initialization, spatial structure, and depthwise convolution each contribute to performance. Inference-time adapter scaling also depends on the transfer setting.

  • Initialization Strategy: 84.29% Base accuracy is achieved by HeBA Full with active initialization, while maintaining comparable Novel performance.Zero-initialization reaches 84.11% Base accuracy and 78.63% Novel accuracy; HeBA Full improves Base accuracy after removing delayed convergence.
  • Spatial Inductive Biases: 81.25% HM without the 2D spatial bias and 81.20% HM without depthwise convolution, compared with 81.35% for the full design.The ablations support preserving 2D visual structure and using local spatial aggregation through the depthwise kernel.
  • Component Ablation: The full HeBA configuration provides the best trade-off between Base and Novel accuracy across the 11-dataset average.The cited ablation summary identifies the configuration using Kaiming initialization and spatial depthwise convolutions as the strongest trade-off.
  • Inference-Time Adapter Scaling: 68.71% average accuracy is obtained for cross-dataset transfer when the adapter uses the base scale at inference.Dampening the adapter allows frozen CLIP features to take precedence in distinct downstream tasks.
  • Inference-Time Adapter Scaling: 60.67% is achieved for domain generalization at the best reported inference scale, whereas reducing the scale hurts performance.For ImageNet variants with shared semantics, the reported best setting uses αnovel = 0.05.

7. Conclusion

The conclusion presents HeBA as a parameter-efficient VLM adaptation framework that matches modality-specific structure with heterogeneous adapters and bottleneck regularization. Across 11 benchmarks, it reports strong base-to-novel generalization, robustness, and transfer performance.

  • Conclusion: Keeping the adapter scale fixed is optimal for domain generalization, while cross-dataset transfer benefits from the reported alternative scaling behavior.The conclusion distinguishes the preferred scaling behavior across these evaluation settings.
  • Conclusion: HeBA addresses the modality gap by combining a bottleneck linear text adapter with depthwise-separable convolutional processing in the visual branch.The text branch preserves semantic integrity through compression, while the visual branch captures local geometric inductive biases.
  • Conclusion: HeBA achieves a new state-of-the-art 81.35% Harmonic Mean for Base-to-Novel generalization.The conclusion identifies this as the principal result of the proposed adapter framework.
  • Conclusion: Active Kaiming Initialization provides an alternative to zero-initialization by ensuring early gradient flow and balancing plasticity with stability.The conclusion links early gradient flow to feature adaptation from the first iteration.
  • Conclusion: HeBA demonstrates robustness in cross-dataset transfer and domain generalization, particularly on structure-sensitive EuroSAT and DTD tasks.The conclusion reports significant margins over existing methods in these settings.

Declaration of Interests

The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.

  • The authors declare no known competing financial interests or personal relationships relevant to the reported work.

CRediT authorship contribution statement

Md Jahidul Islam is credited with the paper’s conceptualization, methodology, software, validation, analysis, investigation, resources, data curation, writing, visualization, and project administration.

  • Md Jahidul Islam contributed to conceptualization, methodology, software, validation, formal analysis, investigation, resources, and data curation.
  • Md Jahidul Islam contributed to original drafting, review and editing, visualization, and project administration.
Loading 2603.16653v1…