LAS-CLIP: A Lightweight Adapter Steering Approach for CLIP's Visual Encoder

• CLIP’s global image representation limits region-level tasks, while visual prompting, input masking, and encoder modification can alter pixels, destroy context, or change pretrained weights.Source: §1 · pp. 1–2
• MaskAdapter generates per-head, per-layer attention biases from a binary mask and injects them into frozen self-attention without modifying pretrained weights.With no mask, LAS-CLIP reverts to vanilla CLIP behavior.Source: §1 · p. 2
• Gated token-mask interaction conditions attention biases on the spatial mask and current visual tokens, selectively modulating heads to preserve diverse feature representations.Source: §1 · p. 2
• LAS-CLIP achieves competitive or superior performance to Alpha-CLIP on zero-shot classification and referring expression comprehension, using 100K training samples and two T4 GPUs.Source: §1 · p. 2
• Qualitative analysis of attention visualization and subject-driven image generation finds stronger representational fidelity with the frozen backbone.Source: §1 · p. 2

2.1 Vision-Language Contrastive Pre-training

• CLIP and its successors produce global image representations without the ability to attend to user-specified regions.Source: §2.1 · p. 3

2.2 Region-Aware CLIP Adaptation

• Visual prompting methods alter pixel content and introduce a distribution gap, while FALIP uses uniform attention bias across heads, limiting fine-grained steering.Source: §2.2 · p. 3
• Input masking can destroy contextual information, whereas ReCLIP blurs backgrounds and RegionCLIP pre-trains region-text alignment for detection.Source: §2.2 · p. 3
• Alpha-CLIP adds an alpha channel and fine-tunes on millions of RGBA region-text pairs, permanently altering pretrained weights and requiring substantial resources.Source: §2.2 · p. 3

2.3 Parameter-Efficient Fine-Tuning

• Existing parameter-efficient approaches target mainly classification and class-agnostic representations; none natively ingests binary masks to steer region-specific attention at individual-head level.Source: §2.3 · pp. 3–4

3.1 Preliminary

• CLIP produces normalized image and text embeddings, and LAS-CLIP steers attention at the ViT self-attention site; Alpha-CLIP instead adds an alpha channel and fine-tunes the encoder.CLIP’s final image representation comes from the last-layer [CLS] token.Source: §3.1 · pp. 4–5

3.2 Overview

• LAS-CLIP injects per-head attention biases from a binary region mask into the last K frozen CLIP vision layers, reverting to vanilla CLIP when no mask is supplied.The adapter is optimized while the original weights remain untouched.Source: §3.2 · pp. 5–6

3.3 Gated Token-Mask Interaction

• The MaskAdapter combines patch-level mask features with layer-conditioned visual-token features; sigmoid gates produce per-head biases while limiting mask override of global-context heads.Mask features are generated by a convolutional branch, and the layer embedding lets different depths produce distinct steering signals.Source: §3.3 · pp. 6–7

3.4 Training Objective

• Training combines symmetric CLIP contrastive loss with identity regularization toward vanilla features, plus full-image and caption augmentation to retain holistic understanding.The identity term controls how far adapted features drift from the original representation.Source: §3.4 · pp. 7–8

3.5 Data Preparation

• GRIT boxes and referring expressions are paired with YOLOE masks; predictions below 0.30 IoU fall back to box regions, and filtering retains 100K samples.Filtering excludes masks with area ratios outside [0.01, 0.95] and expressions shorter than three characters.Source: §3.5 · pp. 8–9

4 Experiments / 4.1 Experimental Setup

• Evaluation covers ImageNet-S zero-shot classification, zero-shot REC on RefCOCO, RefCOCO+, and RefCOCOg, and BLIP-Diffusion generation; the MaskAdapter is inserted into the last K = 3 vision-transformer layers unless stated otherwise.REC uses MAttNet object proposals and selects the proposal with highest similarity to the referring expression.Source: §4 · p. 9, §4.1 · p. 9

4.2 Zero-shot Image Classification / 4.3 Zero-shot Referring Expression Comprehension

• LAS-CLIP surpasses FALIP and Alpha-CLIP on ViT-B/16 and ViT-L/14 Top-1 ImageNet-S results, and leads on RefCOCO and RefCOCO+ and RefCOCOg val, while Alpha-CLIP slightly leads on RefCOCOg test.The ImageNet-S table reports mean per-class accuracy; the REC table reports accuracy and uses an ensemble of ViT-B/16 and ViT-L/14 for LAS-CLIP and Alpha-CLIP.Source: §4.2 · pp. 9–10, Table 1 · p. 10, §4.3 · p. 10

4.4 Ablation Study

• Using fewer adapter layers preserves classification accuracy, while grounding improves with depth up to K = 3, the adopted default.The ablation varies K and reports ImageNet-S per-backbone results and ensemble RefCOCO accuracy.Source: §4.4 · p. 10, Table 3 · p. 11
• Removing identity regularization at K = 4 lowers classification accuracy but improves grounding, making its weight a control for balancing global semantics and spatial grounding.The comparison is reported for per-backbone ImageNet-S accuracy and ensemble RefCOCO accuracy.Source: §4.4 · pp. 10–11, Table 4 · p. 12
• The full adapter architecture achieves the best grounding accuracy across all ViT-B/16 ablation splits, compared with CLS-only, head-agnostic, and no-token variants.Supplementary experiments report graceful degradation under imperfect masks, gains attributable to the adapter design rather than the data pipeline, and latency comparable to Alpha-CLIP but higher GPU memory use.Source: §4.4 · pp. 11–12, Table 5 · p. 12

4.5 Qualitative Analysis

• With incorrect masks, LAS-CLIP attention remains closer to text-relevant regions than Alpha-CLIP, whose attention stays strongly tied to the mismatched mask.The heatmaps use text-based decomposition and compare correct-mask and incorrect-mask conditions.Source: §4.5 · p. 12, Figure 2 · p. 13
• In BLIP-Diffusion subject-driven generation, LAS-CLIP outputs better preserve input-subject identity than baseline CLIP or Alpha-CLIP.The comparison uses ViT-L/14; prompts are blank so generation relies on visual inputs.Source: §4.5 · p. 13

5 Conclusion

• LAS-CLIP injects per-head, per-layer attention biases through a lightweight MaskAdapter without modifying pretrained weights, and reverts to vanilla CLIP when no mask is provided.Experiments cover ImageNet-S, RefCOCO, RefCOCO+, and RefCOCOg; the paper reports competitive or superior performance to full encoder fine-tuning methods with a fraction of their trainable parameters and training data.Source: §5 · pp. 13–14
• The frozen backbone retains stronger representational fidelity for downstream tasks, according to the paper’s qualitative analyses.The conclusion’s example figure compares subject-driven generation using baseline CLIP, Alpha-CLIP, and LAS-CLIP.Source: §5 · p. 14, Figure 3 · p. 14
• The approach relies on pre-computed segmentation masks and has a higher GPU memory footprint from dense per-head attention bias matrices.Source: §5 · p. 14

A Implementation Details

• The MaskAdapter has 116,004 trainable parameters with ViT-B/16 and 145,200 with ViT-L/14, using hidden dimension 64 and dropout 0.2.Both configurations initialize from publicly available OpenAI CLIP weights.Source: A Implementation Details · p. 15
• Training uses AdamW for five epochs, a OneCycleLR schedule, 10% linear warmup, and an effective batch size of 2048.Adapter learning rate is 2 × 10^-3; the logit-scale learning rate is 1 × 10^-5.Source: A Implementation Details · p. 15
• About 2 hours for ViT-B/16 and 6 hours for ViT-L/14 training on two NVIDIA T4 GPUs.The authors attribute the low overhead to the compact adapter and small number of trainable parameters.Source: A Implementation Details · p. 15
• For referring-expression evaluation, proposal scores sum original- and blurred-image similarities, and the highest-scoring proposal is selected.The blurred input uses Gaussian blur with σ = 100; the reported backbone ensemble sums scores from ViT-B/16 and ViT-L/14.Source: A Implementation Details · p. 16

B Additional Qualitative Results

• With incorrect masks, LAS-CLIP maintains query-relevant attention in the “Man in USA T-shirt” and “Book on left” examples, unlike Alpha-CLIP’s attention shift to the wrong region.Under correct masks, all region-aware methods concentrate attention on the target object; the wrong-mask improvement varies across samples.Source: B Additional Qualitative Results · p. 16

C Additional Subject-Driven Generation Results

• LAS-CLIP preserves more branching structure in a neuron illustration and retains both the spider and web pattern in another example.Alpha-CLIP introduces text-like artifacts in the spider-web example, while both it and the baseline show less structural similarity in the neuron example.Source: C Additional Subject-Driven Generation… · p. 16

D Mask Robustness Analysis

• The analysis perturbs ImageNet-S ground-truth masks through erosion, dilation, and spatial shifts at approximate IoU levels 0.90, 0.70, 0.50, and 0.30.All experiments use ViT-L/14.Source: D Mask Robustness Analysis · p. 17
• At IoU ≈0.30, LAS-CLIP’s accuracy-drop advantage over Alpha-CLIP reaches 2.0% for dilation and 1.7% for spatial shifts.Under erosion, both methods degrade comparably.Source: D Mask Robustness Analysis · p. 17
• The authors interpret the smaller drops under dilation and shifts as evidence that token-conditioned gating mitigates overreliance on the spatial-mask prior.This interpretation is stated as consistent with the main paper’s qualitative attention analysis.Source: D Mask Robustness Analysis · p. 17

• The ablation retrains LAS-CLIP with SAM-generated masks on the same 100K training samples used by the default YOLOE-based pipeline.Source: E Training Data Source Ablation · p. 17
• Across both backbones, SAM-trained LAS-CLIP outperforms YOLOE-trained LAS-CLIP and Alpha-CLIP on ImageNet-S classification.Source: Table S1 · p. 18, E Training Data Source Ablation · p. 18
• Under dilation and spatial-shift perturbations on ImageNet-S, LAS-CLIP degrades more gracefully than Alpha-CLIP.Source: Figure S3 · p. 19
• SAM- and YOLOE-trained variants perform comparably on RefCOCO, with YOLOE slightly ahead on most splits despite its noisier masks.Source: E Training Data Source Ablation · p. 18, Table S2 · p. 19

• The higher memory footprint comes from dense N × N attention-bias matrices; reducing adapter layers K or hidden dimension h can mitigate it.Source: F Inference Efficiency · p. 19
• 0.3% GFLOPs overhead over vanilla CLIP accompanies throughput and latency comparable to Alpha-CLIP across batch sizes on a single T4 GPU.Source: F Inference Efficiency · p. 19, Table S3 · p. 20
• At batch size 64, LAS-CLIP uses approximately 2.6× vanilla CLIP's memory, compared with 1.8× for Alpha-CLIP.Source: F Inference Efficiency · p. 19, Table S3 · p. 20