Source-linked AI summary
GATE: Reliability-Gated Gaussian Evidence Fusion for Training-Free Test-Time Adaptation of Vision-Language Models
Pedram MohajerAnsari, Amir Salarpour, Run Wang, Mert D. Pesé
TL;DR
VLM zero-shot predictions can weaken under target-domain shifts, while existing adaptation methods often emphasize either prompts or image evidence. GATE uses frozen, training-free two-pass adaptation with complementary text and image Gaussians plus reliability-gated fusion. Across benchmark groups and backbones, it achieves the strongest average performance and improves zero-shot accuracy by 5.41 points on average.
Problem
Zero-shot VLM predictions can degrade under target-domain shifts, while existing adaptation methods often model language-side or image-side evidence separately.
Method
GATE is a training-free two-pass method that uses text and image Gaussians with class-wise reliability-gated score fusion while keeping the VLM encoders and prompts frozen.
Results
GATE achieves the best average accuracy in every benchmark/backbone group, improving zero-shot performance by an average of 5.41 percentage points and beating the strongest non-GATE baseline by 1.94 points.
Takeaways & Limitations
Language-derived semantic variation and target-image statistics provide complementary evidence for improving frozen VLM zero-shot classification.
Takeaways & Limitations
GATE is transductive, depends on pseudo-evidence quality and coverage, and may underperform prompt-specialized methods when errors are primarily semantic.
Abstract
from arXiv · showhide
Vision-language models such as CLIP and SigLIP provide strong zero-shot recognition, but their predictions can degrade when deployed on target data that differ from the pretraining distribution. Test-time adaptation offers a practical way to improve robustness without source data or target labels, yet existing methods often rely on either prompt-side adaptation or image-side target evidence alone. In this work, we introduce GATE, a training-free two-pass transductive test-time adaptation framework that uses the unlabeled target set while keeping the image encoder, text encoder, and prompt parameters fully frozen. Instead of representing each class with a single prototype, GATE builds two complementary Gaussian sources of evidence in the shared vision-language feature space: a text Gaussian estimated from multiple language descriptions and an image Gaussian estimated from reliable unlabeled target samples. A class-wise reliability gate controls the influence of image-derived pseudo-evidence, and a score-level generalized Product-of-Experts fusion produces a normalized residual correction to the original zero-shot logits. Across fine-grained recognition datasets, ImageNet-family distribution shifts, multiple CLIP backbones, and SigLIP-B/16, GATE achieves the best average accuracy in every benchmark/backbone group. It improves zero-shot performance by an average of 5.41 points and outperforms the strongest non-GATE baseline by 1.94 points, demonstrating the benefit of reliability-gated distributional evidence for frozen VLM adaptation.
1 Introduction
GATE addresses degraded zero-shot recognition under target-domain shifts by combining complementary language- and image-derived Gaussian evidence while keeping the VLM fully frozen. Across diverse benchmarks and backbones, it improves robustness without model updates.
- Motivation: Zero-shot VLM predictions can degrade when target samples differ from the pretraining distribution, motivating unlabeled test-time adaptation.Relevant shifts include domain, style, sensor, and corruption changes.
- Motivation: Existing methods mainly adapt either language-side prompts or image-side target evidence, rarely modeling both class-wise variability sources together.Language methods improve semantic coverage, whereas image methods capture target-domain regularities.
- GATE: GATE represents each class with text and image Gaussians in the shared vision-language space, using a reliability gate before score-level fusion.Text Gaussians capture prompt-induced semantic variability; image Gaussians summarize reliable unlabeled target structure.
- Results: GATE consistently improves zero-shot robustness across fine-grained datasets, ImageNet-family shifts, and multiple VLM backbones without model updates.The main comparisons cover multiple benchmark groups and backbones.
- GATE: GATE uses only unlabeled target data and leaves image encoders, text encoders, and prompt parameters unchanged during adaptation.The method is training-free and designed for frozen CLIP and SigLIP models.
Related Work
Prior VLM adaptation methods modify prompts, aggregate augmented views, correct distributions, or use caches and retrieval. GATE instead combines complementary text and target-image evidence with reliability control while keeping the VLM frozen.
- Existing approaches: Prompt-side methods optimize or reweight language representations, while image-side methods use augmented views, distribution correction, or target-sample caches.These approaches adapt different evidence sources during inference.
- Existing approaches: Some optimization-based methods learn representative target features from low-entropy samples and unlearn unreliable patterns from high-entropy samples.These methods differ from GATE because they update model components at test time.
- GATE's positioning: GATE models both sides of the VLM classifier as complementary class-wise evidence in the shared embedding space.Its text Gaussian captures semantic variability, while its image Gaussian summarizes target-domain visual structure.
- GATE's positioning: A class-wise reliability gate limits image-derived pseudo-evidence before fusion with original zero-shot logits, while all encoder and prompt parameters remain frozen.This combines semantic language priors with target-domain visual statistics without model updates.
3 Method
GATE adapts a frozen VLM in two transductive passes by combining text- and target-image Gaussian evidence while preserving the original zero-shot classifier. Reliability gating controls image-derived evidence, and the resulting normalized correction is added to zero-shot logits.
- GATE operates source-free and label-free, using only unlabeled target images while keeping image, text, and prompt parameters fixed.
- Each class uses complementary text and image Gaussians: language descriptions provide semantic structure, while reliable target pseudo-evidence captures visual structure.
- Text Gaussian Prior: The text Gaussian retains prompt-induced variation through a class mean and covariance rather than compressing descriptions into one prototype.
- Image Gaussian: In Pass 1, sparse top-5 responsibilities select at most B = 20 target samples per class for weighted image means and pooled covariance estimation.
- Reliability Gate: Class-wise reliability uses selected-evidence amount and average confidence to down-weight weak or ambiguous image evidence.
- Evidence Fusion: GATE converts both Gaussians into regularized second-order compatibility scores, fuses them with a reliability-gated score-level generalized Product-of-Experts, and adds a normalized bounded residual correction.
4 Experiments
GATE is evaluated under source-free, label-free test-time adaptation across diverse datasets, distribution shifts, and multiple frozen VLM backbones. It achieves the strongest average performance across benchmark groups while remaining competitive rather than universally best on every dataset.
- Experimental setup: GATE is evaluated on fine-grained, domain-specific, and ImageNet-family benchmarks across CLIP ViT-B/32, ViT-B/16, ViT-L/14, and SigLIP-B/16.The image encoder, text encoder, and prompt parameters remain fixed throughout adaptation.
- Experimental setup: All compared methods use source-free, label-free evaluation, while GATE estimates evidence in a two-pass transductive protocol using the full unlabeled target split.Pass 1 selects pseudo-evidence and estimates Gaussian statistics; Pass 2 predicts with the resulting evidence.
- Main results: 5.41 percentage points: GATE’s average improvement over zero-shot across seven benchmark/backbone groups.It also outperforms the strongest non-GATE baseline by an average of 1.94 points, achieves the best accuracy in 39 of 52 settings, and ranks among the top two in 49.
- Fine-grained and diverse recognition: 77.80%: GATE’s fine-grained mean accuracy on SigLIP-B/16, exceeding CARPRT by 2.40 points.On CLIP ViT-B/16, mean accuracy rises from 67.35% for the strongest baseline to 69.25%.
- Fine-grained and diverse recognition: GATE performs especially well on EuroSAT, OxfordPets, SUN397, and UCF101, but specialized baselines remain stronger on some datasets such as DTD and Food101.Thus, GATE has the strongest average performance rather than the best result on every individual dataset.
5 Ablation Studies
The ablations show that GATE is broadly stable, benefits from complementary text and image Gaussian evidence, and remains useful under degraded pseudo-evidence, although performance still depends on evidence quality and coverage.
- Evidence budget: Up to 20 pseudo-evidence samples per class improve accuracy, whereas retaining lower-confidence samples slightly reduces performance.The fixed main-experiment budget is therefore up to 20 samples per class.
- Component ablations: 65.30% accuracy is achieved by full GATE, exceeding zero-shot inference at 61.24% through complementary text and image Gaussian evidence plus reliability gating.Text Gaussian evidence raises accuracy to 62.63%, while ungated image Gaussian evidence reaches 64.05%.
- Parameter sensitivity: Accuracy changes by only 0.16 points across image-covariance shrinkage values and 0.15 points across evidence-count and confidence exponents.These diagnostic sweeps show broad performance plateaus, and fixed settings are reused across the benchmark.
- Gaussian diagnostics: Gaussian compatibility diagnostics place 92–96% of samples within the theoretical 95% envelope, with remaining deviations concentrated mainly in the upper tail.Henze–Zirkler statistics range from 0.02–0.04, and reliability gating mitigates the remaining deviations.
- Pseudo-evidence robustness: 67.70% accuracy under 70% evidence starvation remains above zero-shot inference at 65.64%, while 40% corrupted pseudo-evidence lowers GATE to 66.14%.The reliability gate and bounded residual correction mitigate, but do not eliminate, dependence on pseudo-evidence quality and coverage.
- Limitations: GATE is transductive and may underperform prompt-specialized methods when the dominant error is semantic rather than visual.Its performance still depends on pseudo-evidence quality and coverage, despite reliability gating and bounded residual correction.
6 Conclusion
GATE is a training-free, two-pass transductive method that fuses text and target-image Gaussian evidence through class-wise reliability gating without updating the VLM. Across fine-grained and ImageNet-family shifts with CLIP and SigLIP backbones, it consistently improves zero-shot classification and achieves the strongest average performance.
- Conclusion: GATE fuses text and target-image Gaussian evidence through class-wise reliability gating without updating the VLM.The text Gaussian supplies a semantic prior, while the image Gaussian summarizes target-domain structure from reliable unlabeled samples.
- Conclusion: Language-derived semantic variation and image-derived target statistics provide complementary evidence for zero-shot classification.The conclusion frames these sources as complementary rather than replacing the original VLM classifier.
- Conclusion: Across fine-grained and ImageNet-family shifts with CLIP and SigLIP backbones, GATE consistently improves zero-shot classification and achieves the strongest average performance.