Source-linked AI summary
Towards Purified Multi-Label Test-Time Adaptation of Vision-Language Models
Yiwen Liang, Hui Chen, Yizhe Xiong, Mengyao Lyu, Yuhan Cao, Zijia Lin, Shuaicheng Niu, Sicheng Zhao, Jungong Han, Guiguang Ding
TL;DR
Multi-label test-time adaptation remains underexplored because distribution shifts make global representations entangle co-occurring objects and make regional evidence unreliable. PuRF purifies regions and caches, combining episodic prototypes with temporal refreshing, and achieves state-of-the-art performance across five benchmarks. Its scope includes assumptions such as fully unlabeled adaptation and the limitations of entropy reweighting designed for single-label consistency.
Problem
Multi-label test-time adaptation remains underexplored because distribution shifts complicate reliable use of global and regional evidence for images containing multiple objects.
Method
PuRF performs region purification for reliable fine-grained alignment, then cache purification with episodic class-specific prototypes and temporal refreshing.
Results
PuRF consistently outperforms existing methods across five benchmarks and establishes state-of-the-art results.
Takeaways & Limitations
PuRF provides a strong baseline for future multi-label test-time adaptation research.
Takeaways & Limitations
The approach targets fully unlabeled online adaptation, and it does not adopt entropy reweighting because that strategy is tailored for single-label consistency.
Abstract
from arXiv · showhide
Test-time adaptation (TTA) has been widely explored in single-label recognition, effectively mitigating distribution shifts, especially when combined with vision-language models. However, real-world images often contain multiple objects, while the more practical multi-label test-time adaptation (MLTTA) has received little attention so far. Recent cache-based TTA methods have shown promising efficiency and effectiveness, yet directly extending them to multi-label scenarios suffers from a one-to-many mapping problem: a shared global representation entangling co-occurring objects is stored as class-wise cache prototypes, inducing dominant-label bias and compromised cache calibration. While introducing region-level cues helps isolate class-specific evidence, such regional evidence can also be unreliable under distribution shifts, making its identification and utilization non-trivial. To address these issues, we introduce PuRF, a novel PuRiFication-driven cache-based method for multi-label test-time adaptation of vision-language models. Specifically, PuRF first performs region purification to identify reliable regions, providing comprehensive regional cues for multi-label recognition and enabling fine-grained alignment. Based on these purified regions, PuRF conducts cache purification to enhance cache representation and adaptability, where episodic purification builds a discriminative region-based cache, and temporal refreshing further promotes long-term cache adaptability. Experiments demonstrate that PuRF consistently outperforms state-of-the-art methods, achieving a notable 4.05% mAP improvement on ViT-B/32 across five datasets.
1 Introduction
Vision-language models struggle under distribution shift, while multi-label test-time adaptation remains challenging because existing approaches inadequately handle co-occurring objects and unreliable regional evidence. PuRF addresses this gap through region and cache purification, with experiments showing consistent gains across datasets and backbones.
- Limitations of Existing Methods: Existing cache-based methods can miscalibrate class-wise prototypes because shared global representations entangle objects and are dominated by salient labels.ML-TTA instead uses prompt optimization but relies on intensive text retrieval and episodic prompt resetting, limiting historical-information reuse.
- Motivation: Vision-language models often fail to generalize when testing distributions differ from pre-training distributions.Such shifts arise from different environments or sensing conditions.
- Motivation: Most test-time adaptation methods target single-label recognition, although real-world images commonly contain multiple co-occurring objects.This makes multi-label adaptation more challenging and complex than single-label adaptation.
- Proposed Method: PuRF combines Region Purification and Cache Purification to exploit reliable regional evidence for fine-grained recognition and more discriminative cache calibration.Its cache purification includes episodic construction of class-specific prototypes and temporal refreshing for long-term adaptability.
- Results: PuRF consistently outperforms the strongest state-of-the-art method across five benchmarks and four backbones.The supplied passages establish the cross-dataset and cross-backbone superiority but do not provide the associated numeric result.
2 Related Work
Prior vision-language test-time adaptation methods mainly use prompts or caches, while multi-label recognition requires handling label dependencies and discriminative label-specific features. The fully unlabeled online setting remains challenging, motivating methods that can adapt precisely to each incoming multi-label sample.
- Vision-Language Test-Time Adaptation: Vision-language test-time adaptation methods are broadly divided into prompt-based and cache-based approaches.Prompt methods optimize textual prompts, whereas cache methods retrieve historical test features for prediction refinement.
- PuRF Framework: PuRF’s framework purifies regions before deriving pseudo supervision and constructing class-specific region-aware cache prototypes.Temporal refreshing then maintains cache adaptability as new samples arrive.
- Vision-Language Test-Time Adaptation: Prompt-based adaptation can be inefficient because it resets prompts per sample and requires heavy backpropagation.Cache-based methods provide a more efficient alternative by dynamically storing test features and aggregating retrieval similarities.
- Multi-Label Recognition with VLMs: Multi-label recognition must model label dependencies and identify discriminative properties for each label.Earlier approaches use graph structures, recurrent networks, or attention mechanisms to capture these aspects.
- Multi-Label Recognition with VLMs: Fully unlabeled online adaptation for each incoming sample remains challenging for precise multi-label recognition.This setting differs from methods that rely on labeled or offline adaptation assumptions.
3 Methodology
PuRF adapts vision-language models for multi-label test-time settings by purifying regional evidence and cache representations. Its pipeline combines region purification, fine-grained alignment, episodic cache construction, and temporal refreshing to improve cache calibration and adaptability.
- Region Purification: PuRF selects informative regions through multi-granularity consistency, using augmented global views and local region features for purified class-specific evidence.Global candidate classes are obtained from the intersection of top-confidence predictions across augmented views, while purified regions support semantic alignment.
- Preliminaries: Cache-based TTA stores class-wise feature entries, replaces high-entropy items at capacity, and combines CLIP logits with retrieved cache logits.Cached prototypes are formed by averaging features within each class-wise cache.
- Motivation: PuRF addresses the limitation of global features, which entangle co-occurring objects and bias class-wise cache prototypes toward salient labels.This entanglement reduces label purity and degrades class-specific cache calibration in multi-label settings.
- Cache Purification: Cache purification operates on two timescales: episodic purification builds reliable region-aware entries, while temporal refreshing updates outdated entries during streaming adaptation.Temporal decay penalizes long-retained entries to reduce early-sample dominance and mitigate cache saturation.
- Overall Objectives: PuRF aligns cached visual prototypes with text using an entropy objective over confident views and a cross-modal alignment loss.The alignment loss encourages consistency between cached visual prototypes and textual prototypes.
4 Experiments
Experiments across five multi-label datasets evaluate PuRF under varied visual backbones, prompt initializations, efficiency settings, and component configurations. PuRF consistently achieves strong performance, while purification modules improve regional evidence, cache calibration, and temporal adaptability.
- Different CLIP Visual Backbones: PuRF achieves a 6.09% mAP gain over ReTA on COCO2014 and consistent improvements on NUS-WIDE across backbones.
- Different Prompt Initialization: PuRF reaches 73.79% mAP with MaPLe initialization, exceeding ReTA at 70.88% and MLTTA at 70.38%.
- Efficiency Analysis: PuRF combines lightweight text residual learning with tolerable region-level computation, achieving an 8.87% mAP gain with low memory cost.
- Generalization across Diverse VLM Backbones: Across diverse VLM backbones, PuRF improves average mAP by 3.56% over DPE and 3.69% over ReTA.
- Effectiveness of the Key Components: Region Purification improves performance by 2.24% on average, while Episodic Purification and Temporal Refreshing improve cache purity, calibration, and long-term adaptability.
- Impact of Region and Cache Purification: Purified-region caching constructs class-specific prototypes, and max aggregation preserves confident regional activations without diluting dominant class responses.
- Hyperparameter Sensitivity Analysis: Smaller κg suppresses noisy candidate classes, δ=500-1000 balances refreshing timing, and λ1, λ2 control loss-term trade-offs.
5 Conclusion
The paper addresses the largely unexplored problem of multi-label test-time adaptation with PuRF, a purification-driven cache-based method. Experiments on five benchmarks report consistent superiority and state-of-the-art performance, positioning PuRF as a strong baseline for future research.
- PuRF purifies regions and caches to exploit reliable evidence, improve recognition accuracy, and strengthen discriminative class-wise cache calibration.
- Experiments on five benchmarks verify PuRF's consistent superiority over existing methods and establish it as a strong baseline for future multi-label adaptation research.