Source-linked AI summary
Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation
Chandler Timm C. Doloriel, Yunbei Zhang, Sarthak Kumar Maharana, Muhammad Salman Siddiqui, Tor Kristian Stevik, Fadi Al Machot, Kristian Hovde Liland, Habib Ullah
TL;DR
Continual test-time adaptation destabilizes OVSS through class collapse, vision-language alignment erosion, and redundant updates from low-shift samples. DAF combines diversity, source anchoring, and salience filtering; across evaluated shifts, it remains stable where entropy minimization collapses and is robust to aggressive adaptation and learning rates.
Problem
Continual test-time distribution shifts can destabilize OVSS because entropy updates collapse patch-level classes, erode vision-language alignment, and waste computation on low-shift samples.
Method
DAF augments entropy-based adaptation with marginal diversity and cross-modal anchor consistency losses, plus source-anchored feature salience filtering for low-value samples.
Results
Across evaluated continual shifts, DAF remains stable where entropy minimization collapses and is robust to aggressive adaptation and learning rate choices.
Takeaways & Limitations
Source-anchored stabilization can preserve OVSS performance across continual shifts while filtering reduces part of the computational cost of anchoring.
Takeaways & Limitations
Under severe vocabulary mismatch, MDIV may resist beneficial adaptation and CMAC may become overly conservative, producing weaker gains under larger domain gaps.
Abstract
from arXiv · showhide
Open-vocabulary semantic segmentation (OVSS) relies on vision-language alignment to recognize arbitrary text-defined categories, yet this alignment is fragile under continual test-time distribution shift. Our diagnostic analysis reveals that entropy minimization drives patch-level class collapse, continual updates erode vision-language alignment, and redundant gradients from low-shift samples waste computation. We propose Diversify, Anchor, and Filter (DAF), a stabilization framework that augments entropy-based adaptation with a marginal diversity loss that resists collapse, a cross-modal anchor consistency loss that constrains feature drift relative to a frozen source model, and feature salience filtering that skips low-value backward passes to offset part of the source-anchor overhead. We evaluate on five datasets spanning natural scenes, autonomous driving, underwater imagery, and remote sensing with their corrupted variants. Across the evaluated continual shifts, DAF remains stable where entropy minimization collapses, improving mIoU by over 8 points on Pascal VOC20-C, over 9 points on LoveDA, and over 3 points on Foggy Cityscapes compared to the source model, and is robust to aggressive adaptation and learning rate choices.
1. Introduction
Continual test-time adaptation is especially fragile for OVSS because evolving domains can erode the patch-to-text alignment required for open-vocabulary segmentation. The paper diagnoses class collapse, cross-modal drift, and redundant updates, then proposes DAF to stabilize adaptation while preserving efficiency.
- OVSS must adapt online to unlabeled samples across changing weather, sensors, corruptions, and underwater domain shifts when labeled target data or offline retraining are impractical.
- CTTA carries model state across domains without past data, making entropy-based updates risky because they can erode patch-to-text alignment.
- Entropy minimization shrinks the active class vocabulary from near 20 to roughly 1 to 3 by step 500, causing large regions to collapse to a few concepts.
- Source-assigned prototype similarity decreases (r = −0.737), unassigned-prototype similarity increases (r = +0.854), and feature shift correlates with prediction change (r = +0.833).
- DAF combines MDIV to resist class collapse, CMAC to constrain feature drift, and SAFS to skip low-value backward passes.
- Across evaluated continual shifts, DAF remains stable where entropy minimization collapses and is robust to aggressive adaptation and learning-rate choices.
2. Related Work
Prior work covers test-time adaptation, segmentation under distribution shift, and open-vocabulary dense prediction, but existing segmentation methods generally assume fixed label spaces. MLMP extends episodic TTA to OVSS, leaving continual open-vocabulary adaptation as the unresolved setting.
- CTTA methods use teacher averaging, self-learning, masked self-supervision, prompt memories, constrained online learning, and related stabilizers for non-stationary streams.
- Sample-selection methods study which incoming samples to trust through entropy filtering and object-sensitive selection.
- Segmentation adaptation methods introduce structural priors, active feedback, visual prompts, and distribution-sensitive parameter adaptation, but assume fixed label spaces and task-specific components.
- Open-vocabulary dense prediction builds on CLIP and includes mask, region-text, distillation, attention, patch-interaction, and class-purification approaches; MLMP brought episodic TTA to OVSS.
3. Methodology
DAF stabilizes continual OVSS adaptation by addressing class collapse, vision-language feature drift, and redundant backward passes with complementary diversity, anchoring, and filtering mechanisms.
- DAF framework: DAF combines MDIV, CMAC, and SAFS to target patch-level class collapse, cross-modal alignment erosion, and redundant gradients from low-shift samples.MDIV regularizes aggregate predictions, CMAC anchors feature geometry to the frozen source model, and SAFS skips low-value updates.
- Marginal diversity loss: MDIV maximizes batch-level marginal class entropy across prompt templates, discouraging aggregate prediction concentration without enforcing uniform class usage in each image.It differs from patch-level entropy minimization because its logarithm operates after aggregation rather than independently per patch.
- Cross-modal anchor consistency: CMAC uses frozen-source assignments to penalize adapted features drifting away from assigned prototypes or toward unassigned prototypes.Its one-sided hinge terms activate only when adapted similarities worsen relative to source similarities.
- Source-anchored feature salience filtering: SAFS measures average angular feature shift between adapted and source patch features, retaining samples above an adaptive threshold for loss computation and updates.If every sample falls below the threshold, filtering is disabled to avoid an empty batch.
- Integration: The integrated adaptation objective adds MDIV and CMAC regularization to the selected baseline loss, while SAFS restricts all loss summations to retained samples.The combined objective is L = Lbase+λdiv Ldiv+λcmac Lcmac.
4. Experiments
DAF is evaluated across diverse datasets and continual shifts, including corrupted, shuffled, recurring, and aggressive-learning-rate settings. Results show that DAF stabilizes OVSS where entropy-based adaptation collapses, while its components preserve accuracy and improve adaptation efficiency.
- Evaluation setup: The evaluation spans five segmentation datasets across natural scenes, autonomous driving, underwater imagery, remote sensing, and their ImageNet-C-style corrupted variants.Corrupted variants use 15 corruption types at severity level 5.
- Long-horizon stability: Over shuffled multi-pass CTTA, TENT and MLMP progressively degrade, while DAF stays above source on Cityscapes-C and DAF-M gains over 10 points above source on LoveDA by the final pass.The model state is carried across passes without resetting.
- Learning-rate robustness: DAF-T and DAF-M exceed source performance even at a learning rate of 1×10^-2, where TENT and MLMP collapse to near-zero.Entropy-based methods remain stable only at 1×10^-4 and below in this sweep.
- Long-horizon stability: DAF-T and DAF-M remain stable across five recurring passes, with DAF-M reaching a mean of 32.3 on LoveDA without catastrophic degradation.This extends the stability pattern from domain-shuffled CTTA to a fixed domain order.
- Component analysis: MDIV and CMAC are complementary core mechanisms, while SAFS reduces backward passes by approximately 30% and lowers DAF-M adaptation time from 68.4 s to 50.8 s.The efficiency gain costs less than one mIoU point: 62.7 versus 63.4.
- Qualitative analysis: Qualitative predictions show TENT and MLMP collapsing toward one class, while DAF-T and DAF-M preserve multi-class structure under defocus blur.The reported mechanism links MDIV to active-vocabulary preservation and CMAC to patch-to-text alignment.
- Consistency analysis: CMAC reaches 62.7 mIoU on VOC20-C versus 51.7 for Lcosine and 40.8 for LL2, supporting text-anchored directionality over feature-only drift penalties.On Cityscapes-C, CMAC leads at 18.5 versus 15.2 for both alternatives.
5. Conclusion
DAF stabilizes continual test-time adaptation for open-vocabulary semantic segmentation by addressing class collapse, alignment erosion, and redundant gradients. Its benefits depend on the source vocabulary and source-relative semantic geometry remaining informative under target shift.
- DAF combines MDIV, CMAC, and SAFS filtering to address class collapse, cross-modal drift, and redundant gradients during continual adaptation.MDIV provides marginal diversity, CMAC anchors cross-modal consistency, and SAFS filters low-value updates.
- Across evaluated continual shifts, DAF remains stable where entropy minimization collapses and sustains performance under long-horizon multi-pass adaptation.The framework is also reported to be robust to aggressive learning rates.
- DAF assumes that the source vocabulary remains appropriate and source-relative semantic geometry remains informative under target-domain shift.Violating these assumptions, such as through severe vocabulary mismatch, may make MDIV resist beneficial adaptation and CMAC become overly conservative.
- Weaker gains under larger domain gaps indicate the practical boundary of DAF's source-anchored stabilization.