Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Models

Weiwei Xiang; Shun Peng; Guangyi Xiao; Hao Chen; Lei Yang

Source-linked AI summary

Dual-Stream Semantic Guidance with Prototype Anchor Calibration for Source-Fully-Free Adaptation of Vision-Language Models

Weiwei Xiang, Shun Peng, Guangyi Xiao, Hao Chen, Lei Yang

arXiv:2608.28145v1Aug 28, 2026cs.CV

TL;DR

SFF-DA must adapt VLMs without source data while controlling static drift from fixed class embeddings and dynamic drift from unconstrained captions. DSSG combines caption and class-anchor guidance with Dynamic CMKD, and DSSG-PAC periodically calibrates cached prototypes; DSSG achieves state-of-the-art performance across benchmarks, while DSSG-PAC largely preserves accuracy with 18.9% lower total adaptation time.

Problem
SFF-DA lacks source data and task-specific source models, while fixed class embeddings and unconstrained captions create complementary semantic drift that complicates adaptation.
Method
DSSG combines instance-specific captions, dynamically updated class anchors, and Dynamic CMKD; DSSG-PAC periodically calibrates and caches prototype anchors.
Results
DSSG achieves new state-of-the-art performance across multiple benchmarks, while DSSG-PAC largely preserves adaptation accuracy with 18.9% lower total adaptation time.
Takeaways & Limitations
Dual-stream guidance reconciles fine-grained target adaptation with global categorical consistency, while prototype calibration reduces redundant text-side computation.
Takeaways & Limitations
DSSG remains dependent.

Abstract

from arXiv · show

Source-Fully-Free Domain Adaptation (SFF-DA) has emerged as a strategic paradigm to adapt Vision-Language Models (VLMs) without any access to source data or task-specific source models. However, we identify a critical Dual Semantic Drift that hinders this process: static drift arising from the rigidity of fixed class embeddings, and dynamic drift stemming from the divergence of generated captions, causing severe semantic misalignment that intensifies the stability-plasticity dilemma. To address this, we propose DSSG (Dual-Stream Semantic Guidance), an end-to-end framework that reconciles fine-grained plasticity with global stability. Our core contribution is the Dual Semantic Guidance (DSG) module, which integrates a caption stream for domain-specific knowledge with a class-anchor stream to anchor global categorical consistency. Furthermore, a Dynamic Cross-Modal Knowledge Distillation (CMKD) module is introduced to leverage the evolving teacher distribution for calibrating teacher-student consistency. Building upon DSSG, we further introduce Prototype Anchor Calibration (PAC), yielding DSSG-PAC, which periodically calibrates prototype anchors and caches them until the next calibration. This design reduces redundant text-side computation while preserving the adaptability of class guidance to the evolving text space. We further establish SFF-DA risk bounds that relate student risk to semantic-teacher quality and teacher--student discrepancy. Extensive experiments demonstrate that DSSG consistently outperforms current state-of-the-art methods across multiple benchmarks, while DSSG-PAC largely preserves its adaptation performance with 18.9% lower total adaptation time. The code is available at https://github.com/mrmenand/DSSG.

I. INTRODUCTION

SFF-DA adapts vision-language models to unlabeled target domains without source data or task-specific source models, but must balance target-specific adaptability against preserving pretrained knowledge. DSSG addresses this tension by combining caption and class-anchor guidance with dynamic distillation, while DSSG-PAC reduces text-side overhead through periodic prototype calibration.

SFF-DA adapts pretrained VLMs directly on unlabeled target domains without source data or task-specific source models.
Direct fine-tuning can accumulate pseudo-label errors, causing catastrophic forgetting and category collapse, while fixed text embeddings create evolutionary asymmetry and static semantic drift.
Prompt tuning offers limited plasticity, whereas encoder tuning risks damaging pretrained knowledge, creating a stability–plasticity dilemma.
DSSG combines VLM-generated captions, dual-stream semantic guidance, and Dynamic CMKD for target adaptation with global categorical consistency.
DSSG-PAC replaces iteration-wise class-prompt encoding with periodic prototype calibration to reduce training overhead while preserving adaptive category guidance.

B. Data Priors: Generative Instance-level Captions

Generated captions provide instance-level target semantics for joint vision–text adaptation, but their unconstrained content can drift dynamically. DSSG therefore pairs the caption stream with re-encoded class anchors to balance local adaptation and global semantic stability.

VLM-generated captions provide instance-level target-domain guidance, including object attributes and contextual information, through image–caption contrastive alignment.
Unconstrained captions may emphasize non-discriminative details, backgrounds, out-of-distribution semantics, or cues from other classes, reducing category-discriminative reliability.
DSG runs caption and class-anchor streams in parallel, using complementary textual representations encoded by the shared text encoder.
Class prompts are re-encoded each iteration to preserve category semantics, while captions capture instance-specific target semantics.
Dynamic anchor self-training derives pseudo-labels from the teacher and applies them only when prediction confidence exceeds γ.

D. Dynamic Cross-Modal Knowledge Distillation

Dynamic CMKD replaces static teacher guidance with an evolving teacher distribution, calibrating student consistency during joint adaptation. Its agreement-sensitive objective combines confidence-based learning, distributional distillation, and teacher sharpening.

Static CMKD cannot adapt its teacher guidance to the evolving representation space, limiting teacher–student consistency.
Dynamic CMKD uses teacher and student distributions from DSG to enforce consistency during adaptation.
The consistency coefficient coe = sg(exp(−KL(P_stu ∥ P_tea))) measures prediction agreement while stopping gradient propagation.
When coe approaches 1, the task loss encourages confident predictions; when coe approaches 0, distillation increasingly uses the averaged student–teacher distribution.
The integrated objective combines caption contrastive alignment, confidence-thresholded self-training, and dynamic teacher–student consistency.

F. From DSSG to DSSG-PAC: Prototype Anchor Calibration

DSSG-PAC replaces iteration-wise class-prompt encoding with periodically calibrated prototype anchors that are reused between calibrations, reducing text-side overhead while retaining adaptation to the evolving text feature space.

Prototype Anchor Calibration: DSSG-PAC directly models text-encoded class features as prototype anchors periodically calibrated in the text feature space.The calibration uses the current text-encoder parameters.
Prototype Anchor Calibration: Calibrated prototypes remain fixed and are reused between calibration steps without propagating gradients through the text encoder.Updated text-encoder parameters regenerate the prototypes at the next calibration.
Calibration Schedule: The default setting uses Δcal = 1, corresponding to one prototype calibration per epoch.Other calibration intervals are evaluated in an ablation study.
Computational Efficiency: DSSG-PAC replaces iteration-level class-anchor generation with periodically calibrated anchors while leaving DSSG's other components and optimization objectives unchanged.This preserves prototype adaptability as the text feature space changes.
Computational Efficiency: With calibration interval Δcal, DSSG-PAC performs approximately ⌈Nep/Δcal⌉ class-prompt encoding operations instead of DSSG's NepI operations.The prototype matrix is maintained between calibrations, reducing workload that otherwise scales with target-domain size and class count.

IV. EXPERIMENTS

Experiments evaluate DSSG and DSSG-PAC across four benchmarks, standard baselines, strict SFF-DA settings, ablations, and prototype-calibration analyses. DSSG delivers strong adaptation accuracy, while DSSG-PAC preserves performance with lower text-side computation and stable teacher predictions.

Experimental Setup: Experiments cover Office-31, Office-Home, MiniDomainNet, and VisDA, using unimodal, multimodal-teacher, and multimodal SFF-DA baselines.Evaluation uses Last Epoch accuracy averaged over three random seeds because target labels are unavailable for model selection.
Standard Benchmarks: 88.8% and 92.8% Avg. Last accuracy are achieved by DSSG on Office-Home with ResNet-50 and ViT-B/16, outperforming ImCapSFDA by 4.6 and 2.1 percentage points.DSSG-PAC closely matches DSSG across both backbones.
Standard Benchmarks: 90.3% and 94.4% Avg. Last accuracy are achieved by DSSG on Office-31 with ResNet-50 and ViT-B/16, consistently outperforming existing SFF baselines.DSSG-PAC maintains competitive performance across both backbones.
Large-Scale Benchmarks: 87.1% and 89.67% Avg. Last accuracy are achieved by DSSG on Mini-DomainNet and VisDA with ResNet.The supplied passage reports these large-scale benchmark results but does not provide the corresponding baseline comparison in full.
Ablation Studies: Removing DSG or Dynamic CMKD generally lowers accuracy, while removing Lcon causes a VisDA RN101 drop from 89.67% to 10.35%.The ablations support complementary semantic guidance and dynamic distillation, with instance-level image-caption alignment mitigating catastrophic forgetting.
Prototype Calibration: Maximum prototype staleness and mean teacher KL decrease below 10^-4 and approximately 10^-5, respectively, after early adaptation.Periodic calibration therefore maintains teacher prediction consistency without iteration-wise class-prompt encoding; calibration intervals from 0.2 to 2 yield comparable results, with 1 adopted by default.

D. Computational Efficiency Analysis

DSSG-PAC reduces the text-side adaptation overhead of DSSG by periodically calibrating and reusing prototype anchors, while preserving inference efficiency. It achieves an average 18.9% reduction in adaptation time over DSSG.

DSSG incurs overhead from iteration-wise class-prompt encoding, whereas DSSG-PAC substantially reduces this cost through periodic calibration.DSSG adds no trainable parameters over ImCapSFDA.
DSSG computes class anchors once at inference, and DSSG-PAC directly uses calibrated prototype anchors without additional downstream inference overhead.
18.9% average adaptation-time reduction over DSSG is achieved by DSSG-PAC through periodic prototype calibration.Its runtime is comparable to ImCapSFDA.

E. Quantitative and Qualitative Analysis of Caption Semantics

The analyses show that generated captions provide useful target-specific semantics but can contain ambiguity, background or out-of-task cues, motivating class-anchor guidance. DSSG improves target-domain cluster separation while retaining strict source-free operation and DSSG-PAC reduces redundant text encoding.

Quantitative Caption Semantics: 16%–39% of captions match multiple task classes, revealing intra-task ambiguity that makes direct class-term matching unreliable for standalone classification.Implicit synonym matching captures more expressions than explicit ground-truth class-name matching.
Quantitative Caption Semantics: Caption-to-class similarity prediction exceeds CLIP zero-shot on several domains but remains below caption-driven adaptation.
Qualitative Caption Semantics: Captions can combine class-discriminative cues with background/OOD semantics or cues from other task classes, shifting attention away from the target category.Caption quality therefore depends on task-relevant semantic focus, not only descriptive richness.
Visualization Analysis: DSSG produces more compact and better-separated target clusters than CLIP zero-shot across all four benchmarks.The t-SNE visualizations indicate improved target-domain discriminability.
Guidance and Calibration: DSSG couples caption guidance with task-relevant class anchors, while Dynamic CMKD promotes teacher–student consistency and DSSG-PAC reuses periodically calibrated anchors.This design balances semantic plasticity and categorical stability while reducing redundant text encoding.
Strict SFF-DA: DSSG adapts a single pretrained VLM using only unlabeled target data and generated captions under the strict SFF-DA protocol.It requires neither source data nor a task-specific source model.
Comparison with Zero-Shot MLLM Inference: DSSG outperforms zero-shot MLLMs on average on Office-Home and MiniDomainNet and matches LLaVA-v1.6-34B on VisDA.DSSG uses a lightweight 0.12B CLIP backbone rather than repeatedly invoking a large MLLM.
Limitations: DSSG remains dependent on caption quality and acquisition, with local computation, offline storage, or online latency and privacy costs.Task-irrelevant or biased descriptions may still affect adaptation despite class-anchor regularization.

VI. APPENDIX MATERIAL

Appendix analyses support the DSSG design, show that Dynamic CMKD improves over static CMKD, and indicate that simple periodic prototype calibration tracks the evolving text space without extra correction mechanisms.

DSSG Ablation: Dynamic CMKD consistently outperforms static CMKD across datasets and backbones.The ablation also reports gains from DSG and static CMKD over the ImCapSFDA baseline.
PAC Variants: Strong momentum smoothing with m = 0.9 degrades performance, while weak smoothing and residual variants produce only marginal or inconsistent changes.
PAC Variants: Periodic calibration is sufficient to track the evolving text representation space without additional smoothing or residual correction.DSSG-PAC updates its prototype cache at calibration events and keeps it fixed until the next event.
Hyper-Parameter Sensitivity: DSSG maintains stable performance across a broad range of λ3 values while consistently outperforming the ImCapSFDA baseline.The sensitivity analysis indicates low sensitivity to λ3.

E. Proofs for the SFF-DA Risk Analysis

The appendix formalizes the SFF-DA risk analysis by specifying finite-sample certification conditions and handling classifier agreement, disagreement, and tied teacher predictions.

The appendix proves Theorem 1 and Corollary 1 and states conditions for estimating the population certificate from finite unlabeled data.
When teacher predictions are tied, the certification quantity is conservatively set to one, treating the prediction as uncertified.This uses the teacher margin m_q(x) = q^(1)(x) − q^(2)(x).
When teacher and student classifiers agree, the classifier-disagreement term is zero; otherwise, the proof lower-bounds the student–teacher score difference using the teacher margin.

3) Target-risk sandwich:

The analysis bounds student risk using semantic-teacher quality and teacher–student discrepancy, yielding a robust interval under stated calibration conditions. It also identifies observability and independence limits on interpreting the bound and its empirical estimates.

Risk bound: Theorem 1 bounds student risk between max{0, η − C_Q(p, q)} and min{1, η + C_Q(p, q)}.The bound depends on semantic-teacher quality η and teacher–student discrepancy C_Q(p, q).
Identifiability: The target-risk interval is not identifiable from the unlabeled marginal Q_X alone because different target labeling functions can share it.Target labels are reserved for post-hoc oracle evaluation rather than adaptation or model selection.
Finite-sample certificate: The finite-sample certificate requires an i.i.d. unlabeled sample independent of training, hyperparameter selection, and stopping decisions.Under boundedness, Hoeffding’s inequality supplies the stated high-probability control.
Finite-sample certificate: Reusing the transductive training sample to compute Ĉ violates the independence condition, so the resulting quantity is only an empirical diagnostic without further analysis.Cross-fitting, uniform convergence, or PAC-Bayes analysis would be needed to recover a formal certificate.

6) Relation to Dynamic CMKD:

Dynamic CMKD explicitly links consistency regularization to teacher–student distribution discrepancy, while the visualizations indicate more task-relevant activation patterns for DSSG. Baseline hyperparameters are reproduced through sensitivity-based selection and then reused for DSSG.

Dynamic CMKD: Dynamic CMKD imposes an explicit consistency penalty whose coefficient increases monotonically with D_KL(p∥q).The coefficient and accompanying Gini terms co-evolve, so this does not establish monotonic decrease of E∥p − q∥2.
Sensitivity analysis: The reproduced ImCapSFDA baseline is tuned for λ1 and λ2 using Fig. 8, and those selected settings are inherited by DSSG without additional tuning.The figure marks the selected optimal configuration with a star.
Grad-CAM visualization: DSSG shows more concentrated activations on discriminative object regions and fewer responses to background regions than the baseline.The reported visualization interprets this pattern as improved task-relevant semantic focus.