Unified modality separation: A vision-language framework for unsupervised domain adaptation

1 INTRODUCTION

• Modality gaps leave vision and text features distinctly distributed, so current methods relying on their similarity cannot align modality-specific knowledge well.The paper illustrates the difference with vision classifiers excelling on visually straightforward chairs and text classifiers on semantically rich file cabinets.Source: §1 · pp. 1–2
• UniMoS disentangles vision features into vision-associated and language-associated components, separates their adaptation, and combines their strengths with modality-adaptive test-time weights.An orthogonal regularization loss helps distinguish the components; MaE estimates thresholds and weights for target samples.Source: §1 · p. 2
• The modality discrepancy metric categorizes target samples by cross-modal prediction confidence and consistency, using modality-invariant samples to learn more robust decision boundaries.Uncertain samples can be annotated to enhance adaptation, drawing on active domain adaptation.Source: §1 · p. 2
• The framework is presented as compatible with prompt tuning and UDA, ADA, source-free ADA, and multi-source domain adaptation settings.The authors report evaluations across backbones, benchmarks, and prompt-tuning methods.Source: §1 · p. 2

2.1 Unsupervised Domain Adaptation

• UDA work measures or adversarially minimizes domain discrepancy, exploits target data structure, and adapts larger vision models and VLMs.VLM approaches include prompt tuning and full parameter tuning; UniMoS is described as disentangling CLIP vision features for modality-aware adaptation.Source: §2.1 · pp. 2–3

2.2 Active Domain Adaptation

• ADA selects informative target samples for labeling, while SFADA combines active labeling with source-free adaptation for data-sensitive settings.Prior methods described here target vision models and are not designed to identify VLMs’ precise annotation needs.Source: §2.2 · p. 3

2.3 Vision-language Models

• VLM adaptation includes prompt learning, parameter-efficient adapters, and training-free caches, while modality-gap research studies its causes, latent spaces, and bridging.The paper distinguishes its aim as protecting modality-specific information during adaptation.Source: §2.3 · p. 3

3.1 Framework Overview

• UniMoS++ separates CLIP vision features into vision-associated and language-associated components, produces separate modality predictions, and combines them for cross-modal inference.Its framework uses domain-specific prompts and learnable prompts for text features; test-time weights balance the contributions of the two components.Source: §3.1 · pp. 3–4

3.2 Modality Separation Networks

• Dual separation branches project vision features into VAC and LAC, while a paired orthogonal loss disentangles them without disrupting original cross-modal correspondences.The text branch classifies using LAC-to-text similarities, adjusted to mitigate prediction imbalance; the vision branch classifies VAC through linear layers.Source: §3.2 · p. 4

3.3 Modality Discrepancy Metric

• MDI categorizes target samples as modality-invariant, modality-specific, or uncertain using modality-output consistency and confidence, then scores samples to guide selection and use.MI-score accounts for both prediction consistency and confidence; MS samples have swapped top-two predictions across modalities, while samples outside MI and MS are UN. Selected confident MI samples support consistency training, and informative UN samples can be annotated.Source: §3.3 · pp. 4–5
• On clp→rel of Mini-DomainNet, MS samples showed the largest ensemble effect (+7.9%), while UN-a samples had the lowest classification accuracy (25%).The paper interprets these observations as supporting modality-specific predictions for MS samples and annotation of UN-a samples.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4 Modality-aware Training and Adaptation

• Modality-aware training distills pretrained CLIP knowledge into LAC and uses clustering and learnable train-time ensemble weights to optimize VAC separately.Strong-augmentation consistency supports semantic learning, annotated target labels can guide LAC, and a modality discriminator aligns source and target VAC and LAC distributions.Source: §3.4 · pp. 5–6

3.5 Modality-adaptive Ensemble

• MaE automatically sets the test-time ensemble weight by estimating the intersection of vision- and text-biased modality-threshold distributions.It derives a threshold δ for each sample and approximates the optimal weight using the most deviated point in the sorted δ values.Source: §3.5 · p. 7

3.6 Overall Algorithm

• For UDA, the algorithm sets the annotation budget b to zero and replaces the target annotation set with confident MI samples; pretrained CLIP encoders remain frozen and trainable modules are linear layers.The overall loss combines modality separation, modality-aware training, and modality-discrimination objectives.Source: §3.6 · p. 7
• UniMoS++ handles multiple adaptation settings with minimal changes, including UDA, ADA, MSDA, and SFADA.For MSDA, source domains are combined into a mixed domain and dynamic adaptation weights are learned with MaE; the passage also introduces SFADA as using only pretrained weights but gives no further detail.Source: §3.6 · p. 7
• MDI identifies instance-level modality characteristics to support active selection in ADA and more reliable pseudo labels in UDA.The algorithm uses MDI for active selection in ADA and for pseudo-labeling in UDA.Source: §3.6 · p. 7
• MaE boosts ensemble results in UDA and ADA and enables dynamic adaptation weights for MSDA.For MSDA, all source domains are combined into a mixed domain, whose distribution the learned weights fit.Source: §3.6 · p. 7
• UniMoS++ quickly adapts to a target domain by tuning prompts on unlabeled target data.The passage points to Section 4.2 for results highlighting the framework’s abilities but supplies no specific values.Source: §3.6 · p. 8

4.1 Dataset and Implementation Details

• Evaluation spans OfficeHome, VisDA-2017, DomainNet, Mini-DomainNet, and three selected image-classification tasks from WILDS.The datasets cover distinct domains and task settings, including synthetic-to-real transfer and in-the-wild adaptation.Source: §4.1 · p. 8

4.2 Main Results

• With MaE, UniMoS++ matches or beats optimal manual ensemble weights; it surpasses prior ResNet- or CLIP-based methods, leads DomainNet overall, and improves qdr accuracy from 21.3% to 25%.It slightly trails PADCLIP on VisDA but surpasses PADCLIP on OfficeHome without tuning.Source: §4.2 · pp. 8–9, Table 3 · p. 9
• On DomainNet MSDA, UniMoS++ surpasses recent state-of-the-art methods, while manually tailored ensemble weights lag under dynamic mixtures of source domains.The comparison covers DomainNet and OfficeHome; the stated state-of-the-art result is on DomainNet.Source: §4.2 · p. 9, Table 4 · p. 9
• On OfficeHome ADA, UniMoS++ variants exceed prior state-of-the-art by up to 4.2%; on Mini-DomainNet, the method improves accuracy by 9.1%.With 5% annotation on OfficeHome, averaged accuracy even exceeds fully supervised training on target data; source-free ADA sets a new state of the art.Source: §4.2 · pp. 9–10, Table 5 · p. 10, Table 6 · p. 10
• For unseen-domain generalization, the method surpasses recent CLIP-based state of the art on 2 of 3 WILDS datasets and raises Camelyon accuracy from 50.1% to 89.5%.Pseudo-labeling is removed because target data are unavailable in this setting.Source: §4.2 · p. 10, Table 7 · p. 11

4.3 Analysis

• Ablations report accuracy increasing from 80.9% to 82.4% as training components are added; removing MaE or MDI lowers overall accuracy, and random selection costs 2.7%.The analysis also reports reduced performance and added computational overhead for alternative attention- and RL-based weighting strategies; smaller source weights help on VisDA, while α=1 and β=1 work best on OfficeHome.Source: §4.3 · pp. 10–12, Table 8 · p. 11, Figure 8 · p. 12
• Feature visualizations show separated target VAC and LAC, aligned source and target vision features, and more informative active samples from MDI than Energy selection.The reported accuracy of MDI-selected active samples is 19.5% lower than Energy-selected samples, which the authors interpret as greater informativeness.Source: §4.3 · pp. 11–12, Figure 6 · p. 11
• UniMoS++ outperforms other ADA methods across active budgets of 3%–20%; its ensemble accuracy exceeds both modality outputs, with the lowest reported computational costs among compared methods.The computation analysis attributes its lower trainable-parameter count and high throughput to leaving CLIP encoders unchanged.Source: §4.3 · p. 12, Figure 7 · p. 12, Table 9 · p. 12

• UniMoS disentangles VLM features into vision- and language-associated components, handles them separately during training, and uses modality-adaptive test-time weights for cross-modal integration.MDI evaluates sample modality characteristics for instance-level domain adaptation.Source: §5 · p. 12
• The framework and MDI are presented as addressing the loss of modality-specific information in current VLM-based UDA methods.Source: §5 · p. 12
• The methods are compatible with varied prompt-tuning techniques and domain-adaptation settings, and the authors present them as insights and solutions for adapting VLMs.Source: §5 · p. 13